Re: Call for vote: Apache James JSPF 1.0.4

[Jean Helou]

+1

Le ven. 7 juin 2024 à 16:32, btell...@linagora.com 
a écrit :

> +1
>
> On 07/06/2024 15:39, Benoit TELLIER wrote:
> > Hi,
> >
> > I would like to propose a new vote for 1.0.4 release of the Apache
> > James JSPF library.
> >
> > You can find:
> >
> >  - The maven release staged in repository.apache.org as the artifact
> > #1083:
> > https://repository.apache.org/content/repositories/orgapachejames-1083/
> >  - The blog post for this release:
> > https://github.com/apache/james-project/pull/2286
> >
> > This release fixes asynchronous JSPF executors.
> >
> > Voting rules:
> >  - This is a majority approval: this release may not be vetoed.
> >  - A quorum of 3 binding votes is required
> >  - The vote starts at Friday 7th of June 2024, 3pm UTC+7
> >  - The vote ends at Friday 14th of June 2024, 3pm UTC+7
> >
> > You can answer to it just with +1 and -1. Down-votes may be motivated.
> >
> > 3 binding votes are expected move forward on this release.
> >
> > Cheers,
> >
> > Benoit TELLIER
> >
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: vote : Remote build cache experiment

[Jean Helou]

Hi again,

You may have forgotten (or think that I have :D) about this topic.
I do followup on a regular basis on the infra jira
https://issues.apache.org/jira/browse/INFRA-25461
The infra team is working with develocity to change the cache deployment
model. At the moment it requires provisionning a VM for each project which
doesn't scale well They haven't given me a timeline for now but have a next
touchpoint in a couple months.

Cheers,
Jean

Le mar. 5 mars 2024 à 22:24, Jean Helou  a écrit :

> hello everyone.
> Just to keep you posted, last week I did comment on the INFRA issue to get
> the process rolling.
> I am unfamiliar with the workflow setup for the infra board and didn't
> notice I was expected to click on waiting for infra for them to process my
> comment. i finally understood tonight why my comment was left unanswered
> :facepalm:
>
> cheers
> jean
>
> Le mer. 28 févr. 2024 à 19:57, Jean Helou  a écrit :
>
>> hello,
>>
>> I'm happy that the vote has succeded with 3 votes (I'm an idiot and
>> forgot to send my vote in a separate message T_T)
>>
>> I'll ask infra to proceed with the creation of the build cache node
>>
>> cheers
>> jean
>>
>>
>>
>> Le mar. 20 févr. 2024 à 10:36, Quan tran hong <
>> quan.tranhong1...@gmail.com> a écrit :
>>
>>> +1,
>>>
>>> Thanks for your enthusiastic effort on the topic.
>>>
>>> Cheers,
>>>
>>> Quan
>>>
>>> Vào Th 3, 20 thg 2, 2024 vào lúc 16:07 Rene Cordier <
>>> rcord...@apache.org>
>>> đã viết:
>>>
>>> > +1,
>>> >
>>> > If there is a chance to make our builds faster, I say we take it!
>>> Thanks
>>> > for this work :)
>>> >
>>> > Rene.
>>> >
>>> > On 2/20/24 15:18, Jean Helou wrote:
>>> > > Hello devs !
>>> > >
>>> > > I have not yet moved forward with enabling remote caching for the
>>> build
>>> > and
>>> > > tests in general. It is unclear to me if you are ok with trying it
>>> from
>>> > the
>>> > > previous discussion. So I'll call a more formal vote:
>>> > >
>>> > > Are you ok with experimenting enabling remote build cache for james
>>> (ci
>>> > > write+local read) ?
>>> > >
>>> > > Deployment of the remote build cache would follow the documented plan
>>> > >
>>> >
>>> https://docs.gradle.com/enterprise/maven-build-cache/#rolling_out_the_cache_in_your_organization
>>> > > TLDR;
>>> > > 1. move forward with the INFRA request to get our build cache node
>>> > > 2. test (read+write) with a few (2-3) volunteers locally (not
>>> enabled on
>>> > CI)
>>> > > 3. enable write on CI
>>> > > 4, enable read for everyone
>>> > >
>>> > > - voting starts 2024-02-20
>>> > > - voting ends 2024-02-27
>>> > >
>>> > > This is lazy consensus, this vote can be vetoed (please motivate your
>>> > > veto), there will be a second formal vote to definitely enable the
>>> cache
>>> > > once the experiment has been conducted and there are some results to
>>> > share.
>>> > >
>>> > > jean
>>> > >
>>> >
>>> > -
>>> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>> > For additional commands, e-mail: server-dev-h...@james.apache.org
>>> >
>>> >
>>>
>>

[jira] [Commented] (JAMES-4022) James server is getting crashed or stopped after 20 mins or with load

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829944#comment-17829944
 ] 

Jean Helou commented on JAMES-4022:
---

can you provide the steps to reproduce ? 
 * information on the binary distribution you used or how you built the server 
and which variant you used 
(https://github.com/apache/james-project/tree/master/server/apps)
 * the command you used
 * the infra the server was running on
 * if possible the script you used to test
 * what the server logs have to say
 * ideally instrumentation of the JVM that was running the server

with the information you provided so far there is little we can do or conclude.

 

cheers

> James server is getting crashed or stopped after 20 mins or with load
> -
>
> Key: JAMES-4022
> URL: https://issues.apache.org/jira/browse/JAMES-4022
> Project: James Server
>  Issue Type: Bug
>  Components: James Core
>Affects Versions: 3.3.0
>Reporter: mohan
>Priority: Major
> Fix For: 3.3.0
>
>
> The below occured on blazemeter perf testing.For 1000 users with 1hour perf 
> test scheduled.it ran only 10 mins after that no requests went jamesserver 
> hosts because ,james are down.
> 2024-03-22 12:20:39,665 INFO o.a.j.t.JMeterThread: Thread started: Thread 
> Group-ThreadStarter 1-249
> 2024-03-22 12:20:39,665 INFO o.a.j.t.JMeterThread: Thread started: Thread 
> Group-ThreadStarter 1-250
> 2024-03-22 12:32:23,824 WARN o.a.j.p.s.s.SmtpSampler: 
> com.sun.mail.smtp.SMTPSendFailedException: [EOF]
>   at 
> com.sun.mail.smtp.SMTPTransport.issueSendCommand(SMTPTransport.java:2108) 
> ~[mail-1.5.0-b01.jar:1.5.0-b01]
>   at com.sun.mail.smtp.SMTPTransport.finishData(SMTPTransport.java:1889) 
> ~[mail-1.5.0-b01.jar:1.5.0-b01]
>   at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:1120) 
> ~[mail-1.5.0-b01.jar:1.5.0-b01]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Commented] (JAMES-3978) Investigate using develocity to improve james build

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823786#comment-17823786
 ] 

Jean Helou commented on JAMES-3978:
---

a formal vote has validated an experimentation enabling develocity remote build 
cache for james builds.In the mean time we now have about a month of metrics 
and build scans and all. 

 

The metrics are mostly about CI as the capture of build scan has not been 
enabled systematically, therefore the local build cache gains are not very 
visible. However the insight brought by the build scans has been translated in 
a few improvements on the build configuration reducing the median time of the 
CI build stage a bit (see 
[https://ge.apache.org/scans/trends?search.relativeStartTime=P28D=*james*=master=clean%2Cinstall=Europe%2FParis)]

 

For the local build cache, I still haven't been able to properly configure 
caching for scala compilation, or scala typecheck which are both quite 
expensive.

 

As reported on the mailing list, enably flakyness detection enables a deeper 
dive into james tests :

[https://ge.apache.org/scans/tests?search.relativeStartTime=P28D=*james*=master=test%2Cjacoco:report-aggregate@jacoco-report=Europe%2FParis=FLAKY#]

it seems that 
[org.apache.james.transport.mailets.RemoteDeliveryErrorHandlingTest|https://ge.apache.org/scans/tests?search.relativeStartTime=P28D=*james*=master=test%2Cjacoco:report-aggregate@jacoco-report=Europe%2FParis=org.apache.james.transport.mailets.RemoteDeliveryErrorHandlingTest=FLAKY]
 is especially flaky both on master and accross all PRs

cassandra based integration tests are also quite flaky, with quite a few 
failures linked to

```

[ com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out 
after 
PT2S|https://ge.apache.org/s/uqvq75ttvkrs6/tests/goal/org.apache.james:blob-cassandra:surefire:test@default-test/details/org.apache.james.blob.cassandra.CassandraBlobStoreClOneTest/blobStoreShouldSupport100MBBlob]

```

> Investigate using develocity to improve james build
> ---
>
> Key: JAMES-3978
> URL: https://issues.apache.org/jira/browse/JAMES-3978
> Project: James Server
>  Issue Type: Improvement
>  Components: Build System
>    Reporter: Jean Helou
>Priority: Major
> Attachments: image-2024-02-02-16-30-20-026.png, 
> image-2024-02-02-16-31-24-883.png
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> Hello,
> A while ago I noticed [this 
> announcement|https://news.apache.org/foundation/entry/the-apache-software-foundation-announces-gradle-as-a-platinum-targeted-sponsor]
>  gradle becoming a platinum sponsor of the ASF.
> In particular the announcement was about gradle offering the develocity 
> service  to  the ASF. It so happens that I investigated develocity as part of 
> my day job, and while we can't afford it there it is definitely something we 
> might leverage here !
> The tool is available at [https://ge.apache.org|https://ge.apache.org/]
> I want to investigate the following leads
>  
>  # local caching for builds enable reuse of tasks output to speed up repeated 
> tasks
>  # build scans to provide detailed build reports and individual build metrics
>  # build metrics are collected across all build scans and allow to target 
> objectively bad contributors to overall build time
>  # enabling the remote build cache
>  **  additionally improves both local and CI build times
>  ** requires some attention to read/write permissions to avoid cache 
> poisoning and supply chain attacks
>  ** most likely requires a dedicated build cache node request to infra
>  ** several projects already use it so it is a known unknown ( we know it can 
> be done we don't yet know how)
>  # leverage advanced  test features of develocity unknown unknowns at least 
> for me:D
>  ## the tool collects tests metrics to pinpoint flaky tests enabling targeted 
> fix efforts
>  ## the tool claims to be able to do test avoidance and not rerun tests that 
> are irrelevant to the changes
>  ## the tool claims to be able to distribute test excecution accross 
> dedicated test workers
> Since we use maven,  we won't get the full benefits of the build cache ( 
> gradle has been pushing plugin authors to write build cache friendly code for 
> a long time ) but it should  improve the current situation quite a bit.
>  
> Apache pulsar is also using develocity with maven:
>  - develocity configuration 
> [https://github.com/apache/pulsar/blob/master/.mvn/gradle-enterprise.xml]
>  - infra wiki has infos on how to onboard 
> https://cwiki.apache.org/confluence/display/INFRA/Project+Onboarding+Instructions+for+Develocity



--
This

Re: vote : Remote build cache experiment

[Jean Helou]

hello everyone.
Just to keep you posted, last week I did comment on the INFRA issue to get
the process rolling.
I am unfamiliar with the workflow setup for the infra board and didn't
notice I was expected to click on waiting for infra for them to process my
comment. i finally understood tonight why my comment was left unanswered
:facepalm:

cheers
jean

Le mer. 28 févr. 2024 à 19:57, Jean Helou  a écrit :

> hello,
>
> I'm happy that the vote has succeded with 3 votes (I'm an idiot and forgot
> to send my vote in a separate message T_T)
>
> I'll ask infra to proceed with the creation of the build cache node
>
> cheers
> jean
>
>
>
> Le mar. 20 févr. 2024 à 10:36, Quan tran hong 
> a écrit :
>
>> +1,
>>
>> Thanks for your enthusiastic effort on the topic.
>>
>> Cheers,
>>
>> Quan
>>
>> Vào Th 3, 20 thg 2, 2024 vào lúc 16:07 Rene Cordier <
>> rcord...@apache.org>
>> đã viết:
>>
>> > +1,
>> >
>> > If there is a chance to make our builds faster, I say we take it! Thanks
>> > for this work :)
>> >
>> > Rene.
>> >
>> > On 2/20/24 15:18, Jean Helou wrote:
>> > > Hello devs !
>> > >
>> > > I have not yet moved forward with enabling remote caching for the
>> build
>> > and
>> > > tests in general. It is unclear to me if you are ok with trying it
>> from
>> > the
>> > > previous discussion. So I'll call a more formal vote:
>> > >
>> > > Are you ok with experimenting enabling remote build cache for james
>> (ci
>> > > write+local read) ?
>> > >
>> > > Deployment of the remote build cache would follow the documented plan
>> > >
>> >
>> https://docs.gradle.com/enterprise/maven-build-cache/#rolling_out_the_cache_in_your_organization
>> > > TLDR;
>> > > 1. move forward with the INFRA request to get our build cache node
>> > > 2. test (read+write) with a few (2-3) volunteers locally (not enabled
>> on
>> > CI)
>> > > 3. enable write on CI
>> > > 4, enable read for everyone
>> > >
>> > > - voting starts 2024-02-20
>> > > - voting ends 2024-02-27
>> > >
>> > > This is lazy consensus, this vote can be vetoed (please motivate your
>> > > veto), there will be a second formal vote to definitely enable the
>> cache
>> > > once the experiment has been conducted and there are some results to
>> > share.
>> > >
>> > > jean
>> > >
>> >
>> > -
>> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>> > For additional commands, e-mail: server-dev-h...@james.apache.org
>> >
>> >
>>
>

Re: Result of Proof Of Concept for Redis Event bus user notifications

[Jean Helou]

Hello Benoit

This is very clear thanks. I was not implying you should use pulsar :)  I
use a sass deployment and I am aware it is quite complex to operate:)
I might well end up deciding that a pulsar+redis makes more sense than
pulsar only when I start trying to deploy the MDA side of James:)

 - Currently the James domain logic would imply one topic per mailbox
> (more or less) and we would too quickly IMO end up with an unconfortable
> number of topics (million+). A refactoring would be needed.
>

Hmm this is interesting. I will have to review if it would be problematic
for a pulsar implem do you have a good entry point (class name(s)) I should
look at to investigate ?

Thanks again for the explanation

Cheers
Jean

 Note that all of that makes it a Linagora topic, at least now. If
> relevant, we would happily contribute it back ;-)
>
> Is this more clear?
>
> Best regards,
>
> Benoit TELLIER
>
> On 01/03/2024 13:17, Jean Helou wrote:
> > Hello 
> >
> > I'm not (yet) familiar with the event bus :)
> >
> > Out of curiosity could you describe what made you chose redis and what
> > other systems you considered ?
> > What characteristics you thought would match well with the problem.
> >
> > I'm interested in part because there is a Jira to implement the event bus
> > over pulsar so I'd like to take the opportunity to better understand the
> > design ideas behind the event bus as that would help with the pulsar
> > implementation.
> >
> > Jean
> >
> >
> > Le mer. 28 févr. 2024 à 10:09, Quan tran hong <
> quan.tranhong1...@gmail.com>
> > a écrit :
> >
> >> Hi folks,
> >>
> >> Following the context of the Jira ticket JAMES-3996 POC: Move RabbitMQ
> >> Event bus user notifications to Redis
> >> <
> >>
> https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3996?filter=allopenissues
> >> (TLDR:
> >> we observed some annoying issues with RabbitMQ in a deployment and we
> think
> >> it could be better to move at least the user notifications part of Event
> >> Bus to Redis), I did a Proof Of Concept about that in the PR JAMES 3996
> >> Redis event bus POC <https://github.com/apache/james-project/pull/2028
> >.
> >>
> >> The POC is considered done to me, and I want to share the POC result to
> >> James devs:
> >> - It is *possible *(the POC worked!) to replace the Event bus user
> >> notifications (key registration) using Redis Pub/Sub. And likely the
> whole
> >> EventBus (the group registration part as well) can rely on Redis (I did
> not
> >> do that part).
> >> - I did several performance tests for the POC, and the results were
> *good*.
> >> Regarding the metrics of Event Bus user notifications, Redis seems to
> >> outperform and show more stability in response time than RabbitMQ. (For
> >> more details, I shared on the PR)
> >>
> >> Despite the POC has been worked and shown some prospects, I think we
> should
> >> monitor it more carefully before applying it to James. Therefore, we
> >> (Linagora) would adopt the Redis Event bus user notifications first in
> our
> >> TMail project (based on James). We would keep an eye on its stability
> >> before contributing the fine-tuned work toward James.
> >>
> >> By the way, it seems someone from the community is interested in
> building
> >> the whole EventBus using Redis cf
> >> https://issues.apache.org/jira/browse/JAMES-3956. This POC could be a
> >> start
> >> for that Redis EventBus.
> >>
> >> What do you think about using Redis for EventBus? Would that sound
> >> interesting to you?
> >>
> >> Regards,
> >>
> >> Quan
> >>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Result of Proof Of Concept for Redis Event bus user notifications

[Jean Helou]

Hello 

I'm not (yet) familiar with the event bus :)

Out of curiosity could you describe what made you chose redis and what
other systems you considered ?
What characteristics you thought would match well with the problem.

I'm interested in part because there is a Jira to implement the event bus
over pulsar so I'd like to take the opportunity to better understand the
design ideas behind the event bus as that would help with the pulsar
implementation.

Jean


Le mer. 28 févr. 2024 à 10:09, Quan tran hong 
a écrit :

> Hi folks,
>
> Following the context of the Jira ticket JAMES-3996 POC: Move RabbitMQ
> Event bus user notifications to Redis
> <
> https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3996?filter=allopenissues
> >
> (TLDR:
> we observed some annoying issues with RabbitMQ in a deployment and we think
> it could be better to move at least the user notifications part of Event
> Bus to Redis), I did a Proof Of Concept about that in the PR JAMES 3996
> Redis event bus POC .
>
> The POC is considered done to me, and I want to share the POC result to
> James devs:
> - It is *possible *(the POC worked!) to replace the Event bus user
> notifications (key registration) using Redis Pub/Sub. And likely the whole
> EventBus (the group registration part as well) can rely on Redis (I did not
> do that part).
> - I did several performance tests for the POC, and the results were *good*.
> Regarding the metrics of Event Bus user notifications, Redis seems to
> outperform and show more stability in response time than RabbitMQ. (For
> more details, I shared on the PR)
>
> Despite the POC has been worked and shown some prospects, I think we should
> monitor it more carefully before applying it to James. Therefore, we
> (Linagora) would adopt the Redis Event bus user notifications first in our
> TMail project (based on James). We would keep an eye on its stability
> before contributing the fine-tuned work toward James.
>
> By the way, it seems someone from the community is interested in building
> the whole EventBus using Redis cf
> https://issues.apache.org/jira/browse/JAMES-3956. This POC could be a
> start
> for that Redis EventBus.
>
> What do you think about using Redis for EventBus? Would that sound
> interesting to you?
>
> Regards,
>
> Quan
>

Re: vote : Remote build cache experiment

[Jean Helou]

hello,

I'm happy that the vote has succeded with 3 votes (I'm an idiot and forgot
to send my vote in a separate message T_T)

I'll ask infra to proceed with the creation of the build cache node

cheers
jean



Le mar. 20 févr. 2024 à 10:36, Quan tran hong 
a écrit :

> +1,
>
> Thanks for your enthusiastic effort on the topic.
>
> Cheers,
>
> Quan
>
> Vào Th 3, 20 thg 2, 2024 vào lúc 16:07 Rene Cordier  >
> đã viết:
>
> > +1,
> >
> > If there is a chance to make our builds faster, I say we take it! Thanks
> > for this work :)
> >
> > Rene.
> >
> > On 2/20/24 15:18, Jean Helou wrote:
> > > Hello devs !
> > >
> > > I have not yet moved forward with enabling remote caching for the build
> > and
> > > tests in general. It is unclear to me if you are ok with trying it from
> > the
> > > previous discussion. So I'll call a more formal vote:
> > >
> > > Are you ok with experimenting enabling remote build cache for james (ci
> > > write+local read) ?
> > >
> > > Deployment of the remote build cache would follow the documented plan
> > >
> >
> https://docs.gradle.com/enterprise/maven-build-cache/#rolling_out_the_cache_in_your_organization
> > > TLDR;
> > > 1. move forward with the INFRA request to get our build cache node
> > > 2. test (read+write) with a few (2-3) volunteers locally (not enabled
> on
> > CI)
> > > 3. enable write on CI
> > > 4, enable read for everyone
> > >
> > > - voting starts 2024-02-20
> > > - voting ends 2024-02-27
> > >
> > > This is lazy consensus, this vote can be vetoed (please motivate your
> > > veto), there will be a second formal vote to definitely enable the
> cache
> > > once the experiment has been conducted and there are some results to
> > share.
> > >
> > > jean
> > >
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>

Build metrics

[Jean Helou]

Hi fellows,

The Develocity  integration of james-project build
has now been running for long enough that we can start seeing interesting
things beyond the raw build scans.
First we can give a short look at build trends

for the `clean install` command on master (the build stage), its hard to
derive actual trends since
- we only have 15 days of history
- we run on runners with varying cpu power ( build scans show T1C go from
16 to 24)
but it can be used to establish a baseline so we can compare later on
In average this stage took 9min10s
the week from feb 5 to feb 11th has had a lot more spread 5th-95th
percentile was 4m54s to 11m26s
last week was a bit more homogenous with the same percentiles being 7m54 -
10m18
this week so far looks more similar to last week than the previous, 7m12 -
9m4

Adding the local build cache added about 14s of overhead and provided no
benefits on CI (completely expected, maybe I should disable local caching
on CI since it spawns in a fresh env everytime) but this will be
interesting to compare with the remote build cache.

The tests

monitoring is also interesting. This screen monitors 2 things:
- which tests fail the most often
- which tests are flaky
To measure this I enabled retries for surefire in this commit
 so that a failing
test is retried once.
Failed is both tries failed
Flakyness is derived by looking at retries (1 fail followed by 1 success).

The results probably won't surprise you too much :

The test suite with most failures (11% of the runs) is
org.apache.james.blob.cassandra.CassandraBlobStoreTest

The test suite with most flakyness (6% of the runs) is
org.apache.james.blob.cassandra.CassandraBlobStoreDAOTest

The longest running testsuite is
org.apache.james.backends.rabbitmq.RabbitMQTest

at 4min27s

Top ten failing test suites
org.apache.james.blob.cassandra.CassandraBlobStoreTest
org.apache.james.blob.cassandra.cache.CachedBlobStoreTest
org.apache.james.blob.cassandra.CassandraBucketDAOTest
org.apache.james.blob.cassandra.CassandraDefaultBucketDAOTest
org.apache.james.blob.cassandra.CassandraPassTroughBlobStoreTest
org.apache.james.blob.cassandra.CassandraBlobStoreClOneTest
org.apache.james.blob.cassandra.CassandraBlobStoreDAOTest
org.apache.james.blob.cassandra.cache.CassandraBlobStoreCacheTest
org.apache.james.blob.objectstorage.aws.S3MinioTest
org.apache.james.blob.objectstorage.aws.S3BlobStoreDAOTest

Top ten flaky doesn't completely overlap
org.apache.james.blob.cassandra.CassandraBlobStoreDAOTest
org.apache.james.user.ldap.ReadOnlyUsersLDAPRepositoryWithLocalPartAsLoginNameTest$WhenEnableVirtualHosting$LocalPartLogin
org.apache.james.user.ldap.ReadOnlyUsersLDAPRepositoryWithLocalPartAsLoginNameTest$WhenEnableVirtualHosting$OneAppPasswordLogin
org.apache.james.user.ldap.ReadOnlyUsersLDAPRepositoryTest$WhenDisableVirtualHosting
org.apache.james.blob.cassandra.CassandraBlobStoreTest
org.apache.james.user.ldap.ReadOnlyUsersLDAPRepositoryWithLocalPartAsLoginNameTest$WhenEnableVirtualHosting
org.apache.james.blob.cassandra.CassandraPassTroughBlobStoreTest
org.apache.james.user.ldap.LdapHealthCheckTest
org.apache.james.user.ldap.ReadOnlyUsersLDAPRepositoryTest$WhenEnableVirtualHosting
org.apache.james.webadmin.integration.rabbitmq.vault.RabbitMQDeletedMessageVaultIntegrationTest

you can follow all the details on the apache develocity :D

Cheers
jean

vote : Remote build cache experiment

[Jean Helou]

Hello devs !

I have not yet moved forward with enabling remote caching for the build and
tests in general. It is unclear to me if you are ok with trying it from the
previous discussion. So I'll call a more formal vote:

Are you ok with experimenting enabling remote build cache for james (ci
write+local read) ?

Deployment of the remote build cache would follow the documented plan
https://docs.gradle.com/enterprise/maven-build-cache/#rolling_out_the_cache_in_your_organization
TLDR;
1. move forward with the INFRA request to get our build cache node
2. test (read+write) with a few (2-3) volunteers locally (not enabled on CI)
3. enable write on CI
4, enable read for everyone

- voting starts 2024-02-20
- voting ends 2024-02-27

This is lazy consensus, this vote can be vetoed (please motivate your
veto), there will be a second formal vote to definitely enable the cache
once the experiment has been conducted and there are some results to share.

jean

Re: Vote to merge the Java 21 adoption for Apache James PR

[Jean Helou]

Hello Eugen,

Libraries IMO should be released with support for older versions versions.
> Not every project can use the latest and greatest JVM.
> And I don't think low level libraries (mime4j) will benefit a lot from
> new Java language features.


Just to clarify : what do you mean by libraries ? some of the modules in
the james-project reactor or only other projects such as mime4j ?
I don't think the switch is affecting mime4j, jsieve, jdkim or jspf as far
as I have seen, it's only for things in the james-project repo.

Given this are you fine with source 21 and target 21  ?

Jean

Re: Develocity

[Jean Helou]

>
> > The risk is caching a negative result coming from a flaky test. And I
> feel
> like James test suite is quite unstable.
>
> On that, maybe we can find if the Gradle extension / surefire supports
> caching only if the build (test phase for example) succeeds?
>
>
I searched the documentation again (
https://docs.gradle.com/enterprise/maven-extension/#maven_surefire_plugin_and_maven_failsafe_plugin
)
> Test results for surefire:test are stored in the Build Cache whenever the
goal succeeds. Thus, by default, only successful or skipped test results
are cached. However, if 
<https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore>
is set to true, test failures are cached as well.

And good news  is not used in the james build so I can´t
see a reason not to enable remote caching.

jean

Quan
>
> Vào Th 4, 7 thg 2, 2024 vào lúc 10:22 Rene Cordier 
> đã viết:
>
> > I agree caching the test suite is very tempting, it would help gain
> > so much time on the build.
> >
> > But yes the test suite is flaky unfortunately. Maybe we need extra work
> > making it more stable first, but that's not easy task ^^'
> >
> > Or maybe for remote caching on test execution, maybe we can try to
> > execute that in an other branch in parallel to master, giving us time to
> > monitor the behavior of the build, before using it for good? Requires
> > extra work, efforts and attention yes, but I hardly see how else we can
> > go safely forward on this. But somebody else has a better idea maybe? :)
> >
> > Regards,
> >
> > Rene.
> >
> > On 2/7/24 02:58, Jean Helou wrote:
> > > Thanks for trying
> > >
> > > As you can see in the output only about 10% of the tasks you ran were
> > > cached.
> > >
> > > Which explains why the gains are not so great.
> > >
> > > There are 2 good news
> > > - it is possible to configure more goals to be cached. I have an
> attempt
> > at
> > > caching scala compile (which doesn't work for now as that goal has
> rather
> > > complex input and outputs).
> > > I intend to look at build scans to cache more high CPU consuming goals
> > and
> > > to avoid running some goals entirely when possible (as per my comments
> on
> > > the PR)
> > >
> > > Where it becomes really interesting (the second good news) is that
> > surefire
> > > is supported out of the box, so if one pays the cost of running the
> > tests,
> > > every subsequent build would only run the tests for the changed
> projects
> > > and their dependencies
> > >
> > > And testing is by far the most time consuming goal in James build.
> > >
> > > Remote caching of test execution would have the most impact,
> accelerating
> > > both local and CI builds because it would allow sharing the cache
> between
> > > different machines/nodes. (Only the ci would write but everyone could
> > read
> > > if I understand correctly)
> > >
> > > The risk is caching a negative result coming from a flaky test. And I
> > feel
> > > like James test suite is quite unstable.
> > >
> > > So I'm wondering if I should go ahead with the remote build cache node
> > > request to apache INFRA or revisit that request in a few months.
> > >
> > > Jean
> > >
> > >
> > >
> > > Le mar. 6 févr. 2024 à 11:18, Quan tran hong <
> > quan.tranhong1...@gmail.com>
> > > a écrit :
> > >
> > >> Hi Jean,
> > >>
> > >> I tried your work locally and the build time is truly better.
> > >> `mvn clean install -DskipTests -Dmaven.skip.doc=true
> > >> -Dgradle.cache.local.enabled=true`
> > >>
> > >> 1st build:
> > >> [INFO] Total time:  22:18 min
> > >> [INFO] 8053 goals, 7996 executed, 57 from cache, saving at least 13s
> > >>
> > >> 2nd build:
> > >> [INFO] Total time:  15:07 min
> > >> [INFO] 8053 goals, 7296 executed, 757 from cache, saving at least 3m
> 11s
> > >>
> > >> Thank you for the initiative :-)
> > >>
> > >> Quan
> > >>
> > >> Vào Th 3, 6 thg 2, 2024 vào lúc 14:48 Jean Helou  >
> > đã
> > >> viết:
> > >>
> > >>> Hi devs,
> > >>>
> > >>> I realized this morning that I initiated the conversation on the
> wrong
> > >> list
> > >>> (server-user inste

Re: Develocity

[Jean Helou]

Thanks for trying

As you can see in the output only about 10% of the tasks you ran were
cached.

Which explains why the gains are not so great.

There are 2 good news
- it is possible to configure more goals to be cached. I have an attempt at
caching scala compile (which doesn't work for now as that goal has rather
complex input and outputs).
I intend to look at build scans to cache more high CPU consuming goals and
to avoid running some goals entirely when possible (as per my comments on
the PR)

Where it becomes really interesting (the second good news) is that surefire
is supported out of the box, so if one pays the cost of running the tests,
every subsequent build would only run the tests for the changed projects
and their dependencies

And testing is by far the most time consuming goal in James build.

Remote caching of test execution would have the most impact, accelerating
both local and CI builds because it would allow sharing the cache between
different machines/nodes. (Only the ci would write but everyone could read
if I understand correctly)

The risk is caching a negative result coming from a flaky test. And I feel
like James test suite is quite unstable.

So I'm wondering if I should go ahead with the remote build cache node
request to apache INFRA or revisit that request in a few months.

Jean



Le mar. 6 févr. 2024 à 11:18, Quan tran hong 
a écrit :

> Hi Jean,
>
> I tried your work locally and the build time is truly better.
> `mvn clean install -DskipTests -Dmaven.skip.doc=true
> -Dgradle.cache.local.enabled=true`
>
> 1st build:
> [INFO] Total time:  22:18 min
> [INFO] 8053 goals, 7996 executed, 57 from cache, saving at least 13s
>
> 2nd build:
> [INFO] Total time:  15:07 min
> [INFO] 8053 goals, 7296 executed, 757 from cache, saving at least 3m 11s
>
> Thank you for the initiative :-)
>
> Quan
>
> Vào Th 3, 6 thg 2, 2024 vào lúc 14:48 Jean Helou  đã
> viết:
>
> > Hi devs,
> >
> > I realized this morning that I initiated the conversation on the wrong
> list
> > (server-user instead of server-dev) ...
> > see the archives
> > https://lists.apache.org/list.html?server-u...@james.apache.org for
> Rene's
> > positive answer
> >
> > sorry about that
> >
> > jean
> >
> > -- Forwarded message -
> > From: Jean Helou 
> > Date: Mon, Feb 5, 2024 at 3:52 PM
> > Subject: Develocity
> > To: James Users List 
> >
> >
> > Hello fellow james devs,
> >
> > I'm not sure how much context to provide in the email you can read more
> in
> >
> >
> https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3978?filter=allopenissues
> >
> > TLDR; A few months ago the apache foundation announced a sponsorship of
> the
> > "gradle" company for a tool to diagnose, monitor and optimize builds  (
> > called gradle enterprise at the time and renamed develocity since)
> >
> > I posted a first PR to plug james build in that tool
> > https://github.com/apache/james-project/pull/1972
> >
> > This first integration brings only build scans and local caching
> > - build scans can help us understand what slows our build
> > - local caching allows local build perf improvements (I posted a small
> > benchmark which shows a divide by 3 the time spent building
> > james-server-core in a repeat build with no changes)
> >
> > The build scans also show that many tasks in the build are not actually
> > cached so there is room for more improvement.
> >
> > The local cache does apply to surefire tasks,unfortunately the benefits
> for
> > CI are very small :
> >
> > In the first stage we cache the compilation result itself but that
> > represents a very small part of the test stages (even if we had to
> compile
> > everything from scratch that's barely 20 minutes out of the 3h30 of the
> > build so caching only marginally improves the build time at the moment
> >
> > the next stage would be to :
> > - make more tasks cacheable
> > - enable the remote build cache
> >
> > The remote build cache allows to share caching accross builds which means
> > that if a change affects only a small portion of the build, only that
> > change would effectively be recompiled and retested, so the closer to the
> > "leafs" of the reactor a change would be the shorter its CI pipelines
> would
> > be.
> > I pinged INFRA in https://issues.apache.org/jira/browse/INFRA-25461 to
> > know
> > what the process is for enabling the remote build cache and it seems it
> > starts with a mailing list thread :D
> >
> > note that
> > - only the CI would b

Fwd: Develocity

[Jean Helou]

Hi devs,

I realized this morning that I initiated the conversation on the wrong list
(server-user instead of server-dev) ...
see the archives
https://lists.apache.org/list.html?server-u...@james.apache.org for Rene's
positive answer

sorry about that

jean

-- Forwarded message -
From: Jean Helou 
Date: Mon, Feb 5, 2024 at 3:52 PM
Subject: Develocity
To: James Users List 


Hello fellow james devs,

I'm not sure how much context to provide in the email you can read more in
https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3978?filter=allopenissues

TLDR; A few months ago the apache foundation announced a sponsorship of the
"gradle" company for a tool to diagnose, monitor and optimize builds  (
called gradle enterprise at the time and renamed develocity since)

I posted a first PR to plug james build in that tool
https://github.com/apache/james-project/pull/1972

This first integration brings only build scans and local caching
- build scans can help us understand what slows our build
- local caching allows local build perf improvements (I posted a small
benchmark which shows a divide by 3 the time spent building
james-server-core in a repeat build with no changes)

The build scans also show that many tasks in the build are not actually
cached so there is room for more improvement.

The local cache does apply to surefire tasks,unfortunately the benefits for
CI are very small :

In the first stage we cache the compilation result itself but that
represents a very small part of the test stages (even if we had to compile
everything from scratch that's barely 20 minutes out of the 3h30 of the
build so caching only marginally improves the build time at the moment

the next stage would be to :
- make more tasks cacheable
- enable the remote build cache

The remote build cache allows to share caching accross builds which means
that if a change affects only a small portion of the build, only that
change would effectively be recompiled and retested, so the closer to the
"leafs" of the reactor a change would be the shorter its CI pipelines would
be.
I pinged INFRA in https://issues.apache.org/jira/browse/INFRA-25461 to know
what the process is for enabling the remote build cache and it seems it
starts with a mailing list thread :D

note that
- only the CI would be allowed to write to the remote build cache
- build caches both local and remote can be disabled with properties (IIRC
there are pretty strict constraints on release processes in the apache
foundation)
- I think it is possible to have the remote build cache be activated in
readonly mode for everyone  so that anyone pulling the repo would
experience a very fast first build (but that remains to be confirmed as I
have never experienced it)

what do you think ?

jean

[jira] [Updated] (JAMES-3978) Investigate using develocity to improve james build

[Jean Helou (Jira)]

 [ 
https://issues.apache.org/jira/browse/JAMES-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean Helou updated JAMES-3978:
--
Description: 
Hello,

A while ago I noticed [this 
announcement|https://news.apache.org/foundation/entry/the-apache-software-foundation-announces-gradle-as-a-platinum-targeted-sponsor]
 gradle becoming a platinum sponsor of the ASF.

In particular the announcement was about gradle offering the develocity service 
 to  the ASF. It so happens that I investigated develocity as part of my day 
job, and while we can't afford it there it is definitely something we might 
leverage here !

The tool is available at [https://ge.apache.org|https://ge.apache.org/]

I want to investigate the following leads

 
 # local caching for builds enable reuse of tasks output to speed up repeated 
tasks
 # build scans to provide detailed build reports and individual build metrics
 # build metrics are collected across all build scans and allow to target 
objectively bad contributors to overall build time
 # enabling the remote build cache
 **  additionally improves both local and CI build times
 ** requires some attention to read/write permissions to avoid cache poisoning 
and supply chain attacks
 ** most likely requires a dedicated build cache node request to infra
 ** several projects already use it so it is a known unknown ( we know it can 
be done we don't yet know how)
 # leverage advanced  test features of develocity unknown unknowns at least for 
me:D
 ## the tool collects tests metrics to pinpoint flaky tests enabling targeted 
fix efforts
 ## the tool claims to be able to do test avoidance and not rerun tests that 
are irrelevant to the changes
 ## the tool claims to be able to distribute test excecution accross dedicated 
test workers

Since we use maven,  we won't get the full benefits of the build cache ( gradle 
has been pushing plugin authors to write build cache friendly code for a long 
time ) but it should  improve the current situation quite a bit.

 

Apache pulsar is also using develocity with maven: 

- develocity configuratio 
[https://github.com/apache/pulsar/blob/master/.mvn/gradle-enterprise.xml]

  was:
Hello,

A while ago I noticed [this 
announcement|https://news.apache.org/foundation/entry/the-apache-software-foundation-announces-gradle-as-a-platinum-targeted-sponsor]
 gradle becoming a platinum sponsor of the ASF.

In particular the announcement was about gradle offering the develocity service 
 to  the ASF. It so happens that I investigated develocity as part of my day 
job, and while we can't afford it there it is definitely something we might 
leverage here !

The tool is available at https://ge.apache.org

I want to investigate the following leads

 
 # local caching for builds enable reuse of tasks output to speed up repeated 
tasks
 # build scans to provide detailed build reports and individual build metrics
 # build metrics are collected across all build scans and allow to target 
objectively bad contributors to overall build time
 # enabling the remote build cache
 **  additionally improves both local and CI build times
 ** requires some attention to read/write permissions to avoid cache poisoning 
and supply chain attacks
 ** most likely requires a dedicated build cache node request to infra
 ** several projects already use it so it is a known unknown ( we know it can 
be done we don't yet know how)
 # leverage advanced  test features of develocity unknown unknowns at least for 
me:D
 ## the tool collects tests metrics to pinpoint flaky tests enabling targeted 
fix efforts
 ## the tool claims to be able to do test avoidance and not rerun tests that 
are irrelevant to the changes
 ## the tool claims to be able to distribute test excecution accross dedicated 
test workers

Since we use maven,  we won't get the full benefits of the build cache ( gradle 
has been pushing plugin authors to write build cache friendly code for a long 
time ) but it should  improve the current situation quite a bit.


> Investigate using develocity to improve james build
> ---
>
> Key: JAMES-3978
> URL: https://issues.apache.org/jira/browse/JAMES-3978
> Project: James Server
>  Issue Type: Improvement
>  Components: Build System
>    Reporter: Jean Helou
>Priority: Major
> Attachments: image-2024-02-02-16-30-20-026.png, 
> image-2024-02-02-16-31-24-883.png
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Hello,
> A while ago I noticed [this 
> announcement|https://news.apache.org/foundation/entry/the-apache-software-foundation-announces-gradle-as-a-platinum-targeted-sponsor]
>  gradle becoming a platinum sponsor of the ASF.
> In particular the announcement was about gradle offering the develocity 
> service 

[jira] [Updated] (JAMES-3978) Investigate using develocity to improve james build

[Jean Helou (Jira)]

 [ 
https://issues.apache.org/jira/browse/JAMES-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean Helou updated JAMES-3978:
--
Description: 
Hello,

A while ago I noticed [this 
announcement|https://news.apache.org/foundation/entry/the-apache-software-foundation-announces-gradle-as-a-platinum-targeted-sponsor]
 gradle becoming a platinum sponsor of the ASF.

In particular the announcement was about gradle offering the develocity service 
 to  the ASF. It so happens that I investigated develocity as part of my day 
job, and while we can't afford it there it is definitely something we might 
leverage here !

The tool is available at [https://ge.apache.org|https://ge.apache.org/]

I want to investigate the following leads

 
 # local caching for builds enable reuse of tasks output to speed up repeated 
tasks
 # build scans to provide detailed build reports and individual build metrics
 # build metrics are collected across all build scans and allow to target 
objectively bad contributors to overall build time
 # enabling the remote build cache
 **  additionally improves both local and CI build times
 ** requires some attention to read/write permissions to avoid cache poisoning 
and supply chain attacks
 ** most likely requires a dedicated build cache node request to infra
 ** several projects already use it so it is a known unknown ( we know it can 
be done we don't yet know how)
 # leverage advanced  test features of develocity unknown unknowns at least for 
me:D
 ## the tool collects tests metrics to pinpoint flaky tests enabling targeted 
fix efforts
 ## the tool claims to be able to do test avoidance and not rerun tests that 
are irrelevant to the changes
 ## the tool claims to be able to distribute test excecution accross dedicated 
test workers

Since we use maven,  we won't get the full benefits of the build cache ( gradle 
has been pushing plugin authors to write build cache friendly code for a long 
time ) but it should  improve the current situation quite a bit.

 

Apache pulsar is also using develocity with maven:
 - develocity configuration 
[https://github.com/apache/pulsar/blob/master/.mvn/gradle-enterprise.xml]
 - infra wiki has infos on how to onboard 
https://cwiki.apache.org/confluence/display/INFRA/Project+Onboarding+Instructions+for+Develocity

  was:
Hello,

A while ago I noticed [this 
announcement|https://news.apache.org/foundation/entry/the-apache-software-foundation-announces-gradle-as-a-platinum-targeted-sponsor]
 gradle becoming a platinum sponsor of the ASF.

In particular the announcement was about gradle offering the develocity service 
 to  the ASF. It so happens that I investigated develocity as part of my day 
job, and while we can't afford it there it is definitely something we might 
leverage here !

The tool is available at [https://ge.apache.org|https://ge.apache.org/]

I want to investigate the following leads

 
 # local caching for builds enable reuse of tasks output to speed up repeated 
tasks
 # build scans to provide detailed build reports and individual build metrics
 # build metrics are collected across all build scans and allow to target 
objectively bad contributors to overall build time
 # enabling the remote build cache
 **  additionally improves both local and CI build times
 ** requires some attention to read/write permissions to avoid cache poisoning 
and supply chain attacks
 ** most likely requires a dedicated build cache node request to infra
 ** several projects already use it so it is a known unknown ( we know it can 
be done we don't yet know how)
 # leverage advanced  test features of develocity unknown unknowns at least for 
me:D
 ## the tool collects tests metrics to pinpoint flaky tests enabling targeted 
fix efforts
 ## the tool claims to be able to do test avoidance and not rerun tests that 
are irrelevant to the changes
 ## the tool claims to be able to distribute test excecution accross dedicated 
test workers

Since we use maven,  we won't get the full benefits of the build cache ( gradle 
has been pushing plugin authors to write build cache friendly code for a long 
time ) but it should  improve the current situation quite a bit.

 

Apache pulsar is also using develocity with maven: 

- develocity configuratio 
[https://github.com/apache/pulsar/blob/master/.mvn/gradle-enterprise.xml]


> Investigate using develocity to improve james build
> ---
>
> Key: JAMES-3978
> URL: https://issues.apache.org/jira/browse/JAMES-3978
> Project: James Server
>  Issue Type: Improvement
>  Components: Build System
>    Reporter: Jean Helou
>Priority: Major
> Attachments: image-2024-02-02-16-30-20-026.png, 
> image-2024-02-02-16-31-24-883.png
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Hello,

[jira] [Commented] (JAMES-3978) Investigate using develocity to improve james build

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813731#comment-17813731
 ] 

Jean Helou commented on JAMES-3978:
---

benchmark for the first naive approach

 

mvn clean

build cache disabled `mvn package -pl :james-server-core -T1C -am` 

initial build 04:21 min https://ge.apache.org/s/6hubxpg4cnt3y
repeat build no file changes  03:34 min https://ge.apache.org/s/oqqk7t2tv6d7s

mvn clean
build cache enabled `mvn package -pl :james-server-core -T1C -am` 

initial build 04:00 min [https://ge.apache.org/s/gcqnpsbcspwd2]

repeat build 03:39 min  [https://ge.apache.org/s/jsnkbznbpewvu]

 

at this stage the build cache doesn't bring much gains. Actually this is normal 
because clean *must* be included in the lifecycle for the cache to store new 
entries. 

playing this again with `mvn clean package -pl :james-server-core -T1C -am`  
and caching enabled

initial build 04:14 min [https://ge.apache.org/s/3fk33vwguv2za]

repeat build 01:19 min  
[https://ge.apache.org/s/iha526ra7enfq|https://ge.apache.org/s/jsnkbznbpewvu]

the remaining time is from mojos for which the extension has no embedded 
configuration

 

develocity allows us to investigate the build performance to know which goals 
have not been cached 
!image-2024-02-02-16-30-20-026.png!

 

and list them easily

!image-2024-02-02-16-31-24-883.png!

 

 

as you can see a the scala compilation and style checking is not cached

> Investigate using develocity to improve james build
> ---
>
> Key: JAMES-3978
> URL: https://issues.apache.org/jira/browse/JAMES-3978
> Project: James Server
>  Issue Type: Improvement
>  Components: Build System
>    Reporter: Jean Helou
>Priority: Major
> Attachments: image-2024-02-02-16-30-20-026.png, 
> image-2024-02-02-16-31-24-883.png
>
>
> Hello,
> A while ago I noticed [this 
> announcement|https://news.apache.org/foundation/entry/the-apache-software-foundation-announces-gradle-as-a-platinum-targeted-sponsor]
>  gradle becoming a platinum sponsor of the ASF.
> In particular the announcement was about gradle offering the develocity 
> service  to  the ASF. It so happens that I investigated develocity as part of 
> my day job, and while we can't afford it there it is definitely something we 
> might leverage here !
> The tool is available at https://ge.apache.org
> I want to investigate the following leads
>  
>  # local caching for builds enable reuse of tasks output to speed up repeated 
> tasks
>  # build scans to provide detailed build reports and individual build metrics
>  # build metrics are collected across all build scans and allow to target 
> objectively bad contributors to overall build time
>  # enabling the remote build cache
>  **  additionally improves both local and CI build times
>  ** requires some attention to read/write permissions to avoid cache 
> poisoning and supply chain attacks
>  ** most likely requires a dedicated build cache node request to infra
>  ** several projects already use it so it is a known unknown ( we know it can 
> be done we don't yet know how)
>  # leverage advanced  test features of develocity unknown unknowns at least 
> for me:D
>  ## the tool collects tests metrics to pinpoint flaky tests enabling targeted 
> fix efforts
>  ## the tool claims to be able to do test avoidance and not rerun tests that 
> are irrelevant to the changes
>  ## the tool claims to be able to distribute test excecution accross 
> dedicated test workers
> Since we use maven,  we won't get the full benefits of the build cache ( 
> gradle has been pushing plugin authors to write build cache friendly code for 
> a long time ) but it should  improve the current situation quite a bit.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Updated] (JAMES-3978) Investigate using develocity to improve james build

[Jean Helou (Jira)]

 [ 
https://issues.apache.org/jira/browse/JAMES-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean Helou updated JAMES-3978:
--
Attachment: image-2024-02-02-16-31-24-883.png

> Investigate using develocity to improve james build
> ---
>
> Key: JAMES-3978
> URL: https://issues.apache.org/jira/browse/JAMES-3978
> Project: James Server
>  Issue Type: Improvement
>  Components: Build System
>    Reporter: Jean Helou
>Priority: Major
> Attachments: image-2024-02-02-16-30-20-026.png, 
> image-2024-02-02-16-31-24-883.png
>
>
> Hello,
> A while ago I noticed [this 
> announcement|https://news.apache.org/foundation/entry/the-apache-software-foundation-announces-gradle-as-a-platinum-targeted-sponsor]
>  gradle becoming a platinum sponsor of the ASF.
> In particular the announcement was about gradle offering the develocity 
> service  to  the ASF. It so happens that I investigated develocity as part of 
> my day job, and while we can't afford it there it is definitely something we 
> might leverage here !
> The tool is available at https://ge.apache.org
> I want to investigate the following leads
>  
>  # local caching for builds enable reuse of tasks output to speed up repeated 
> tasks
>  # build scans to provide detailed build reports and individual build metrics
>  # build metrics are collected across all build scans and allow to target 
> objectively bad contributors to overall build time
>  # enabling the remote build cache
>  **  additionally improves both local and CI build times
>  ** requires some attention to read/write permissions to avoid cache 
> poisoning and supply chain attacks
>  ** most likely requires a dedicated build cache node request to infra
>  ** several projects already use it so it is a known unknown ( we know it can 
> be done we don't yet know how)
>  # leverage advanced  test features of develocity unknown unknowns at least 
> for me:D
>  ## the tool collects tests metrics to pinpoint flaky tests enabling targeted 
> fix efforts
>  ## the tool claims to be able to do test avoidance and not rerun tests that 
> are irrelevant to the changes
>  ## the tool claims to be able to distribute test excecution accross 
> dedicated test workers
> Since we use maven,  we won't get the full benefits of the build cache ( 
> gradle has been pushing plugin authors to write build cache friendly code for 
> a long time ) but it should  improve the current situation quite a bit.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Updated] (JAMES-3978) Investigate using develocity to improve james build

[Jean Helou (Jira)]

 [ 
https://issues.apache.org/jira/browse/JAMES-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean Helou updated JAMES-3978:
--
Attachment: image-2024-02-02-16-30-20-026.png

> Investigate using develocity to improve james build
> ---
>
> Key: JAMES-3978
> URL: https://issues.apache.org/jira/browse/JAMES-3978
> Project: James Server
>  Issue Type: Improvement
>  Components: Build System
>    Reporter: Jean Helou
>Priority: Major
> Attachments: image-2024-02-02-16-30-20-026.png
>
>
> Hello,
> A while ago I noticed [this 
> announcement|https://news.apache.org/foundation/entry/the-apache-software-foundation-announces-gradle-as-a-platinum-targeted-sponsor]
>  gradle becoming a platinum sponsor of the ASF.
> In particular the announcement was about gradle offering the develocity 
> service  to  the ASF. It so happens that I investigated develocity as part of 
> my day job, and while we can't afford it there it is definitely something we 
> might leverage here !
> The tool is available at https://ge.apache.org
> I want to investigate the following leads
>  
>  # local caching for builds enable reuse of tasks output to speed up repeated 
> tasks
>  # build scans to provide detailed build reports and individual build metrics
>  # build metrics are collected across all build scans and allow to target 
> objectively bad contributors to overall build time
>  # enabling the remote build cache
>  **  additionally improves both local and CI build times
>  ** requires some attention to read/write permissions to avoid cache 
> poisoning and supply chain attacks
>  ** most likely requires a dedicated build cache node request to infra
>  ** several projects already use it so it is a known unknown ( we know it can 
> be done we don't yet know how)
>  # leverage advanced  test features of develocity unknown unknowns at least 
> for me:D
>  ## the tool collects tests metrics to pinpoint flaky tests enabling targeted 
> fix efforts
>  ## the tool claims to be able to do test avoidance and not rerun tests that 
> are irrelevant to the changes
>  ## the tool claims to be able to distribute test excecution accross 
> dedicated test workers
> Since we use maven,  we won't get the full benefits of the build cache ( 
> gradle has been pushing plugin authors to write build cache friendly code for 
> a long time ) but it should  improve the current situation quite a bit.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Created] (JAMES-3978) Investigate using develocity to improve james build

[Jean Helou (Jira)]

Jean Helou created JAMES-3978:
-

 Summary: Investigate using develocity to improve james build
 Key: JAMES-3978
 URL: https://issues.apache.org/jira/browse/JAMES-3978
 Project: James Server
  Issue Type: Improvement
  Components: Build System
Reporter: Jean Helou


Hello,

A while ago I noticed [this 
announcement|https://news.apache.org/foundation/entry/the-apache-software-foundation-announces-gradle-as-a-platinum-targeted-sponsor]
 gradle becoming a platinum sponsor of the ASF.

In particular the announcement was about gradle offering the develocity service 
 to  the ASF. It so happens that I investigated develocity as part of my day 
job, and while we can't afford it there it is definitely something we might 
leverage here !

The tool is available at https://ge.apache.org

I want to investigate the following leads

 
 # local caching for builds enable reuse of tasks output to speed up repeated 
tasks
 # build scans to provide detailed build reports and individual build metrics
 # build metrics are collected across all build scans and allow to target 
objectively bad contributors to overall build time
 # enabling the remote build cache
 **  additionally improves both local and CI build times
 ** requires some attention to read/write permissions to avoid cache poisoning 
and supply chain attacks
 ** most likely requires a dedicated build cache node request to infra
 ** several projects already use it so it is a known unknown ( we know it can 
be done we don't yet know how)
 # leverage advanced  test features of develocity unknown unknowns at least for 
me:D
 ## the tool collects tests metrics to pinpoint flaky tests enabling targeted 
fix efforts
 ## the tool claims to be able to do test avoidance and not rerun tests that 
are irrelevant to the changes
 ## the tool claims to be able to distribute test excecution accross dedicated 
test workers

Since we use maven,  we won't get the full benefits of the build cache ( gradle 
has been pushing plugin authors to write build cache friendly code for a long 
time ) but it should  improve the current situation quite a bit.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Call for vote: Apache James 3.7.5

[Jean Helou]

same +1 as for 3.8.1

Le mer. 10 janv. 2024 à 12:11, Karsten Otto
 a écrit :

> -1
>
> I propose to also fix JAMES-3968 in the next release, which is a
> critical mail loss bug. This way we can avoid another release
> immediately after.
>
> Cheers,
> Karsten
>
> On 09.01.24 9:15 AM, Benoit TELLIER wrote:
> > Hi,
> >
> > I would like to propose a new vote for 3.7.5 release of the Apache James
> > server.
> >
> > You can find:
> >
> >   - The maven release staged in repository.apache.org as the artifact
> > #1079:
> > https://repository.apache.org/content/repositories/orgapachejames-1079/
> >   - The changelog for 3.7.5:
> > https://github.com/apache/james-project/blob/master/CHANGELOG.md
> >
> > This release mostly ships bug fixes.
> >
> > Voting rules:
> >   - This is a majority approval: this release may not be vetoed.
> >   - A quorum of 3 binding votes is required
> >   - The vote starts at Tuesday 9th of January 2024, 9am UTC+2
> >   - The vote ends at Tuesday 16th of January 2024, 9am UTC+2
> >
> > You can answer to it just with +1 and -1. Down-votes may be motivated.
> >
> > 3 binding votes are expected move forward on this release.
> >
> > Cheers,
> >
> > Benoit TELLIER
> >
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Call for vote: Apache James 3.8.1

[Jean Helou]

then +1

Le jeu. 11 janv. 2024 à 16:21, Benoit TELLIER  a
écrit :

> And putting 1/2  for uploading new release materials ;-)
>
> That's OK I will find the time to do it.
>
> Best regards,
>
> Benoit
>
> On 11/01/2024 11:48, Rene Cordier wrote:
> > Hi jean,
> >
> > FYI Karsten already proposed fixes for that issue that have been
> > approved and will be part of the releases. It's just a matter now of
> > getting a green build from a rather a bit capricious CI^^'
> >
> > Regards,
> >
> > Rene.
> >
> > On 1/11/24 15:09, Jean Helou wrote:
> >> Much clearer thank you !
> >> What do you think about karsten's request ? i have no idea how
> >> painful it
> >> is to add that to the release ( i have a vague notion that the release
> >> process is quite complex). Though i don´t use rabbitmq I think a bug
> >> that
> >> results in hard mail loss is  severe enough to warrant a hotfix
> >>
> >> jean
> >>
> >> Le mer. 10 janv. 2024 à 16:44, btell...@linagora.com
> >> 
> >> a écrit :
> >>
> >>> I just merged the changelog updates, this should be better now.
> >>>
> >>> On 10/01/2024 11:26, Jean Helou wrote:
> >>>> hmm the linked changelog doesn't list any unreleased changes is that
> >>> normal
> >>>> ?
> >>>>
> >>>> Le mar. 9 janv. 2024 à 09:17, Benoit TELLIER  a
> >>> écrit :
> >>>>> Hi,
> >>>>>
> >>>>> I would like to propose a new vote for 3.8.1 release of the Apache
> >>>>> James
> >>>>> server.
> >>>>>
> >>>>> You can find:
> >>>>>
> >>>>> - The maven release staged in repository.apache.org as the
> >>>>> artifact
> >>>>> #1077:
> >>>>>
> https://repository.apache.org/content/repositories/orgapachejames-1077/
> >>>>>
> >>>>> - The changelog for 3.8.1:
> >>>>> https://github.com/apache/james-project/blob/master/CHANGELOG.md
> >>>>>
> >>>>> This release mostly ships bug fixes.
> >>>>>
> >>>>> Voting rules:
> >>>>> - This is a majority approval: this release may not be vetoed.
> >>>>> - A quorum of 3 binding votes is required
> >>>>> - The vote starts at Tuesday 9th of January 2024, 9am UTC+2
> >>>>> - The vote ends at Tuesday 16th of January 2024, 9am UTC+2
> >>>>>
> >>>>> You can answer to it just with +1 and -1. Down-votes may be
> >>>>> motivated.
> >>>>>
> >>>>> 3 binding votes are expected move forward on this release.
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Benoit TELLIER
> >>>>>
> >>>>>
> >>>>> -
> >>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> >>>>> For additional commands, e-mail: server-dev-h...@james.apache.org
> >>>>>
> >>>>>
> >>> -
> >>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> >>> For additional commands, e-mail: server-dev-h...@james.apache.org
> >>>
> >>>
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Call for vote: Apache James 3.8.1

[Jean Helou]

Much clearer thank you !
What do you think about karsten's request ? i have no idea how painful it
is to add that to the release ( i have a vague notion that the release
process is quite complex). Though i don´t use rabbitmq I think a bug that
results in hard mail loss is  severe enough to warrant a hotfix

jean

Le mer. 10 janv. 2024 à 16:44, btell...@linagora.com 
a écrit :

> I just merged the changelog updates, this should be better now.
>
> On 10/01/2024 11:26, Jean Helou wrote:
> > hmm the linked changelog doesn't list any unreleased changes is that
> normal
> > ?
> >
> > Le mar. 9 janv. 2024 à 09:17, Benoit TELLIER  a
> écrit :
> >
> >> Hi,
> >>
> >> I would like to propose a new vote for 3.8.1 release of the Apache James
> >> server.
> >>
> >> You can find:
> >>
> >>- The maven release staged in repository.apache.org as the artifact
> >> #1077:
> >> https://repository.apache.org/content/repositories/orgapachejames-1077/
> >>- The changelog for 3.8.1:
> >> https://github.com/apache/james-project/blob/master/CHANGELOG.md
> >>
> >> This release mostly ships bug fixes.
> >>
> >> Voting rules:
> >>- This is a majority approval: this release may not be vetoed.
> >>- A quorum of 3 binding votes is required
> >>- The vote starts at Tuesday 9th of January 2024, 9am UTC+2
> >>- The vote ends at Tuesday 16th of January 2024, 9am UTC+2
> >>
> >> You can answer to it just with +1 and -1. Down-votes may be motivated.
> >>
> >> 3 binding votes are expected move forward on this release.
> >>
> >> Cheers,
> >>
> >> Benoit TELLIER
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> >> For additional commands, e-mail: server-dev-h...@james.apache.org
> >>
> >>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Call for vote: Apache James MIME4J 0.8.10

[Jean Helou]

+1

Le mar. 9 janv. 2024 à 03:50, Quan tran hong 
a écrit :

> +1
>
> Cheers,
>
> Quan
>
> Vào Th 2, 8 thg 1, 2024 vào lúc 21:36 Benoit TELLIER <
> btell...@apache.org>
> đã viết:
>
> > Hi,
> >
> > I would like to propose a new vote for 0.8.10 release of the Apache
> > MIME4J library.
> >
> > You can find:
> >
> >   - The maven release staged in repository.apache.org as the artifact
> > #1076:
> > https://repository.apache.org/content/repositories/orgapachejames-1076/
> >   - The changelog:
> > https://github.com/chibenwa/james-mime4j/blob/master/CHANGELOG.md
> >
> > This release ships mostly small enhancements.
> >
> > Voting rules:
> >   - This is a majority approval: this release may not be vetoed.
> >   - A quorum of 3 binding votes is required
> >   - The vote starts at Monday 8th of January 2024, 3pm UTC+2
> >   - The vote ends at Monday 15th of January 2024, 3pm UTC+2
> >
> > You can answer to it just with +1 and -1. Down-votes may be motivated.
> >
> > 3 binding votes are expected move forward on this release.
> >
> > Cheers,
> >
> > Benoit TELLIER
> >
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>

Re: Call for vote: Apache James 3.8.1

[Jean Helou]

hmm the linked changelog doesn't list any unreleased changes is that normal
?

Le mar. 9 janv. 2024 à 09:17, Benoit TELLIER  a écrit :

> Hi,
>
> I would like to propose a new vote for 3.8.1 release of the Apache James
> server.
>
> You can find:
>
>   - The maven release staged in repository.apache.org as the artifact
> #1077:
> https://repository.apache.org/content/repositories/orgapachejames-1077/
>   - The changelog for 3.8.1:
> https://github.com/apache/james-project/blob/master/CHANGELOG.md
>
> This release mostly ships bug fixes.
>
> Voting rules:
>   - This is a majority approval: this release may not be vetoed.
>   - A quorum of 3 binding votes is required
>   - The vote starts at Tuesday 9th of January 2024, 9am UTC+2
>   - The vote ends at Tuesday 16th of January 2024, 9am UTC+2
>
> You can answer to it just with +1 and -1. Down-votes may be motivated.
>
> 3 binding votes are expected move forward on this release.
>
> Cheers,
>
> Benoit TELLIER
>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Proposal about deprecation and removal

[Jean Helou]

looks like my reploy below never reached the list

Le lun. 27 nov. 2023 à 15:04, Jean Helou  a écrit :

> I'm all for it as long as we give a fighting chance to users :)
>
> I see 3 different user population with regard to spring support
>
> - basic users who want an pre-assembled email server to run.
> They don't care about the dependency injection engine as long as the app
> starts, honors their existing configuration (or there is a clear migration
> guide to adapt the configuration) and there is a drop-in assembly they can
> use instead of the spring one.
> I'm pretty sure this already exists but confirmation would be nice.
>
> - more advanced users who inject custom code through the extension
> mechanisms.
> Those would/could be affected by the dependency engine change. For then we
> need a longer deprecation warning and maybe even start breaking edge case
> features for them to notice and take action.
> I can imagine a few aggressive ways to make people aware of the
> deprecation from a log at error level to that log repeated 15 times so it
> has a better chance to trigger any prod monitors to throwing an exception
> on startup that reports the deprecation notice and  includes workaround
> instructions.
>
> - Most advanced users building from source with custom extension
> Hopefully they are already using guice
>
>
> Jean
>
>
> Le ven. 24 nov. 2023 à 17:31, Matthieu Baechler <
> matth...@baechler-craftsmanship.fr> a écrit :
>
>> Hi,
>>
>> I had a hack session with Benoit today and the sentence "this would break
>> Spring" came many times along the day.
>>
>> As I'm less active on James than I used to be, I must admin I have no
>> idea how popular the Spring version of James is nowadays.
>>
>> However, what strikes me when I hack on James is how the size of the
>> project and its legacy makes it so slow to make progress.
>>
>> We did some deprecation and removal in the past but we have been
>> conservative about that.
>>
>> I would like to argue that being conservative to preserve existing users
>> may actually prevents from attracting new ones. Moreover, it probably also
>> prevents new developers to involve as they are quickly overwhelmed.
>>
>> So, what would you think about removing more aggressively features and
>> modules, starting with the Spring support?
>>
>> Cheers,
>>
>> -- Matthieu Baechler
>
>

OpenJPA

[Jean Helou]

Hello,

I have been following the discussion on pg mailbox but every jpa mention
makes me think of how bad I found the developer experience with open jpa
when playing with scaling-smtp.
(having to set the agent manually for tests in the ide is a pain. having to
list the classes to instrument isn't much better and we had quite a few
issues when moving scaling-smtp away from Cassandra and to PgSQL)

So I'm forking the initial discussion to ask if there would be interest in
migrating away from openjpa ?  at least for new features.

I have basically come to dislike most automatic ORMs but pure jdbc is very
low level. I thus tend to favor wrappers that make my life easier without
trying to hide the SQL. My current favorite wrapper is jooq (
https://www.jooq.org/).

I am not sure if it would be an acceptable choice for James. While there is
an open source version which uses apache 2, access to proprietary databases
relies on proprietary code with a paying license.

Apart from this I'm sure jooq would be nice especially if we are going for
a PG only solution.

Jooq DSL mirrors SQL quite closely to provide typesafety. it still allows
writing SQL explicitly if the DSL is not enough.

It returns a resultset in a thin wrapper that can be generated from the DDL
to allow typed access to the result rowsm items.
This then allows using a user provided mapper to build applicative objects.

I tried searching the archives of the mailing list for references to jooq
and found nothing but I may have missed things.

The downside of migrating to jooq is that I'm not sure how easy it is to
swap db backend with it and  support for Oracle/DB2/commercial databases
would require to buy a license.
As I said I don´t know how much of a deal breaker that is

jean

Le ven. 6 oct. 2023 à 23:49, Benoit TELLIER  a
écrit :

> Hey there!
>
> The goal: deliver James "stateless email server" concept to smaller
> deployments than those addressable with the Distributed server.
>
> Why Postgres? Rock solid. And more options than other SQL stores (see
> below)
>
> The requirements would be:
>   - Leverage the blobStore for binary storage (email bodies +
> attachements). Those big binaries are not meant to be stored into SQL rows
> - blaming you, JPA!
>   - Bring choice on blob store : PGSQL native solution (
> https://www.postgresql.org/docs/7.4/jdbc-binary-data.html ) for small
> deployments OR S3
>   - Bring choice on search: PGSQL native solution (
> https://www.postgresql.org/docs/current/textsearch.html ) for small
> deployments OR OpenSearch
>   - Bring choice on PubSub: PGSQL native solution (
> https://www.postgresql.org/docs/current/plpgsql-trigger.html  ) OR
> RabbitMQ
>   - Enforce strict tenant isolation: domain A won't access domain B data
> even if we screw up James access control layer. This can be done with Row
> security https://www.postgresql.org/docs/current/ddl-rowsecurity.html .
>   - Be reactive. This can be achieved by using a reactive firendly driver
> like r2dbc...
>   - Ensure that we can easily run on some largely scaling postgres...
> CitusData ?
>
> An other outcome might be to drop JPA implementation, ideally... (we
> provide something similar but wy better)
>
> Ideally I would like to deliver this before september 2024...
>
> Thoughts?
> Would this be something interesting people in here?
> Would some people be interested contributing to this effort?
> Would some people desire sponsoring this effort?
>
> If this is non consensual, I can also contribute this into
> https://github.com/linagora/tmail-backend/ without annoying people in
> here...
>
>
>
> --
>
> Best regards,
>
> Benoit TELLIER
>
> General manager of Linagora VIETNAM.
> Product owner for Team-Mail product.
> Chairman of the Apache James project.
>
> Mail: btell...@linagora.com
> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>
>
>
>
>

Re: Building a PostgreSQL mailbox for James?

[Jean Helou]

I somehow messed up my email config and the following message never reached
the list (but never got a nack either so it took me a week and 2 emails
without response to detect the problem)



Hello Benoit

I think having a postgresql backed mailbox server sounds great 

It should not come as a surprise since Matthieu and I already play with
deploying the pulsar+postgresql+s3 server for SMTP
Mailbox is a next step once we get SMTP up and running with a reasonable
level of stability. But at our place it will take some time :)

So I (we) would definitely contribute, not very fast as our free time
dictates but still contribute.

Le ven. 6 oct. 2023 à 23:49, Benoit TELLIER  a
écrit :

> Hey there!
>
> The goal: deliver James "stateless email server" concept to smaller
> deployments than those addressable with the Distributed server.
>
> Why Postgres? Rock solid. And more options than other SQL stores (see
> below)
>
> The requirements would be:
>   - Leverage the blobStore for binary storage (email bodies +
> attachements). Those big binaries are not meant to be stored into SQL rows
> - blaming you, JPA!
>   - Bring choice on blob store : PGSQL native solution (
> https://www.postgresql.org/docs/7.4/jdbc-binary-data.html ) for small
> deployments OR S3
>   - Bring choice on search: PGSQL native solution (
> https://www.postgresql.org/docs/current/textsearch.html ) for small
> deployments OR OpenSearch
>   - Bring choice on PubSub: PGSQL native solution (
> https://www.postgresql.org/docs/current/plpgsql-trigger.html  ) OR
> RabbitMQ
>   - Enforce strict tenant isolation: domain A won't access domain B data
> even if we screw up James access control layer. This can be done with Row
> security https://www.postgresql.org/docs/current/ddl-rowsecurity.html .
>   - Be reactive. This can be achieved by using a reactive firendly driver
> like r2dbc...
>   - Ensure that we can easily run on some largely scaling postgres...
> CitusData ?
>
> An other outcome might be to drop JPA implementation, ideally... (we
> provide something similar but wy better)
>
> Ideally I would like to deliver this before september 2024...
>
> Thoughts?
> Would this be something interesting people in here?
> Would some people be interested contributing to this effort?
> Would some people desire sponsoring this effort?
>
> If this is non consensual, I can also contribute this into
> https://github.com/linagora/tmail-backend/ without annoying people in
> here...
>
>
>
> --
>
> Best regards,
>
> Benoit TELLIER
>
> General manager of Linagora VIETNAM.
> Product owner for Team-Mail product.
> Chairman of the Apache James project.
>
> Mail: btell...@linagora.com
> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal)
>
>
>
>
>

[jira] [Created] (JAMES-3877) Allow to configure jpa pool max connections in juice module

[Jean Helou (Jira)]

Jean Helou created JAMES-3877:
-

 Summary: Allow to configure jpa pool max connections in juice 
module
 Key: JAMES-3877
 URL: https://issues.apache.org/jira/browse/JAMES-3877
 Project: James Server
  Issue Type: Improvement
  Components: jpa
Reporter: Jean Helou


When using the juice JPA module, the configuration is read from 
`james-database.properties` by the JPAConfiguration class. In doing so it 
creates `openjpa.*` properties which are then consumed by the entity manager 
factory.

Among these properties `openjpa.ConnectionFactoryProperties` is used to 
configure the underlying apache dbcp connection pool.

While it is already aware of some properties, it doesn't allow for configuring 
the `MaxTotal` which is at 8 by default (my quota is 5 :) )

The list and use of all the properties  is documented at 
[https://commons.apache.org/proper/commons-dbcp/configuration.html 
|https://commons.apache.org/proper/commons-dbcp/configuration.html]

Adding parsing for more properties is pretty easy and could be a nice entry 
point to beginner contributors even though I couldn't figure out how to run the 
tests within my IDE



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Resolved] (JAMES-3836) Proposal: Improved mail repository loading

[Jean Helou (Jira)]

 [ 
https://issues.apache.org/jira/browse/JAMES-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean Helou resolved JAMES-3836.
---
Resolution: Fixed

> Proposal: Improved mail repository loading
> --
>
> Key: JAMES-3836
> URL: https://issues.apache.org/jira/browse/JAMES-3836
> Project: James Server
>  Issue Type: Improvement
>    Reporter: Jean Helou
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> At the moment we have 2 approaches to instanciating mail repositories:
>  * constructor based instantiation with technical dependencies and the 
> MailRepositoryUrl as a constructor argument ( MemoryMailRepository, 
> CassandraMailRepository)
>  * constructor based instantiation with technical dependencies. The 
> MailRepositoryUrl is used in a PostConstruct through Configurable to isolate 
> instances from one another (FileMailRepository, JDBCMailRepository, 
> JPAMailRepository)
>  
> The former instantiation technique requires hacking around the guice loader 
> in order to create a subcontext for the correct url when loading the mail 
> repository instance ( see 
> org.apache.james.modules.mailrepository.guice.GuiceMailRepositoryLoader#load )
> The second technique creates an invalid instance which becomes valid only 
> once the configure method has been called.
> We were faced with this when building BlobMailRepository and tried to find a 
> better way. We propose a factory based approach where factories capture the 
> technical requirements of the mail repository implementation, are bound to 
> the general context  using a @ProvideIntoSet The loader then selects the 
> appropriate factory and uses it to build the requested instance.
> This requires no custom hacking of the guice context while allowing for pure 
> constructor based instantiation.
>  
> We propose a corresponding pull request which 
>  * implements and demonstrates the new loading mechanism
>  * provides factories and bindings all used repository implementations 
> (memory, blob, cassandra, jpa) 
> Creating the corresponding factories for the remaining implementations should 
> be fairly straightforward.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Created] (JAMES-3836) Proposal: Improved mail repository loading

[Jean Helou (Jira)]

Jean Helou created JAMES-3836:
-

 Summary: Proposal: Improved mail repository loading
 Key: JAMES-3836
 URL: https://issues.apache.org/jira/browse/JAMES-3836
 Project: James Server
  Issue Type: Improvement
Reporter: Jean Helou


At the moment we have 2 approaches to instanciating mail repositories:
 * constructor based instantiation with technical dependencies and the 
MailRepositoryUrl as a constructor argument ( MemoryMailRepository, 
CassandraMailRepository)
 * constructor based instantiation with technical dependencies. The 
MailRepositoryUrl is used in a PostConstruct through Configurable to isolate 
instances from one another (FileMailRepository, JDBCMailRepository, 
JPAMailRepository)

 

The former instantiation technique requires hacking around the guice loader in 
order to create a subcontext for the correct url when loading the mail 
repository instance ( see 
org.apache.james.modules.mailrepository.guice.GuiceMailRepositoryLoader#load )

The second technique creates an invalid instance which becomes valid only once 
the configure method has been called.

We were faced with this when building BlobMailRepository and tried to find a 
better way. We propose a factory based approach where factories capture the 
technical requirements of the mail repository implementation, are bound to the 
general context  using a @ProvideIntoSet The loader then selects the 
appropriate factory and uses it to build the requested instance.

This requires no custom hacking of the guice context while allowing for pure 
constructor based instantiation.

 

We propose a corresponding pull request which 
 * implements and demonstrates the new loading mechanism
 * provides factories and bindings all used repository implementations (memory, 
blob, cassandra, jpa) 

Creating the corresponding factories for the remaining implementations should 
be fairly straightforward.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Commented] (JAMES-3793) OOM when loading a very large object from S3?

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574924#comment-17574924
 ] 

Jean Helou commented on JAMES-3793:
---

Regarding the second part of the issue, it would be nice to  crash the app in a 
more controlled way by detecting oversized arrays and maybe providing 
diagnostic information before shutting down the process. I find it suprising 
that reactor doesn't kill the jvm on such failures. Akka does if properly 
configured.

Do you think there could be a way to derive a limit from the heap settings ?

> OOM when loading a very large object from S3?
> -
>
> Key: JAMES-3793
> URL: https://issues.apache.org/jira/browse/JAMES-3793
> Project: James Server
>  Issue Type: Bug
>Reporter: Benoit Tellier
>Priority: Major
>
> h2. What?
> We encountered recurring OutOfMemory exception on one of our production 
> deployment.
> Memory dump analysis was unconclusive and this tends to disqualify an 
> explanation based on a memory leak (300MB of objects only on the heap a few 
> minutes after the OOM).
> A careful log analysis lead to find what seems to be the "original OOM":
> {code:java}
> java.lang.OutOfMemoryError: Java heap space
> at java.base/java.util.Arrays.copyOf(Unknown Source)
> at software.amazon.awssdk.core.BytesWrapper.asByteArray(BytesWrapper.java:64)
> at 
> org.apache.james.blob.objectstorage.aws.S3BlobStoreDAO$$Lambda$4237/0x0008019f5ad8.apply(Unknown
>  Source)
> at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
> at 
> reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.run(MonoPublishOn.java:181)
> at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
> at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
> at 
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>  Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown 
> Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> {code}
>  
> Following this OOM the application is in a zombie state, unresponsive, 
> throwing OOMs without stacktraces, with Cassandra queries that never 
> finishes, unable to obtain a rabbitMQ connection and have issues within the 
> S3 driver... This sound like reactive programming limitations, that prevents 
> the java platform to handle the OOM like it should (crash the app, take a 
> dump, etc...)
> We did audit quickly our dateset and found several emails/attachment 
> exceeding the 100MB, with a partial and quick audit (we might very well have 
> some larger data!).
> Thus the current explanation is that somehow we successfully saved in S3 a 
> very big mail and now have OOMs when one tries to read it (as S3 blob store 
> DAO does defensive copies).
> h2. Possible actions
> This is an ongoing events, thus our understanding of it can evolve yet as it 
> raises interesting fixes that are hard to understand without the related 
> context, I decided to share it here anyway. I will report upcoming 
> developments here.
> Our first action is to confirm the current diagnostic:
>   - Further audit our datasets to find large items
>   - Deploy a patched version of James that rejects and log S3 objects larger 
> than 50MB
> Yet our current understanding leads to interesting questions...
> *Is it a good idea to load big objects from S3 into our memory?*
> As a preliminary answer, upon email reads we are using `byte[]` for 
> simplicity (no resource management, full view of the data). Changing this is 
> not the scope of this ticket at this is likely a major rework with many 
> unthought impacts. (I dont want to open that Pandora box...)
> SMTP, IMAP, JMAP, and the mailet container all have configuration preventing 
> sending/saving/receiving/uploading too big of a mail/attachment/blob, so we 
> likely have the convincing defense line at the protocol level. Yet this can 
> be defeated by either bad configuration (in our case JMAP was not checking 
> the size of sent email) history (rules were not the same in the past so 
> we ingested too big of a mail in the past), 'malicious action' (if all it 
> takes to crash james is to replace a 1 MB mail by a 1 GB mail). It thus 
> sounds interesting to me to have additional protection at the data access 
> layer, and be able to (optionally) configure S3 to not load objects of say 
> more than 50 MBs. This can be added wit

[jira] [Comment Edited] (JAMES-3793) OOM when loading a very large object from S3?

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574919#comment-17574919
 ] 

Jean Helou edited comment on JAMES-3793 at 8/3/22 8:46 PM:
---

+1 for a hard crash on non recoverable errors.

The Scala standard library provides an[ interesting definition of what a 
NonFatal 
error|http://example.com][https://github.com/scala/scala/blob/2.13.x/src/library/scala/util/control/NonFatal.scala]
 is, it definitely doesn't include OOM :)


was (Author: jeantil):
+1 for a hard crash on non recoverable errors.

The Scala standard library provides an[ interesting definition of what a 
NonFatal 
error|http://example.com]https://github.com/scala/scala/blob/2.13.x/src/library/scala/util/control/NonFatal.scala
 is (see scala.util.control.NonFatal) it definitely doesn't include OOM :)

> OOM when loading a very large object from S3?
> -
>
> Key: JAMES-3793
> URL: https://issues.apache.org/jira/browse/JAMES-3793
> Project: James Server
>  Issue Type: Bug
>Reporter: Benoit Tellier
>Priority: Major
>
> h2. What?
> We encountered recurring OutOfMemory exception on one of our production 
> deployment.
> Memory dump analysis was unconclusive and this tends to disqualify an 
> explanation based on a memory leak (300MB of objects only on the heap a few 
> minutes after the OOM).
> A careful log analysis lead to find what seems to be the "original OOM":
> {code:java}
> java.lang.OutOfMemoryError: Java heap space
> at java.base/java.util.Arrays.copyOf(Unknown Source)
> at software.amazon.awssdk.core.BytesWrapper.asByteArray(BytesWrapper.java:64)
> at 
> org.apache.james.blob.objectstorage.aws.S3BlobStoreDAO$$Lambda$4237/0x0008019f5ad8.apply(Unknown
>  Source)
> at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
> at 
> reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.run(MonoPublishOn.java:181)
> at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
> at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
> at 
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>  Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown 
> Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> {code}
>  
> Following this OOM the application is in a zombie state, unresponsive, 
> throwing OOMs without stacktraces, with Cassandra queries that never 
> finishes, unable to obtain a rabbitMQ connection and have issues within the 
> S3 driver... This sound like reactive programming limitations, that prevents 
> the java platform to handle the OOM like it should (crash the app, take a 
> dump, etc...)
> We did audit quickly our dateset and found several emails/attachment 
> exceeding the 100MB, with a partial and quick audit (we might very well have 
> some larger data!).
> Thus the current explanation is that somehow we successfully saved in S3 a 
> very big mail and now have OOMs when one tries to read it (as S3 blob store 
> DAO does defensive copies).
> h2. Possible actions
> This is an ongoing events, thus our understanding of it can evolve yet as it 
> raises interesting fixes that are hard to understand without the related 
> context, I decided to share it here anyway. I will report upcoming 
> developments here.
> Our first action is to confirm the current diagnostic:
>   - Further audit our datasets to find large items
>   - Deploy a patched version of James that rejects and log S3 objects larger 
> than 50MB
> Yet our current understanding leads to interesting questions...
> *Is it a good idea to load big objects from S3 into our memory?*
> As a preliminary answer, upon email reads we are using `byte[]` for 
> simplicity (no resource management, full view of the data). Changing this is 
> not the scope of this ticket at this is likely a major rework with many 
> unthought impacts. (I dont want to open that Pandora box...)
> SMTP, IMAP, JMAP, and the mailet container all have configuration preventing 
> sending/saving/receiving/uploading too big of a mail/attachment/blob, so we 
> likely have the convincing defense line at the protocol level. Yet this can 
> be defeated by either bad configuration (in our case JMAP was not checking 
> the size of sent email) history (rules were not the same in the past so 
> we ingested too big of a mail in the past), 'malicious action' (if all it 
> t

[jira] [Commented] (JAMES-3793) OOM when loading a very large object from S3?

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574919#comment-17574919
 ] 

Jean Helou commented on JAMES-3793:
---

+1 for a hard crash on non recoverable errors.

The Scala standard library provides an[ interesting definition of what a 
NonFatal 
error|http://example.com]https://github.com/scala/scala/blob/2.13.x/src/library/scala/util/control/NonFatal.scala
 is (see scala.util.control.NonFatal) it definitely doesn't include OOM :)

> OOM when loading a very large object from S3?
> -
>
> Key: JAMES-3793
> URL: https://issues.apache.org/jira/browse/JAMES-3793
> Project: James Server
>  Issue Type: Bug
>Reporter: Benoit Tellier
>Priority: Major
>
> h2. What?
> We encountered recurring OutOfMemory exception on one of our production 
> deployment.
> Memory dump analysis was unconclusive and this tends to disqualify an 
> explanation based on a memory leak (300MB of objects only on the heap a few 
> minutes after the OOM).
> A careful log analysis lead to find what seems to be the "original OOM":
> {code:java}
> java.lang.OutOfMemoryError: Java heap space
> at java.base/java.util.Arrays.copyOf(Unknown Source)
> at software.amazon.awssdk.core.BytesWrapper.asByteArray(BytesWrapper.java:64)
> at 
> org.apache.james.blob.objectstorage.aws.S3BlobStoreDAO$$Lambda$4237/0x0008019f5ad8.apply(Unknown
>  Source)
> at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
> at 
> reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.run(MonoPublishOn.java:181)
> at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
> at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
> at 
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>  Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown 
> Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> {code}
>  
> Following this OOM the application is in a zombie state, unresponsive, 
> throwing OOMs without stacktraces, with Cassandra queries that never 
> finishes, unable to obtain a rabbitMQ connection and have issues within the 
> S3 driver... This sound like reactive programming limitations, that prevents 
> the java platform to handle the OOM like it should (crash the app, take a 
> dump, etc...)
> We did audit quickly our dateset and found several emails/attachment 
> exceeding the 100MB, with a partial and quick audit (we might very well have 
> some larger data!).
> Thus the current explanation is that somehow we successfully saved in S3 a 
> very big mail and now have OOMs when one tries to read it (as S3 blob store 
> DAO does defensive copies).
> h2. Possible actions
> This is an ongoing events, thus our understanding of it can evolve yet as it 
> raises interesting fixes that are hard to understand without the related 
> context, I decided to share it here anyway. I will report upcoming 
> developments here.
> Our first action is to confirm the current diagnostic:
>   - Further audit our datasets to find large items
>   - Deploy a patched version of James that rejects and log S3 objects larger 
> than 50MB
> Yet our current understanding leads to interesting questions...
> *Is it a good idea to load big objects from S3 into our memory?*
> As a preliminary answer, upon email reads we are using `byte[]` for 
> simplicity (no resource management, full view of the data). Changing this is 
> not the scope of this ticket at this is likely a major rework with many 
> unthought impacts. (I dont want to open that Pandora box...)
> SMTP, IMAP, JMAP, and the mailet container all have configuration preventing 
> sending/saving/receiving/uploading too big of a mail/attachment/blob, so we 
> likely have the convincing defense line at the protocol level. Yet this can 
> be defeated by either bad configuration (in our case JMAP was not checking 
> the size of sent email) history (rules were not the same in the past so 
> we ingested too big of a mail in the past), 'malicious action' (if all it 
> takes to crash james is to replace a 1 MB mail by a 1 GB mail). It thus 
> sounds interesting to me to have additional protection at the data access 
> layer, and be able to (optionally) configure S3 to not load objects of say 
> more than 50 MBs. This can be added within the blob.properties file.
> Something like:
> {code:java}
> # Maximum s

Merging

[Jean Helou]

 Hello committers,

So I have a couple MRs approved CI is unhappy for now but I hope I can get
to a state where merging is ok at some point. I realize I am unsure what
the merge policy is. Looking at the repo history it seems to be rebase &
merge ( fully linearized history ) but I wonder how other committers
actually go about this ( especially with regards to signed commits )

Do you :
- click rebase and merge in github and use github driven signatures ?
- locally checkout,rebase,  forcepush to branch, merge then push to master
using local gpg sig ?

Thanks !
jean


ps: I checked
https://james.staged.apache.org/james-project/3.7.0/community/contributing.html#
but didn't see anything particularly relevant.

Re: ElasticSearch upgrade to 8.2

[Jean Helou]

2. Today we could add an extension mechanism exposing metrics and
> enabling exports.
>
> Concretely, takes a dropwizard MetricRegistry in, outputs metrics out.
>
> The downside is that we enforce the dropwizard choice onto out users,
> thus making it harder to change this component when required.
>

I'm not sure I understand the enforcing thing.

The metrics-api only has 3 implementations
- test which is not used for prod
- logs not sure how usable that is in production
- dropwizard metrics

when we have another implementation we can offer choice all I am aiming for
is an alternative to metrics-es-reporter-v7
I clearly don't understand the james extension mechanism :D


> > I would first attempt to implement the graphite/collectd reporter
> binding,
> > then look into opentelemetry since the former is supposed to be less work
> > than the latter.
> A lot less actually
> https://metrics.dropwizard.io/4.2.0/manual/collectd.html //
> https://metrics.dropwizard.io/3.1.0/manual/graphite/ :-)
>
> It should be trivial.


well my attempt at cloning metrics-es-reporter-v7 into a
metrics-influxdb-reporter didn't prove all that trivial but that may be
linked to both my lack of understanding of james and/or a particularly poor
reporter implementation for influxdb ( it was 3rd party not part of the
metrics core as graphite and collectd are)


> The hardest would be to keep the classpath of non
> collectd users clean, which could be achieved via an extension mechanism
> located into
> https://github.com/apache/james-project/tree/master/third-party. You
> would drop the JAR into extension-jars folder, register your metric
> reporter FQDN into extensions.properties and go. Even user supplied Jars
> would be supported.
>
 yeh that's something I don´t understand. This is not how the es reporter
is implemented, instead it is statically bound at compile time in the guice
container configuration
again I must be missing something
jean

Re: MailstoreApi (was Re: ElasticSearch upgrade to 8.2)

[Jean Helou]

On Fri, Jun 10, 2022 at 11:28 AM Benoit TELLIER  wrote:

> By the way, https://github.com/apache/james-project/pull/1046
>
> There is zero isolation between blob based mail repositories.
>
> /var/mail/error are mixed in with (eg) /var/mail/spam not allowing
> treatment based on the mail repositories the email was stored in
>
> I have commented on the PR, I feel the lack of isolation comes from an
invalid setup in the blob test. If you use the same approach as the one
used for the cassandra implementation you won't have any issue.


> I think a fix would be to infer the bucket used for storage, based on
> the mail repository URL that could be injected upon creation by the
> guice stack (just like Cassandra mail repository).
>
> Would you agree fixing this?
>

This is something we became aware of while implementing the guice module,
we are going to address it during our next session, see attached picture of
our wip marker :D

Regards,
Jean


> Regards,
>
> Benoit
>
> On 10/06/2022 15:53, Benoit TELLIER wrote:
> > Hello Jean,
> >
> > Answers inlined.
> >
> > On 10/06/2022 15:22, Jean Helou wrote:
> >> I fork the thread to respond on the MailRepository part :D
> >>
> >>
> >> > fun quiz can you tell looking
> >> > only at the documentation and code comments the difference
> >> between :
> >> > MailRepositoryStore, MailRepositoryUrlStore and MailRepository
> >> all of which
> >> > are in mailrepository-api  )
> >> Game accepted.
> >>   - Mail repository is storage for email, with their processing
> >> context
> >> (long term storage, differs from mail queue which is a flow).
> >>   - Mail repository are identified by their URL
> >>   - Mail repository can be created through the use of mail
> >> repository
> >> store by supplying an URL
> >>   - MailRepositoryUrlStore is an implementation detail of
> >> MailRepositoryStore, and brings persistance to mail repositories
> >> (that
> >> are created through webadmin, configuration changes etc..)
> >>
> >>
> >> Almost :D
> >> MailRepositoryStore only has 2 implementations :  in memory and a
> >> spring based
> >>
> >> If I understand it correctly the MailRepositoryStore is actually a
> >> computing cache.
> >> Roughly  equivalent to Map with a
> >> generic factory method to create the MailRepository when it is not
> >> already in the cache.
> > Yes.
> >> The factory method relies on a statically injected config that maps
> >> "protocols" to the FQDN of the corresponding implementation. When the
> >> "store" resolves a MailRepository through its MailRepositoryUrl, it
> >> retrieves the FQDN from the protocol part of the url then delegates
> >> to spring, guice or a static map to actually get the corresponding
> >> implementation.
> >> The naming Store and InMemory got me mightily confused when I tried
> >> to sort this out to inject the blob mail repository store.
> >>
> >> MailRepositoryUrls start with a protocol such as cassandra:// blob://
> >> file://, then a repository "id"
> >>
> >> The MailRepositoryUrlStore has 3 implementations : cassandra, jpa and
> >> inmemory. I am not yet clear on what having a persistent store brings
> >> over using the in memory store.
> >>
> >> Now to have a MailRepositoryStore not based on Cassandra, the memory
> >> implementation is good enough if manual creation of mail
> >> repository is
> >> forbidden (akka through webadmin) and if configuration is
> >> homogeneous in
> >> the James cluster.
> >>
> >>
> >> Even if you were to create a mail repository manually, I don't
> >> understand how anything would be stored in it if it is not mentioned
> >> in the server's configuration (mailet container's config most likely).
> > One case is "it was mentioned in the server configuration, and no
> > longer is".
> >
> > Without such persistence you could not, for instance, reprocess mail
> > repositories that you had been using.
> >
> > --
> >
> > An other case is "parametric mail repository" ie
> > cassandra://var/mail/customera.com/rejected
> >
> > One such exemple is Data Leak Prevention cf
> >
> https://github.com/apache/james-project/tree/master/server/mailet/mailets/src/main/java/org/

Re: OpenTelemetry (was Re: ElasticSearch upgrade to 8.2)

[Jean Helou]

hi,

I did not read much about the topic, but I got the impression that using
> OpenTelemetry requires to use OpenTelemetry server and integrations
> happens in there.
>
>
> https://grafana.com/blog/2022/05/10/how-to-collect-prometheus-metrics-with-the-opentelemetry-collector-and-grafana/
>
>
This do not sounds like a friendly SPI, and I would be personally
> reluctant to enforce usage of an external service in such a core feature...
>


> Am I missing something?
>

The article you refer to is about how to configure the opentelemetry
collector, that's  an independant proxy process that can collect metrics
using multiple protocols, process them and push them using multiple
protocols, it's kind of like a logstash on steroids. or as if the push
gateway and "retrieval" components of a prometheus install had been
extracted and merged in a single tool.
what would end up in james is the opentelemetry library
https://opentelemetry.io/docs/instrumentation/java/ which if I understood
the documentation properly is able to export using 4 fiarily common modern
protocols (oltp based on grpc, prometheus, zipkin and jaeger) based on
configuration only.
If you are worried by the "alpha status" listed for metrics in the java lib
documentation, you may want to read this article :
https://opentelemetry.io/blog/2022/metrics-announcement/ the metrics
components should reach general availabitliy soon.

Jean

> Regards
>
> >
> > Cheers
> > jean
> >
> > PS while writing this message I realized that 3.7.0 is not listed on
> > https://james.apache.org/server/release-notes.html despite having a blog
> > announcement
> > https://james.apache.org/james/update/2022/03/21/james-3.7.0.html
> >
> > [1] https://github.com/logfellow/logstash-logback-encoder
> > [2]
> >
> https://www.elastic.co/guide/en/observability/master/monitor-java-app.html
> > [4] https://james.apache.org/server/monitor.html
> > [5]
> >
> https://james.apache.org/server/manage-guice-distributed-james.html#Basic_Monitoring
> > [6] https://metrics.dropwizard.io/4.2.0/manual/graphite.html
> > [7] https://hub.docker.com/r/graphiteapp/graphite-statsd/
> > [8] https://opentelemetry.io/
> > [9]
> >
> https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#exporters
> >
> > On Thu, Jun 9, 2022 at 9:21 AM Rene Cordier  wrote:
> >
> >> I would like to add that it seems this module has been deprecated when
> >> James migrated from es6 to es7, as stated in the documentation of the
> >> configuration here for example:
> >>
> >>
> https://github.com/apache/james-project/blob/master/src/site/xdoc/server/config-elasticsearch.xml#L137
> >>
> >> For likely all the reasons given below.
> >>
> >> So it makes sense to remove it now that we are migrating to es8.
> >>
> >> Regards,
> >> Rene.
> >>
> >> On 09/06/2022 14:13, Benoit TELLIER wrote:
> >>> +1
> >>>
> >>> We IMO need to add that direct reporting (for logs or metrics) from the
> >>> application to ElasticSearch is an anti-pattern, and pretty uncommon.
> >>>
> >>> It's usage led, at least at Linagora, to metrics/logs loss. The code
> >>> quality of these extensions was low and proved several time
> problematic.
> >>>
> >>> Alternatives:
> >>>- Anything compatible with prometheus (pretty much all of metrics
> >>> ecosystem)
> >>>- Things like fluentbit to scrap logs from the container/a file and
> >>> move it to your analytic stack
> >>>
> >>> Regards,
> >>>
> >>> Benoit
> >>>
> >>> On 09/06/2022 11:02, Rene Cordier wrote:
>  Hello guys,
> 
>  As working on the upgrade to ElasticSearch 8.2, I've been wondering if
>  it was still worth it to keep the elasticsearch metrics reporter
> module.
> 
>  Since we expose a metrics endpoint that tools like prometheus can just
>  call to scrap metrics out of James, I think the es metrics reporter
>  has not been really maintained and/or used since es v6.
> 
>  As such, we decided to remove it. It allows to reduce the number of
>  modules loaded by James and also reduce the cost of maintenance.
> 
>  But maybe other people are still using it? If a part of the community
>  still want to use it with the switch to ES8, they can put it back and
>  migrate it, but we would appreciate that they can assume the
>  maintenance of it as well.
> 
>  Also removed the logback-elasticsearch-appender. It's not correlated
>  to our code and easy to add the JAR back on the classpath if needed.
> 
>  The PR in question: https://github.com/apache/james-project/pull/1018
> 
>  Don't hesitate to answer this thread if you have any concerns.
> 
>  Cheers,
>  Rene.
> 
>  -
>  To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>  For additional commands, e-mail: server-dev-h...@james.apache.org
> 
> 
> >>> 

Re: MailstoreApi (was Re: ElasticSearch upgrade to 8.2)

[Jean Helou]

Thanks a lot for the additional context, benoit

I have added that as a comment on the issue to make it more readily
discoverable. When we reach cleanup phase, I'll try to add some
documentation to the interfaces in the code.

Cheers
jean

> One case is "it was mentioned in the server configuration, and no longer
> is".
>

> Without such persistence you could not, for instance, reprocess mail
> repositories that you had been using.
>
> --
>
> An other case is "parametric mail repository" ie cassandra://var/mail/
> customera.com/rejected
>
> One such exemple is Data Leak Prevention cf
> https://github.com/apache/james-project/tree/master/server/mailet/mailets/src/main/java/org/apache/james/transport/matchers/dlp
>
> And his friend
> https://github.com/apache/james-project/blob/master/server/mailet/mailets/src/main/java/org/apache/james/transport/mailets/ToSenderDomainRepository.java
>
> I might want to access a mail repository that exist, contains stuff, but
> is not provisionned localy because the James server I am using did not yet
> reject an email for this domain since it had been started.
>
> --
>
> Another thing is the difference between mailrepository URL / path (which I
> am not a fan of)
>
> The idea was not to leak through webadmin the underlying storage structure
>
> URL: cassandra://var/mail/error
> PATH: var/mail/error
>
> Then you need to do translation between the path and the URL, which is not
> trivial in face of several underlying storage technologies (jdbc + file for
> example)
>
> Even if there is a way to dynamically make james store mails in a mail
> repository that is not mentioned in the configuration, the in memory
> implementation will still register it when it is used.  I guess that only
> leaves discoverability of existing MailRepositoryUrls across restarts when
> an Url is not used much. That leaves me wondering what the actual use case
> is ...
>
> Well the one time I had to deal with mail repository with a customer,
> listing them was handy.
>
> That being said, I also share the feeling that "listing URLs in use"
> through MailRepositoryUrlStore might be overkill.
>
> Instead we could rely on each MailRepository implementation to list the
> URLs it do actually contain, thus drop MailRepositoryUrlStore alltogether,
> make it an implementation detail.
>
> We would get :
>   - MailRepositoryUrlSupplier interface with an implementation for each
> MailRepository implementation.
>   - Implementations can base decisions on their underlying storage thus
> removing the needs for additional metadata.
>
> I would support such a refactoring. One less Cassandra table makes me
> happy ;-)
>
>
>
>> Ideally MailRepositoryUrlStore should not have had been in the API.
>>
>
> Interesting, according to git history it was introduced by
> https://issues.apache.org/jira/browse/JAMES-2418 but that only says that
> the last point I mention above ( the discoverability part)  is needed but
> not why it is needed :D
>
> I hope I did get better at writting issues since then :-P
>
>
> cheers
> jean
>
>
> Cheers
>
>

[jira] [Commented] (JAMES-2418) Store repository APIs that are being used

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552639#comment-17552639
 ] 

Jean Helou commented on JAMES-2418:
---

Additional context that popped up in the mailing list while I reported 
discovering the API :

 
{quote}One case is "it was mentioned in the server configuration, and no longer 
is". {quote}
{quote}
Without such persistence you could not, for instance, reprocess mail 
repositories that you had been using.

--

An other case is "parametric mail repository" ie 
cassandra://var/mail/[customera.com/rejected|http://customera.com/rejected]

One such exemple is Data Leak Prevention cf 
[https://github.com/apache/james-project/tree/master/server/mailet/mailets/src/main/java/org/apache/james/transport/matchers/dlp]

And his friend 
[https://github.com/apache/james-project/blob/master/server/mailet/mailets/src/main/java/org/apache/james/transport/mailets/ToSenderDomainRepository.java]

I might want to access a mail repository that exist, contains stuff, but is not 
provisionned localy because the James server I am using did not yet reject an 
email for this domain since it had been started.

--

Another thing is the difference between mailrepository URL / path (which I am 
not a fan of)

The idea was not to leak through webadmin the underlying storage structure

URL: cassandra://var/mail/error
PATH: var/mail/error

Then you need to do translation between the path and the URL, which is not 
trivial in face of several underlying storage technologies (jdbc + file for 
example)
{quote}Even if there is a way to dynamically make james store mails in a mail 
repository that is not mentioned in the configuration, the in memory 
implementation will still register it when it is used.  I guess that only 
leaves discoverability of existing MailRepositoryUrls across restarts when an 
Url is not used much. That leaves me wondering what the actual use case is 
...{quote}Well the one time I had to deal with mail repository with a customer, 
listing them was handy.

That being said, I also share the feeling that "listing URLs in use" through 
MailRepositoryUrlStore might be overkill.

Instead we could rely on each MailRepository implementation to list the URLs it 
do actually contain, thus drop MailRepositoryUrlStore alltogether, make it an 
implementation detail.

We would get :
  - MailRepositoryUrlSupplier interface with an implementation for each 
MailRepository implementation.
  - Implementations can base decisions on their underlying storage thus 
removing the needs for additional metadata.

I would support such a refactoring. One less Cassandra table makes me happy 
;-){quote}

> Store repository APIs that are being used
> -
>
> Key: JAMES-2418
> URL: https://issues.apache.org/jira/browse/JAMES-2418
> Project: James Server
>  Issue Type: Improvement
>  Components: MailStore  MailRepository
>Reporter: Benoit Tellier
>Priority: Major
>
> They are only stored in memory, thus rebooting James will loose all 
> dinamically generated mail repository (at the very least until they are 
> created again).
> To solve this major flow, we need a MailRepositoryUrlStore API (basically a 
> persistant Set) with generic contract tests and memory, cassandra, 
> jpa implementations.
> This should be used for MailRepositoryStore::getUrls operation as well as 
> lazyly get repository.
> Needs Guice and (ideally) Spring bindings



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: ElasticSearch upgrade to 8.2

[Jean Helou]

Hey benoit,

First I am happy to see this component raise such interest, and gets so
>
much attention!


For the record, I want to establish that I don't use nor actually like ES
for such use cases so the ES part doesn't really matter to me.
I do care a lot about metrics collection and I generally dislike being tied
to a single system.

However it's very easy to spend loads of time on technical issue that
> delivers little value to end users.
> My Prometheus deployments works well and I don't intend to invest more
> of the topic. I don't think I will beat you on that ;-)
>

Today you (linagora?) are happy with prometheus and the platform I target
supports it so I will be using it too.
What worries me is the total lack of choice, the difficulty to add said
choice in the current james ( and I did try) despite boasting a metrics
collection system that is supposed to have pluggable implementations.
I do consider that metrics collection brings a lot of value to end users,
and that being able to integrate in an existing metrics collection
infrastructure is an important point for james adoption beyond your
deployments :D
So yes, when we are done with the pulsar/blob repository thing I will
probably spend some time to either submit a couple modules for established
and stable protocols that won't require much maintenance beyond the
occasional version bump ( graphite and collectd reporters most likely since
they are 1st party modules of dropwizard metrics ) or to implement
opentelemetry in a way that doesn't require such internal bindings, brings
additional benefits in terms of features such as tracing and is more
actively maintained than dropwizard metrics.

Also, ELK stack is not used for metrics.
>

actually it is,  APMs and metrics ingestion have been part of the elastic
solution for quite a while not that I have actually used them


> > However, there are many monitoring systems based on push instead of pull.
> > The debate has been going for a long time and will probably be going on
> > forever ( google lists 54 million results for 'push vs pull monitoring
> > systems' ) it would be nice for james to support at least one of each
> model
> > even if pull is currently the favored approach.
> >
> > Prometheus is popular these days because it is deployed internally in
> > kubernetes which is at the top of the hype curve but james should
> probably
> > support at least one push based protocol
> You know we `could` support many things it do not mean that we should.
>
> We are a small community with limited means, the maintenance effort most
> of the time falls on the same people.
>
> Our efforts should be directed to things that are useful, not a
> deprecated metric module with alternatives that targets another Elastic
> Search version.
>

Note that I don't push to keep the es module, I push for having at least
one push based alternative to prometheus out of the box if only to
demonstrate/document how plugging a dropwizard metrics reporter in james
can be achieved if you don't use prometheus


> >   or at least document how to plug
> > one in for people who build their own assembly.
> +1 thanks for opening with consensus. (was about to do the same ;-) )
>
> I agree we could have an extension mechanism for metric exporters.
> Ideally underlying metric library should come up with an SPI so we, as
> application writers, don't need to care.\
>

I'm not going so far as to want an SPI for dropwizard metrics reporters, as
I said a couple integrations for the reporters provided by metrics itself
to document how to integrate such a reporter is just fine.
As I wrote above I will eventually contribute such an integration if it has
not been done before I get to it.


> Though, if there's a potential move to open telemetry, then adding our
> own custom extension mechanism is likely not the right thing.
>

I would first attempt to implement the graphite/collectd reporter binding,
then look into opentelemetry since the former is supposed to be less work
than the latter.


> CF
>
> https://github.com/apache/james-project/blob/master/server/apps/distributed-app/docs/modules/ROOT/pages/operate/metrics.adoc
>
> Upon recent documentation on the topic we likely forgot to update legacy
> documentation. That's a fail that will be fixed timely.
>

I have been unable to navigate to that document on the current website :(
I am not sure how an end user is supposed to discover this ...

cheers
jean

MailstoreApi (was Re: ElasticSearch upgrade to 8.2)

[Jean Helou]

I fork the thread to respond on the MailRepository part :D


> fun quiz can you tell looking
> > only at the documentation and code comments the difference between :
> > MailRepositoryStore, MailRepositoryUrlStore and MailRepository all of
> which
> > are in mailrepository-api  )
> Game accepted.
>   - Mail repository is storage for email, with their processing context
> (long term storage, differs from mail queue which is a flow).
>   - Mail repository are identified by their URL
>   - Mail repository can be created through the use of mail repository
> store by supplying an URL
>   - MailRepositoryUrlStore is an implementation detail of
> MailRepositoryStore, and brings persistance to mail repositories (that
> are created through webadmin, configuration changes etc..)
>

Almost :D
MailRepositoryStore only has 2 implementations :  in memory and a spring
based

If I understand it correctly the MailRepositoryStore is actually a
computing cache.
Roughly  equivalent to Map with a
generic factory method to create the MailRepository when it is not already
in the cache.
The factory method relies on a statically injected config that maps
"protocols" to the FQDN of the corresponding implementation. When the
"store" resolves a MailRepository through its MailRepositoryUrl, it
retrieves the FQDN from the protocol part of the url then delegates to
spring, guice or a static map to actually get the corresponding
implementation.
The naming Store and InMemory got me mightily confused when I tried to sort
this out to inject the blob mail repository store.

MailRepositoryUrls start with a protocol such as cassandra:// blob://
file://, then a repository "id"

The MailRepositoryUrlStore has 3 implementations : cassandra, jpa and
inmemory. I am not yet clear on what having a persistent store brings over
using the in memory store.

Now to have a MailRepositoryStore not based on Cassandra, the memory
> implementation is good enough if manual creation of mail repository is
> forbidden (akka through webadmin) and if configuration is homogeneous in
> the James cluster.
>

Even if you were to create a mail repository manually, I don't understand
how anything would be stored in it if it is not mentioned in the server's
configuration (mailet container's config most likely).
Even if there is a way to dynamically make james store mails in a mail
repository that is not mentioned in the configuration, the in memory
implementation will still register it when it is used.  I guess that only
leaves discoverability of existing MailRepositoryUrls across restarts when
an Url is not used much. That leaves me wondering what the actual use case
is ...


> Ideally MailRepositoryUrlStore should not have had been in the API.
>

Interesting, according to git history it was introduced by
https://issues.apache.org/jira/browse/JAMES-2418 but that only says that
the last point I mention above ( the discoverability part)  is needed but
not why it is needed :D

cheers
jean


Cheers

Re: ElasticSearch upgrade to 8.2

[Jean Helou]

Hello all,

>We IMO need to add that direct reporting (for logs or metrics) from the
application to ElasticSearch is an anti-pattern, and pretty uncommon.

Doesn't sound quite right :
Pushing logs from the app the ES is exactly what the whole ELK thing is all
about. Given the amount of literature that promotes it and the popularity
of logstash-logback-encoder [1], I would like to argue that this is not all
that uncommon :)
I'm less familiar with the metrics side of things and the current elastic
documentation [2] seems to mix both a prometheus like ingester and a jvm
agent that pushes apm metrics to ES which seems to offer a migration path
for people still using ES for metrics monitoring.
However, there are many monitoring systems based on push instead of pull.
The debate has been going for a long time and will probably be going on
forever ( google lists 54 million results for 'push vs pull monitoring
systems' ) it would be nice for james to support at least one of each model
even if pull is currently the favored approach.

Prometheus is popular these days because it is deployed internally in
kubernetes which is at the top of the hype curve but james should probably
support at least one push based protocol or at least document how to plug
one in for people who build their own assembly.
>I think the es metrics reporter has not been really maintained and/or used
since es v6.

Note that the ES module is still the *first* metrics monitoring mechanism
listed in the james documentation [3] and the *only* one listed for
distributed james[4]. That same documentation has no mention of the
deprecation nor of prometheus, there is lots of information about JMX and
folder monitoring (quite frankly I don't know how much of it is actually
still relevant ( especially with the removal of FileDir IIRC )).

Also note that
- there is no other options for push based monitoring model in the current
code base
- implementing such an alternative has proven not to be trivial ( we spent
a couple evenings with matthieu trying to write an influxdb reporter when
assembling the first pulsar based app, we *failed* and gave up by plugging
the ES reporter in the pulsar app)

If james drops the only existing push based dropwizard reporter
integration, I would very very much like to see documentation on how to
write and integrate another (maybe the metrics-graphite [6] could be used
as a sample since there is a readily available graphite-statsd docker image
[7] ?)  Matthieu and I can't currently revisit this subject, we are both
limited by our available free time and we are currently fully occupied
subduing the mailrepository subs system so it accepts using the blob store
based implementation in the pulsar assembly ( fun quiz can you tell looking
only at the documentation and code comments the difference between :
MailRepositoryStore, MailRepositoryUrlStore and MailRepository all of which
are in mailrepository-api  )

One of the things we discussed with Matthieu as a possible contrib after
the pulsar mailqueue based assembly was implementing metrics-api using
opentelemetry[8] and to drop the need for custom reporter integration in
favor of configuration based reporters[9]  ( don't hesitate to beat us to
it ! :p )

Cheers
jean

PS while writing this message I realized that 3.7.0 is not listed on
https://james.apache.org/server/release-notes.html despite having a blog
announcement
https://james.apache.org/james/update/2022/03/21/james-3.7.0.html

[1] https://github.com/logfellow/logstash-logback-encoder
[2]
https://www.elastic.co/guide/en/observability/master/monitor-java-app.html
[4] https://james.apache.org/server/monitor.html
[5]
https://james.apache.org/server/manage-guice-distributed-james.html#Basic_Monitoring
[6] https://metrics.dropwizard.io/4.2.0/manual/graphite.html
[7] https://hub.docker.com/r/graphiteapp/graphite-statsd/
[8] https://opentelemetry.io/
[9]
https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#exporters

On Thu, Jun 9, 2022 at 9:21 AM Rene Cordier  wrote:

> I would like to add that it seems this module has been deprecated when
> James migrated from es6 to es7, as stated in the documentation of the
> configuration here for example:
>
> https://github.com/apache/james-project/blob/master/src/site/xdoc/server/config-elasticsearch.xml#L137
>
> For likely all the reasons given below.
>
> So it makes sense to remove it now that we are migrating to es8.
>
> Regards,
> Rene.
>
> On 09/06/2022 14:13, Benoit TELLIER wrote:
> > +1
> >
> > We IMO need to add that direct reporting (for logs or metrics) from the
> > application to ElasticSearch is an anti-pattern, and pretty uncommon.
> >
> > It's usage led, at least at Linagora, to metrics/logs loss. The code
> > quality of these extensions was low and proved several time problematic.
> >
> > Alternatives:
> >   - Anything compatible with prometheus (pretty much all of metrics
> > ecosystem)
> >   - Things like fluentbit to scrap 

Re: "Branding" of pulsar-cassandra-smtp-relay

[Jean Helou]

Hello :)

Did you had a look at configuration based server-data implementations?
>

We considered it for a while, maybe we should revisit it.

They could fit your goals while minimizing storage costs.
>

indeed, but xml is scary ;)

Jean


> I'm thinking to XMLDomainList, XMLRacipientRewriteTable. We would likely
> miss a corresponding XML user repository but this should not be hard to
> implement if needed.
>

> For the artifact name, +1 for scaling-pulsar-smtp.
>
> Best regards,
>
> Benoit
>
> On 11/05/2022 04:50, Jean Helou wrote:
> > Hello Benoit !
> >
> > Matthieu and I are trying to create a server instance to host our
> family's
> > email :)
> > Being the geeks that we are, we are not satisfied with a single node JPA
> > deployment and want
> > to preserve the scalability and resilience properties of distributed
> james
> > while minimizing the
> > run cost of a small scale deployment so as to keep our hobby in a
> > reasonable[1] price range.
> >
> > Since a mail server is such a  complex piece of software, we chose to
> > concentrate our efforts
> > on a smaller scope first: SMTP processing only. Our initial deployment
> will
> > thus be SMTP
> > only, then we intend to look into building an IMAP/JMAP instance, a bit
> > like what's described in
> > "distributed james - specialized instances" [2]. At least that's what's
> > guiding our forays into james
> > codebase.
> >
> > We looked at various options and found a pulsar as a service that's
> fairly
> > cheap over at clever-cloud, they also have a reasonably cheap S3 clone.
> > This is what led to the initial pulsar dev as the cost of other
> > MQ saas offering was much higher.
> > That leaves the primary datastore, we initially targeted cassandra
> because
> > we knew that worked but that made the run cost explode.
> >
> > At the moment we will intend to start with a hosted postgresql instance
> > instead. If only to finally be able to see the pulsar code run without
> > ruining ourselves :)
> >
> > With this clarified, I'll try to answer your mail :
> >
> > The name makes it hard to grasp what goals are researched, we also miss
> >> a little statement/README about this new product.
> >>
> > The goals did change a couple times but it always wanted to be a
> "simpler",
> > SMTP-specialized instance than what could be achieved with the main
> > distributed server app.
> >
> > Here is a little explaination of the goal of this application I can
> propose:
> >> ```
> >> This artifact leverage Apache Pulsar to deliver a mailet container that
> >> scales your mail processing.
> >> It can be used both for inbound and outbound behavior customization.
> >> Thanks to Apache Pulsar, get your hands on a fully featured distributed
> >> mail queue to manage efficiently your email delivery.
> >> It targets minimal dependencies: solely Apache Pulsar and an object
> store.
> >> ```
> >>
> > If we reach a consensus I would be happy to write its README, including
> >> sample configuration and start instructions.
> >>
> > Minimal dependencies to this level would be great but we looked into it
> and
> > we don't expect to be able to minimize them that much, in the
> short/medium
> > term.
> >
> >
> >> I understand from recent tickets (JAMES-3761 JAMES-3762 and JAMES-3763)
> >> that it is intendeed to not get Cassandra as a dependency. Some features
> >> are optional and can easily be dropped (recipient rewritting) however
> >> some others would eventually need an alternative implementation (I am
> >> thinking to domains, users). What is the plan regarding this?
> >>
> > Offering an alternative to cassandra which is an expensive fit for
> smaller
> > scale deployments would indeed require an alternative implementation to:
> > - DomainList (5 methods)
> > - RecipientRewriteTable with it's regex mapping (26 methods)
> > - UserRepository (15 methods)
> >
> > I am not sure we can completely drop RecipientRewriteTable, I think we
> > looked into it with Matthieu and it wasn't as optional as it first
> looked.
> > I think maybe it had something to do with error handling which made it a
> > mandatory dependency for the mailet container.
> > Maybe we will try dropping it again in our next attempt at starting the
> > app...
> >
> > If we can't drop it, it means 46 behaviours to implement. That's quite a
> > lot the time we can allocate for our
> >

Re: "Branding" of pulsar-cassandra-smtp-relay

[Jean Helou]

Hello Benoit !

Matthieu and I are trying to create a server instance to host our family's
email :)
Being the geeks that we are, we are not satisfied with a single node JPA
deployment and want
to preserve the scalability and resilience properties of distributed james
while minimizing the
run cost of a small scale deployment so as to keep our hobby in a
reasonable[1] price range.

Since a mail server is such a  complex piece of software, we chose to
concentrate our efforts
on a smaller scope first: SMTP processing only. Our initial deployment will
thus be SMTP
only, then we intend to look into building an IMAP/JMAP instance, a bit
like what's described in
"distributed james - specialized instances" [2]. At least that's what's
guiding our forays into james
codebase.

We looked at various options and found a pulsar as a service that's fairly
cheap over at clever-cloud, they also have a reasonably cheap S3 clone.
This is what led to the initial pulsar dev as the cost of other
MQ saas offering was much higher.
That leaves the primary datastore, we initially targeted cassandra because
we knew that worked but that made the run cost explode.

At the moment we will intend to start with a hosted postgresql instance
instead. If only to finally be able to see the pulsar code run without
ruining ourselves :)

With this clarified, I'll try to answer your mail :

The name makes it hard to grasp what goals are researched, we also miss
> a little statement/README about this new product.
>

The goals did change a couple times but it always wanted to be a "simpler",
SMTP-specialized instance than what could be achieved with the main
distributed server app.

Here is a little explaination of the goal of this application I can propose:
>
> ```
> This artifact leverage Apache Pulsar to deliver a mailet container that
> scales your mail processing.
> It can be used both for inbound and outbound behavior customization.
> Thanks to Apache Pulsar, get your hands on a fully featured distributed
> mail queue to manage efficiently your email delivery.
> It targets minimal dependencies: solely Apache Pulsar and an object store.
> ```
>
If we reach a consensus I would be happy to write its README, including
> sample configuration and start instructions.
>

Minimal dependencies to this level would be great but we looked into it and
we don't expect to be able to minimize them that much, in the short/medium
term.


> I understand from recent tickets (JAMES-3761 JAMES-3762 and JAMES-3763)
> that it is intendeed to not get Cassandra as a dependency. Some features
> are optional and can easily be dropped (recipient rewritting) however
> some others would eventually need an alternative implementation (I am
> thinking to domains, users). What is the plan regarding this?
>

Offering an alternative to cassandra which is an expensive fit for smaller
scale deployments would indeed require an alternative implementation to:
- DomainList (5 methods)
- RecipientRewriteTable with it's regex mapping (26 methods)
- UserRepository (15 methods)

I am not sure we can completely drop RecipientRewriteTable, I think we
looked into it with Matthieu and it wasn't as optional as it first looked.
I think maybe it had something to do with error handling which made it a
mandatory dependency for the mailet container.
Maybe we will try dropping it again in our next attempt at starting the
app...

If we can't drop it, it means 46 behaviours to implement. That's quite a
lot the time we can allocate for our
evening fun-first pair programming sessions :)
For the time being, we intend to run with JPA instead so we can get our
instances up and running, start accumulating feedback from a real world
deployment and work on documenting the configuration and run sides.

I realize we could build our assembly privately or in a different
repository but we also need a demo place to plug in the implementations we
consider to be the most valuable for the community : the pulsar mailqueue
and the blob store mailrepository.
I do feel that these two are components that most James deployments would
benefit from.

Regarding the artifact name I am uneasy with it. Generally we tend to be
> too technical on the artifact name and base it on backing technologies
> rather than focussing on the intent. Also, 'relay' is one of the many
> possible features, but there are others. As such I think the
> 'smtp-relay' part of the name is too restrictive.
>

I wholeheartedly agree, I am quite ashamed being the author of the current
naming.

Alternative concepts for names:
>   - Replace smtp-relay with smtp only to make it more generic
>
+1

>   - Replace smtp-relay by mail-processing which I think might be the
> main target of such a server.
>

`mail-processing` feels too generic to me, while it might be a good fit I
wouldn't be able to tell what such an app does. I think smtp is a better
name as that is the only mail protocol
spoken by the app.

  - Replace pulsar-cassandra by scaling or distributed 

[jira] [Commented] (JAMES-3760) MailRepositoryContract is not cleaning the created Mails

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534300#comment-17534300
 ] 

Jean Helou commented on JAMES-3760:
---

Wouldn't that make it tricky to distinguish between "expected" warnings that 
stem from the testing code itself from warnings introduced by new production 
code ?

> MailRepositoryContract is not cleaning the created Mails
> 
>
> Key: JAMES-3760
> URL: https://issues.apache.org/jira/browse/JAMES-3760
> Project: James Server
>  Issue Type: Bug
>  Components: tests
>Affects Versions: master
>Reporter: Matthieu Baechler
>Priority: Minor
>
> Once https://issues.apache.org/jira/browse/JAMES-3759 is fixed, a lot of 
> leaks are detected when running 
> {code:java}
> MailRepositoryContract{code}
> The root cause is 
> {noformat}
> createMail{noformat}
> generates MailImpl instances which need to be disposed



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Commented] (JAMES-3700) Dead letter policy for the Pulsar MailQueue

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17505027#comment-17505027
 ] 

Jean Helou commented on JAMES-3700:
---

I'm passing by and it seems the related merge request is merged.

 

Should we close this issue ? (and maybe open an exploratory issue for schema 
support ?)

> Dead letter policy for the Pulsar MailQueue
> ---
>
> Key: JAMES-3700
> URL: https://issues.apache.org/jira/browse/JAMES-3700
> Project: James Server
>  Issue Type: Sub-task
>  Components: pulsar, Queue
>Affects Versions: master
>Reporter: Benoit Tellier
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Currently the Pulsar MailQueue do not come up with a dead-letter policy.
> A bad JSON payload halts the processing. 
> This makes the Pulsar MailQeue brittle:
>   - The ability to inject a single message with a bad payload can cause an 
> entire James cluster to come to a halt.
>  - Could be seen as an attack vector
>  - But also any changes to the underlying JSON schema for payloads is 
> susceptible to cause major downtime.
> We should define a deadletter policy:
>  - Given a number of failures delivery of the message would be abandonned
>  - And moved to a dead-letter topic for later audit (prevent data loss)



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Created] (JAMES-3721) Browse on the Pulsar mailqueue fails with ObjectNotFoundException

[Jean Helou (Jira)]

Jean Helou created JAMES-3721:
-

 Summary: Browse on the Pulsar mailqueue fails with 
ObjectNotFoundException
 Key: JAMES-3721
 URL: https://issues.apache.org/jira/browse/JAMES-3721
 Project: James Server
  Issue Type: Bug
  Components: pulsar, Queue
Reporter: Jean Helou


We have identified a couple interrelated issues around the [puslar 
mailqueue|https://issues.apache.org/jira/browse/JAMES-3695] :

Upon acknowledging a message, the corresponding mime message was not removed 
from the blob store.

Upon fixing that issue (and adding the corresponding verification), we started 
getting more failures on the browse (the failures could also be observed on the 
remove).

 

Because of how pulsar works and because we allow parallel processing of 
messages, the browse operation can end up reading meta data of messages that 
have been removed or acknowledged. The metadata of acknowledged messages is not 
immediately  purged from pulsar. It is somewhat configurable but for a normally 
operating cluster acked messages can [remain readable for 
hours|https://pulsar.apache.org/docs/en/cookbooks-retention-expiry/#delete-messages-from-namespaces]
 (and forcing that to the order of seconds would badly affect the cluster 
operation)

 

We chose to catch the specific exception and simply ignore the corresponding 
mails during the browse operation.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Commented] (JAMES-3700) Dead letter policy for the Pulsar MailQueue

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17494080#comment-17494080
 ] 

Jean Helou commented on JAMES-3700:
---

The schema way is really neat but I feel a small word of warning is required.

I strongly suggest a long experiment with at least  couple schema update on non 
critical data before actually enabling it in production.

I had production crashes in one of the projects where I used pulsar and avro 
schemas: 

On application startup for some reason my pulsar stack ended up skipping older 
versions of the schema and crashed attempting to read older message with the 
latest schema. I got pretty deep in the debugging but unfortunately I never got 
to the bottom of the issue (as in I didn't have time to build a minimized 
reproduction that I could submit as a bug to pulsar). I can't prove that the 
issue was in the pulsar client event if it is my personal conviction. It may 
have been a misuse (but I couldn't identify where), it may be a bug that was 
fixed (it was in an older version of pulsar) but it warrants some caution :)

One of the thing I intended to try was using an external, explicit schema 
registry instead of using the pulsar embedded implicit registry because I 
suspected that the schema was not consistently distributed in the cluster and 
connecting to the wrong node would trigger the issue.

 

> Dead letter policy for the Pulsar MailQueue
> ---
>
> Key: JAMES-3700
> URL: https://issues.apache.org/jira/browse/JAMES-3700
> Project: James Server
>  Issue Type: Sub-task
>  Components: pulsar, Queue
>Affects Versions: master
>Reporter: Benoit Tellier
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Currently the Pulsar MailQueue do not come up with a dead-letter policy.
> A bad JSON payload halts the processing. 
> This makes the Pulsar MailQeue brittle:
>   - The ability to inject a single message with a bad payload can cause an 
> entire James cluster to come to a halt.
>  - Could be seen as an attack vector
>  - But also any changes to the underlying JSON schema for payloads is 
> susceptible to cause major downtime.
> We should define a deadletter policy:
>  - Given a number of failures delivery of the message would be abandonned
>  - And moved to a dead-letter topic for later audit (prevent data loss)



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Commented] (JAMES-3698) Pulsar MailQueue should support deletion of scheduled messages.

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17494074#comment-17494074
 ] 

Jean Helou commented on JAMES-3698:
---

I might be wrong but I think this is fixed no ?

> Pulsar MailQueue should support deletion of scheduled messages.
> ---
>
> Key: JAMES-3698
> URL: https://issues.apache.org/jira/browse/JAMES-3698
> Project: James Server
>  Issue Type: Sub-task
>  Components: pulsar, Queue
>Affects Versions: master
>Reporter: Benoit Tellier
>Priority: Major
>  Labels: bug
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Pulsar MailQueue should support deletion of scheduled messages.
> Also, clearing expired filters is an optimization relying on messages being 
> ordered. Out of order messages, like delayed messages, might make this 
> trivially false. We might have to drop this optimisation.
> I did write 2 tests demostrating the expected behaviour regarding deletion of 
> delayed mails. See https://github.com/apache/james-project/pull/831
> DOD: Make those two tests pass on top of pulsar mail queue.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Comment Edited] (JAMES-3696) Solve instable test for the pulsar mail queue

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17476117#comment-17476117
 ] 

Jean Helou edited comment on JAMES-3696 at 1/14/22, 12:45 PM:
--

> This makes build profiles harder to grasp, issues harder to reproduce. Could 
> also be seen as an entry barreer to people running the build for the first 
> time on slow environments.

As long as we use a constant increase factor shared in all tests and applied to 
all static wait times, it shouldn't affect the build profile. On the other 
hand, don't you think the current build time of James is also a barrier to 
entry (IIRC a full build is somewhere around 2h30, I don't even try to run it 
anymore since it's so slow) ? 

Static wait times mean slower tests *everywhere* ,  tests slower than they need 
to be on developpers machine means wasted human time :( 

 


was (Author: jeantil):
> This makes build profiles harder to grasp, issues harder to reproduce. Could 
> also be seen as an entry barreer to people running the build for the first 
> time on slow environments.

As long as we use a constant increase factor it shouldn't affect the build 
profile much. On the other hand, don't you think the current build time of 
James is also a barrier to entry (IIRC a full build is somewhere around 2h30, I 
don't even try to run it anymore since it's so slow) ? 

Static wait times mean slower tests *everywhere* ,  tests slower than they need 
to be on developpers machine means wasted human time :( 

 

> Solve instable test for the pulsar mail queue
> -
>
> Key: JAMES-3696
> URL: https://issues.apache.org/jira/browse/JAMES-3696
> Project: James Server
>  Issue Type: Sub-task
>  Components: pulsar, Queue
>Affects Versions: master
>Reporter: Benoit Tellier
>Priority: Major
>  Labels: bug
>
> https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-826/7/
> Pulsar mail queue bringed in some unstable tests
> {code:java}
> Test Result (4 failures / +4)
> 
> org.apache.james.queue.pulsar.PulsarMailQueueTest.removeByRecipientShouldRemoveSpecificEmail
> 
> org.apache.james.queue.pulsar.PulsarMailQueueTest.removeByRecipientShouldRemoveSpecificEmailWhenMultipleRecipients{List,
>  MailAddress}[5]
> 
> org.apache.james.queue.pulsar.PulsarMailQueueTest.removeByRecipientShouldRemoveSpecificEmailWhenMultipleRecipients{List,
>  MailAddress}[6]
> 
> org.apache.james.queue.pulsar.PulsarMailQueueTest.removeShouldRemoveMailFromStoreWhenFilteredOut
> {code}
> Looking at the first, some email expected to be deleted were not:
> {code:java}
> Error Message
> Expecting actual:
>   ["name1", "name2"]
> to contain exactly (and in same order):
>   ["name1"]
> but some elements were not expected:
>   ["name2"]
> {code}
> Maybe because propagation of the delete filter is asynchronous? (IE the 
> client delete returns when enqueuing the filter but without any warranty so 
> as when it would be dequeued and applied). If so this isa design flaw and I 
> hardly see how to improve that .
> Or we add a delay in the tests to relax the consistency guaranty?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Commented] (JAMES-3696) Solve instable test for the pulsar mail queue

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17476117#comment-17476117
 ] 

Jean Helou commented on JAMES-3696:
---

> This makes build profiles harder to grasp, issues harder to reproduce. Could 
> also be seen as an entry barreer to people running the build for the first 
> time on slow environments.

As long as we use a constant increase factor it shouldn't affect the build 
profile much. On the other hand, don't you think the current build time of 
James is also a barrier to entry (IIRC a full build is somewhere around 2h30, I 
don't even try to run it anymore since it's so slow) ? 

Static wait times mean slower tests *everywhere* ,  tests slower than they need 
to be on developpers machine means wasted human time :( 

 

> Solve instable test for the pulsar mail queue
> -
>
> Key: JAMES-3696
> URL: https://issues.apache.org/jira/browse/JAMES-3696
> Project: James Server
>  Issue Type: Sub-task
>  Components: pulsar, Queue
>Affects Versions: master
>Reporter: Benoit Tellier
>Priority: Major
>  Labels: bug
>
> https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-826/7/
> Pulsar mail queue bringed in some unstable tests
> {code:java}
> Test Result (4 failures / +4)
> 
> org.apache.james.queue.pulsar.PulsarMailQueueTest.removeByRecipientShouldRemoveSpecificEmail
> 
> org.apache.james.queue.pulsar.PulsarMailQueueTest.removeByRecipientShouldRemoveSpecificEmailWhenMultipleRecipients{List,
>  MailAddress}[5]
> 
> org.apache.james.queue.pulsar.PulsarMailQueueTest.removeByRecipientShouldRemoveSpecificEmailWhenMultipleRecipients{List,
>  MailAddress}[6]
> 
> org.apache.james.queue.pulsar.PulsarMailQueueTest.removeShouldRemoveMailFromStoreWhenFilteredOut
> {code}
> Looking at the first, some email expected to be deleted were not:
> {code:java}
> Error Message
> Expecting actual:
>   ["name1", "name2"]
> to contain exactly (and in same order):
>   ["name1"]
> but some elements were not expected:
>   ["name2"]
> {code}
> Maybe because propagation of the delete filter is asynchronous? (IE the 
> client delete returns when enqueuing the filter but without any warranty so 
> as when it would be dequeued and applied). If so this isa design flaw and I 
> hardly see how to improve that .
> Or we add a delay in the tests to relax the consistency guaranty?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Commented] (JAMES-3696) Solve instable test for the pulsar mail queue

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17472805#comment-17472805
 ] 

Jean Helou commented on JAMES-3696:
---

The tests actually already account for the eventual consistency of the filter 
propagation, they include a call to `awaitRemove()` which is empty for most 
implementations and is defined as 
{code:java}
@Override
public void awaitRemove() {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
} {code}
in PulsarMailqueueTest

 

What likely happens is that 100ms is too little for the CI plateform, there are 
2 options:

- change all the assertions on remove to use awaitility
- increase the wait time (if possible only on CI, we could try to resolve the 
CI env variable and conditionnaly have a larger wait time)

> Solve instable test for the pulsar mail queue
> -
>
> Key: JAMES-3696
> URL: https://issues.apache.org/jira/browse/JAMES-3696
> Project: James Server
>  Issue Type: Sub-task
>  Components: pulsar, Queue
>Affects Versions: master
>Reporter: Benoit Tellier
>Priority: Major
>  Labels: bug
>
> https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-826/7/
> Pulsar mail queue bringed in some unstable tests
> {code:java}
> Test Result (4 failures / +4)
> 
> org.apache.james.queue.pulsar.PulsarMailQueueTest.removeByRecipientShouldRemoveSpecificEmail
> 
> org.apache.james.queue.pulsar.PulsarMailQueueTest.removeByRecipientShouldRemoveSpecificEmailWhenMultipleRecipients{List,
>  MailAddress}[5]
> 
> org.apache.james.queue.pulsar.PulsarMailQueueTest.removeByRecipientShouldRemoveSpecificEmailWhenMultipleRecipients{List,
>  MailAddress}[6]
> 
> org.apache.james.queue.pulsar.PulsarMailQueueTest.removeShouldRemoveMailFromStoreWhenFilteredOut
> {code}
> Looking at the first, some email expected to be deleted were not:
> {code:java}
> Error Message
> Expecting actual:
>   ["name1", "name2"]
> to contain exactly (and in same order):
>   ["name1"]
> but some elements were not expected:
>   ["name2"]
> {code}
> Maybe because propagation of the delete filter is asynchronous? (IE the 
> client delete returns when enqueuing the filter but without any warranty so 
> as when it would be dequeued and applied). If so this isa design flaw and I 
> hardly see how to improve that .
> Or we add a delay in the tests to relax the consistency guaranty?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: About further adoption of Apache Pulsar to back the distributed server messaging use case.

[Jean Helou]

Hello,

I'm obviously favorable to using pulsar :)

You mention :
> I think a good long term objective for the PMC is to drop RabbitMQ in
favor of pulsar

And in the ADR you also mention deprecating the RabbitMQ component.

Can you clarify which implementations would be preserved/dropped (I'm
obviously thinking of JMS implementations)

Cheers and happy new year james team
jean
On Fri, Jan 7, 2022 at 12:25 PM Benoit TELLIER  wrote:

> Hello all,
>
> Thanks to the very nice work of Matthieu and Jean we now have a working
> Pulsar implementation for the James Mail Queue component [1]. Koodo!
>
> [1] https://github.com/apache/james-project/pull/808
>
> I did put together an ADR [2] providing details over:
>
>  - Context over MailQueue and its historical implementation
>  - Why would someone want Pulsar to back the mailqueue
>  - Propose follow up steps for the Pulsar adoption regarding the
> distributed James server
>  - And finally technical details on this implementation.
>
> [2] https://github.com/apache/james-project/pull/829
>
> I think a good long term objective for the PMC is to drop RabbitMQ in
> favor of pulsar (third parties could package their own components using
> RabbitMQ if they wishes...)
>
> This means:
>  - Solve the bugs that were found during the Pulsar MailQueue review
>  - Pulsar MailQueue need to allow listing blobs in order to be
> deduplication friendly.
>  - Provide an event bus based on Pulsar
>  - Provide a task manager based on Pulsar
>  - Package a distributed server backed by pulsar, deprecate then replace
> the current one.
>  - (optionally) support mail queue priorities
>
> While contributions would of course be welcomed on this topic, we could
> offer it as part of GSOC 2022, and we could co-mentor it with mentors of
> the Pulsar community (see [3])
>
> [3] https://lists.apache.org/thread/y9s7f6hmh51ky30l20yx0dlz458gw259
>
> Would such a plan gain traction around here ?
>
> Best regards,
>
> Benoit TELLIER
>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

[jira] [Comment Edited] (JAMES-3687) Implements Apache Pulsar based Mailqueue

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17472650#comment-17472650
 ] 

Jean Helou edited comment on JAMES-3687 at 1/11/22, 10:52 AM:
--

all these already have a wait time built in to account for the eventual 
consistency  of remove
{code:java}
@Override
public void awaitRemove() {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
} {code}
However as usual with fixed wait time maybe this is too short. There is no API 
to observe applicable delete filters and I'm not sure it's a good idea. 
Two options here:

 
 * derive the wait time from ENV defaulting to 100ms, so the 100 can be 
overriden for CI
 * change all the tests assertions to use Awaitility... and wait for eventual 
consistency


was (Author: jeantil):
all these already have a wait time built in to account for the eventual 
consistency  of remove

 
 * verride public void awaitRemove() \{ try { Thread.sleep(100); } catch (
 * InterruptedException e) \{ throw new RuntimeException(e); } }

However as usual with fixed wait time maybe this is too short. There is no API 
to observe applicable delete filters and I'm not sure it's a good idea. 
Two options here:

 
 * derive the wait time from ENV defaulting to 100ms, so the 100 can be 
overriden for CI
 * change all the tests assertions to use Awaitility... and wait for eventual 
consistency

> Implements Apache Pulsar based Mailqueue
> 
>
> Key: JAMES-3687
> URL: https://issues.apache.org/jira/browse/JAMES-3687
> Project: James Server
>  Issue Type: Sub-task
>  Components: Queue
>    Reporter: Jean Helou
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> An apache pulsar based mailqueue offers a different set of compromises over 
> the existing mailqueue implementations:
> pros:
>  * pulsar is a distributed queue
>  * pulsar offers scheduling facilities making it easier to implement delayed 
> queues
> cons:
>  * being fully distributed some consistency guarantees cannot be honored for 
> flush and filter since the flushing and filtering commands take time to 
> propagate in the cluster



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Commented] (JAMES-3687) Implements Apache Pulsar based Mailqueue

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17472650#comment-17472650
 ] 

Jean Helou commented on JAMES-3687:
---

all these already have a wait time built in to account for the eventual 
consistency  of remove

 
 * verride public void awaitRemove() \{ try { Thread.sleep(100); } catch (
 * InterruptedException e) \{ throw new RuntimeException(e); } }

However as usual with fixed wait time maybe this is too short. There is no API 
to observe applicable delete filters and I'm not sure it's a good idea. 
Two options here:

 
 * derive the wait time from ENV defaulting to 100ms, so the 100 can be 
overriden for CI
 * change all the tests assertions to use Awaitility... and wait for eventual 
consistency

> Implements Apache Pulsar based Mailqueue
> 
>
> Key: JAMES-3687
> URL: https://issues.apache.org/jira/browse/JAMES-3687
> Project: James Server
>  Issue Type: Sub-task
>  Components: Queue
>    Reporter: Jean Helou
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> An apache pulsar based mailqueue offers a different set of compromises over 
> the existing mailqueue implementations:
> pros:
>  * pulsar is a distributed queue
>  * pulsar offers scheduling facilities making it easier to implement delayed 
> queues
> cons:
>  * being fully distributed some consistency guarantees cannot be honored for 
> flush and filter since the flushing and filtering commands take time to 
> propagate in the cluster



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [jira] [Commented] (JAMES-3689) JMSCacheableMailQueue remove is buggy

[Jean Helou]

Hello Benoit

Yes the PR https://github.com/apache/james-project/pull/824 fixes this bug
and the errors we encounter on the pulsar mailqueue PR

Jean

Le jeu. 6 janv. 2022 à 03:06, Benoit Tellier (Jira) <
server-dev@james.apache.org> a écrit :

>
> [
> https://issues.apache.org/jira/browse/JAMES-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469628#comment-17469628
> ]
>
> Benoit Tellier commented on JAMES-3689:
> ---
>
> Hello Jean,
>
> I see https://github.com/apache/james-project/pull/824 opened by
> [~matthieu]. Is this addressing this issue?
>
> > JMSCacheableMailQueue remove is buggy
> > -
> >
> > Key: JAMES-3689
> > URL: https://issues.apache.org/jira/browse/JAMES-3689
> > Project: James Server
> >  Issue Type: Bug
> >Reporter: Jean Helou
> >Priority: Major
> >
> > While implementing the pulsar mailqueue we accidentally discovered this
> bug in the  JMSCacheableMailQueue:
> > The test of the manageable mailqueue contract named
> >
> {color:#172b4d}removeByRecipientShouldRemoveSpecificEmailWhenMultipleRecipients{color}
> > {color:#172b4d}can be made to fail by {color}
> >  * {color:#172b4d}swapping the order of the recipients in the mail
> fixture. We tried with a few combinations and the remove never works for
> the first recipient.{color}
> >  * {color:#172b4d}adding a recipient which is similar enough to the
> targeted recipient (for instance removing [b...@example.co|mailto:
> b...@example.co] and having a mail with recipient b...@example.com){color}
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.20.1#820001)
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

[jira] [Updated] (JAMES-3689) JMSCacheableMailQueue remove is buggy

[Jean Helou (Jira)]

 [ 
https://issues.apache.org/jira/browse/JAMES-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean Helou updated JAMES-3689:
--
Description: 
While implementing the pulsar mailqueue we accidentally discovered this bug in 
the  JMSCacheableMailQueue:

The test of the manageable mailqueue contract named

{color:#172b4d}removeByRecipientShouldRemoveSpecificEmailWhenMultipleRecipients{color}

{color:#172b4d}can be made to fail by {color}
 * {color:#172b4d}swapping the order of the recipients in the mail fixture. We 
tried with a few combinations and the remove never works for the first 
recipient.{color}
 * {color:#172b4d}adding a recipient which is similar enough to the targeted 
recipient (for instance removing [b...@example.co|mailto:b...@example.co] and 
having a mail with recipient b...@example.com){color}

  was:
While implementing the pulsar mailqueue we accidentally discovered this bug in 
the  JMSCacheableMailQueue: 

The test of the manageable mailqueue contract named 

{color:#172b4d}removeByRecipientShouldRemoveSpecificEmailWhenMultipleRecipients{color}

{color:#172b4d}can be made to fail by swapping the order of the recipients in 
the mail fixture. We tried with a few combinations and the remove never works 
for the first recipient.{color}

Summary: JMSCacheableMailQueue remove is buggy  (was: 
JMSCacheableMailQueue remove does not works for the the first recipient)

> JMSCacheableMailQueue remove is buggy
> -
>
> Key: JAMES-3689
> URL: https://issues.apache.org/jira/browse/JAMES-3689
> Project: James Server
>  Issue Type: Bug
>    Reporter: Jean Helou
>Priority: Major
>
> While implementing the pulsar mailqueue we accidentally discovered this bug 
> in the  JMSCacheableMailQueue:
> The test of the manageable mailqueue contract named
> {color:#172b4d}removeByRecipientShouldRemoveSpecificEmailWhenMultipleRecipients{color}
> {color:#172b4d}can be made to fail by {color}
>  * {color:#172b4d}swapping the order of the recipients in the mail fixture. 
> We tried with a few combinations and the remove never works for the first 
> recipient.{color}
>  * {color:#172b4d}adding a recipient which is similar enough to the targeted 
> recipient (for instance removing [b...@example.co|mailto:b...@example.co] and 
> having a mail with recipient b...@example.com){color}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Created] (JAMES-3689) JMSCacheableMailQueue remove does not works for the the first recipient

[Jean Helou (Jira)]

Jean Helou created JAMES-3689:
-

 Summary: JMSCacheableMailQueue remove does not works for the the 
first recipient
 Key: JAMES-3689
 URL: https://issues.apache.org/jira/browse/JAMES-3689
 Project: James Server
  Issue Type: Bug
Reporter: Jean Helou


While implementing the pulsar mailqueue we accidentally discovered this bug in 
the  JMSCacheableMailQueue: 

The test of the manageable mailqueue contract named 

{color:#172b4d}removeByRecipientShouldRemoveSpecificEmailWhenMultipleRecipients{color}

{color:#172b4d}can be made to fail by swapping the order of the recipients in 
the mail fixture. We tried with a few combinations and the remove never works 
for the first recipient.{color}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Updated] (JAMES-3687) Implements Apache Pulsar based Mailqueue

[Jean Helou (Jira)]

 [ 
https://issues.apache.org/jira/browse/JAMES-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean Helou updated JAMES-3687:
--
Description: 
An apache pulsar based mailqueue offers a different set of compromises over the 
existing mailqueue implementations:

pros:
 * pulsar is a distributed queue
 * pulsar offers scheduling facilities making it easier to implement delayed 
queues

cons:
 * being fully distributed some consistency guarantees cannot be honored for 
flush and filter since the flushing and filtering commands take time to 
propagate in the cluster

  was:As discussed in JAMES-2896 implementing a mailqueue with proper delay 
support is easier on apache pulsar


> Implements Apache Pulsar based Mailqueue
> 
>
> Key: JAMES-3687
> URL: https://issues.apache.org/jira/browse/JAMES-3687
> Project: James Server
>  Issue Type: Improvement
>  Components: Queue
>    Reporter: Jean Helou
>Priority: Major
>
> An apache pulsar based mailqueue offers a different set of compromises over 
> the existing mailqueue implementations:
> pros:
>  * pulsar is a distributed queue
>  * pulsar offers scheduling facilities making it easier to implement delayed 
> queues
> cons:
>  * being fully distributed some consistency guarantees cannot be honored for 
> flush and filter since the flushing and filtering commands take time to 
> propagate in the cluster



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Created] (JAMES-3687) Implements Apache Pulsar based Mailqueue

[Jean Helou (Jira)]

Jean Helou created JAMES-3687:
-

 Summary: Implements Apache Pulsar based Mailqueue
 Key: JAMES-3687
 URL: https://issues.apache.org/jira/browse/JAMES-3687
 Project: James Server
  Issue Type: Improvement
  Components: Queue
Reporter: Jean Helou


As discussed in JAMES-2896 implementing a mailqueue with proper delay support 
is easier on apache pulsar



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Authentication in James: Enable/require/disable OpenID for JMAP, IMAP and SMTP

[Jean Helou]

+1

On Thu, Dec 2, 2021 at 5:57 AM btell...@linagora.com 
wrote:

> Hopefully rendered correctly in plain text...
>
> -
>
> Hello Jamers!
>
> As part of my work at Linagora, we are looking toward
>  - better integrating James within our product suite and for us this
> means support "Single Sign On" and "Single Log Out" for JMAP following
> the OpenId connect standard [0].
>  - Improving security standards used to opperate James. (We have a
> growing activity within the health care market, sensible to security
> arguments)
>
> Regarding security standards we should ideally:
>  - Be able to NOT advertise AUTH / LOGIN capabilities of unencrypted
> channels (Correspond to IMAP plainAuthDissallowed but generalized to
> other protocols)
>  - Be able to require OAUTH/OIDC authentication for all protocols (IMAP,
> SMTP, JMAP) - this is what some of the health care providers we spoke to
> desired to do.
>  - For our collaborative suite usage, while OIDC only for JMAP makes
> sense, we still wish to maintain PLAIN AUTH for IMAP and SMTP.
>  - Of course settings for regular users not interested in OIDC should
> not change (no breaking changes, OIDC adoption is opt-in only).
>
> As such we propose ourselves to:
>  - Contribute IMAP and SMTP SASL OAUTH extension described in [RFC-7628]
>  - Modularize JMAP authentication mechanisms (letting the admin choose
> which one she wishes to use)
>  - Enable authentication through a header mechanism eg `X-USER:
> btell...@apache.org`, which can be used to delegate OIDC authentication
> through a third party API gateway. We have Krackend [1] in mind.
>  - Share documentation and a docker-compose of OIDC setup for IMAP, SMTP
> and JMAP in
> https://github.com/apache/james-project/tree/master/examples/oidc. This
> would include:
> - A LDAP still used by James UsersRepository. Provisionned with a
> testing user.
> - A pre-configured Keycloack [2] OpenID connect provider.
> - A pre-configured Krakend API gateway proxying JMAP
> - And finally a James server configured to only accept OIDC as an
> authentication mechanism for IMAP, SMTP and JMAP.
>  - Unit tests for existing IMAP `plainAuthDissallowed` configuration
> parameter.
>
> Finally this is a good opportunity to restructure authentication related
> settings in imapserver.xml and smtpserver.xml file.
>
> Here are proposals for both files:
>
> [imapserver.xml]
> 
> 
> imapserver-ssl
> 0.0.0.0:993
> 
> file://conf/private.key
> file://conf/certs.self-signed.csr
> 
> 
> 
> true|false 
>  
> file://conf/imapJWT.pub
>
> https://example.com/.well-known/openid-config
> 
> mailAddress
> 
> 
> 
> 
>
> [smtpserver.xml]
> 
> 
> smtpserver-ssl
> 0.0.0.0:465
> 
> file://conf/private.key
> file://conf/certs.self-signed.csr
> 
> 
> true|false
> 
> true|false 
>  
> file://conf/imapJWT.pub
>
> https://example.com/.well-known/openid-config
> 
> mailAddress
> 
> 
> 
> 
>
> You can see that:
>  - The `plainAuthDissallowed` parameter is proposed to be renamed to
> `auth.requireSSL`. (of course we should support fallback NOT to have a
> breaking change)
>  - `auth.plainAuthEnabled` enable turning on/off plain auth, which
> allows having OIDC only mechanism.
>  - `auth.requireSSL` will be generalized to SMTP.
>  - In SMTP `requireAuth` setting is very misleading as it rather is
> `requireAuthForRelay`. I propose we rename this configuration option (of
> course we should support fallback NOT to have a breaking change).
>
> Here is the implementation strategies we would follow:
>
>  - For JMAP have Krakend doing all the hard job for us, and use a
> dedicated header to carry the information over to James.
>- Our code contributions aims at easing such a setup (that would only
> require configuration)
>- Provide an informative example using krakend. We understand that
> this choice is ours, yet sharing it could allow reuse or similar setup
>- If some people wishes to write an OIDC authentication strategy
> directly in James then they perfectly can! (Reusing the modularization
> of JMAP authentication strategies we would provide).
>
>  - For IMAP and SMTP then we proposes to check the bearer payload
> against a statically configured public key for these protocols. (Sadly
> there is no API gateway for those protocols)
>- Drawbacks includes no revocation of access tokens (once it's signed
> it is always valid), revocation do not shut down existing connections
> authenticated with the revocated token, and key rotation would require
> reconfiguration and reboot.
>- One alternative would be to systematically ask the OpenID server to
> validate the bearer. 

[jira] [Commented] (JAMES-3487) Configure MimeMessageInputStreamSource THRESHOLD

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437838#comment-17437838
 ] 

Jean Helou commented on JAMES-3487:
---

It would definitely be an improvement over the current situation :)

 

> Configure MimeMessageInputStreamSource THRESHOLD
> 
>
> Key: JAMES-3487
> URL: https://issues.apache.org/jira/browse/JAMES-3487
> Project: James Server
>  Issue Type: Improvement
>  Components: James Core
>Reporter: Benoit Tellier
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> This represents the point at which we should switch from a memory storage 
> into a file storage.
> Defaults is 100 Kb.
> Obviously this parameter is important:
>  - Higher values will operate mostly in-memory thus will have low latencies 
> but will trash the heap and might trigger a GC hell.
>  - Lower values will defensively operate on files. Higher latencies but 
> predictable throughtput. Modern SSDs and FS cache should enable to keep up 
> with high rates.
> Optimally we should have some bench showing the impact of this parameter.
> Related to JAMES-3477.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Time to retire maildir from James 3.7.X ?

[Jean Helou]

Hello devs,

I switch threads because I agree with the removal of the FS based mailqueue
but still find the maildir removal to be a bit more difficult.

Quoting from the other thread at
https://www.mail-archive.com/server-dev@james.apache.org/msg70960.html

What protocols are supported by the 2.3 release line? As far as I am
> aware of, it mostly is POP3, and - again to my own experience - POP3
> user do not tend to have a large mailbox (as POP3 causes it to be read
> again and again and over again), I doubt that we are speaking of the
> same volumetry than for instance an IMAP server migration.
>

Undoubtedly I don't know james code as well as you do but while gary
mentioned a 2.3 migration, it doesn't really matter ...
The maildir implementation currently exists in james 3.x[1], it is listed
as an official mailbox implementation and not marked deprecated [2]
Therefore it would be legitimate for users to use it with James 3 and
dropping it from 3.7 without a deprecation period sounds a bit cavalier to
me.

Also maildir is a fairly standard mailbox format, support for it allows for
migration from other software to james

Now you state:

> With such a high number of critical issues that no one dare fixing
>

I may be mistaken as I didn't investigate in details but it sounds like
most of the issues stem from a lack of sanitizing in FS paths. You mention
in JAMES-3646[3] that you are going to fix the sieve and FileRepositories.
Considering the practices in the james codebase most of the sanitizing code
is going to end up in a shared service. Once that's in place it would be
easier to fix a few of the maildir bugs using this code and it could become
a nice first contrib no ?

On a side note MAILBOX-183[4] "readUidFile is taking too much time for
large uid file and blocks other threads" is open but marked as a duplicate
of a closed bug ... not sure if it is still applicable.

Jean

[1] https://github.com/apache/james-project/tree/master/mailbox/maildir
[2] https://github.com/apache/james-project/blob/master/mailbox/README.md
[3] https://issues.apache.org/jira/browse/JAMES-3646
[4] https://issues.apache.org/jira/browse/MAILBOX-183

On Mon, Sep 6, 2021 at 5:47 AM Rene Cordier  wrote:

> Hello,
>
> I think you are making good points. It's probably better to focus on
> what works well if it's enough of an alternative to maildir (like
> embedded JPA as you mentionned), with little resources we have.
>
> +1 !
>
> Rene.
>
> On 05/09/2021 19:58, btell...@apache.org wrote:
> > Hello all,
> >
> > While reviewing overall file path sanitizing (JAMES-3646) I found
> > maildir to incorrectly sanitize some file path.
> >
> > Playing around, I discovered some magic names that leads to maildir
> > errors namely MAILBOX-406, and a NPE with an empty mailbox (MAILBOX-407).
> >
> > Moreover, over time the list of maildir defects keeps on growing:
> >
> >   - MAILBOX-389 Mailbox rename fails with Maildir
> >   - MAILBOX-393 mailboxId support for mailDir is partial
> >
> > Now looking on the bug tracker I notice people encounter (and nobody
> > answers):
> >
> >   - MAILBOX-390 Unexpected error accessing mailbox
> >   - MAILBOX-344 Maildir do not support MODSEQ search
> >   - MAILBOX-299 Maildir should fail gracefully when mailbox name is too
> long
> >   - MAILBOX-241 Maildir implementation does not check namespace on
> existance
> >   - MAILBOX-183 readUidFile is taking too much time for large uid file
> > and blocks other threads
> >   - JAMES-3612 Cryptic errors upon bad permissions
> >   - JAMES-1515 Maildirstorage backend crashes with Exception due to
> > "wrong" filename
> >
> > With such a high number of critical issues that no one dare fixing, it
> > sounds reasonable to me to state that maildir implementation do not
> > match quality standards that one would expect from an Apache project.
> >
> > Other mail servers do have great maildir implenentation, which
> > mistakenly lead users to think the maildir implementation is a quality
> > one, which is very far from the truth.
> >
> > Moreover, maildir code is outdated,  hard to work with, and pulls us
> > back in many way. Given the currently small amount of contributors, we
> > do not have time and resources to give it the love it deserves.
> >
> > Also, alternatives exist: embedded JPA database do also run well in
> > single instance mode, and could server as a maildir replacement. People
> > could use utilities like imapsync to plan their migration.
> >
> > As such, I propose myself to retire maildir and remove related code from
> > the repository.
> >
> > Regards,
> >
> > Benoit
> >
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, 

Re: Retiring the File mail queue from James 3.7.x

[Jean Helou]

Hi garry,

I think this concern is more valid for the retirement of the maildir[1]
implementation (mailbox) than for the mailqueue.
I don't think messages are expected to remain in the mailqueue for a very
long time, which is why the advocated migration path includes downtime :
- stop accepting new email traffic
- let the mailqueue process in-flight messages
- switch mailqueue implementation
- start accepting traffic again

for maildir (fs based mailbox implementation) it is more tricky as you need
to do a data migration :
read the whole mailbox content and rewrite it in the new store

[1]
http://mail-archives.apache.org/mod_mbox/james-server-dev/202109.mbox/browser

On Tue, Sep 7, 2021 at 3:19 PM Garry Hurley 
wrote:

> My only concern would be those people who are still, for some reason, not
> yet ready to move to James 3.x - yes, there are still some out there.  They
> are more likely to be using the file repo than 3.x users.  If they are
> using a file repo, we might want to keep a migration tool around to migrate
> their 'old messages' to one of the newer repos.  I know we don't always
> think about it, but some organizations (particularly government ones) have
> a records retention policy that mandates they keep old emails for years
> rather than days.  Those users need a - maintained - way of migrating from
> the file repo to the newer one.
>
>
> Just my two cents.
>
> On Mon, Sep 6, 2021 at 10:18 AM Raphaël Ouazana-Sustowski <
> rouaz...@apache.org> wrote:
>
> > +1
> >
> > Le 06/09/2021 à 14:54, Jean Helou a écrit :
> > > awesome :)
> > >
> > > On Mon, Sep 6, 2021 at 1:52 PM btell...@apache.org <
> btell...@apache.org>
> > > wrote:
> > >
> > >> Hello Jean,
> > >>
> > >> On 06/09/2021 18:42, Jean Helou wrote:
> > >>> Hello benoit,
> > >>>
> > >>>
> > >>>> There are some alternatives: embedded activeMQ has zero
> dependencies.
> > >>>> Furthermore migrating is easy: just migrate with an empty queue -
> > which
> > >>>> can be done easily with a SMTP downtime.
> > >>>>
> > >>>> Also the component is deprecated for 1.5 years, given the many flows
> > it
> > >>>> have, given that no contributors shows up to keep it alive, I
> advocate
> > >>>> removal.
> > >>>>
> > >>> Sounds fine to me. Maybe make sure the documentation/sample server
> > >> explains
> > >>> how to properly configure activemq to have filesystem based msg
> > >> persistence
> > >>> (https://activemq.apache.org/kahadb) as an inmemory only activemq
> > would
> > >>> reduce the safety of the mailqueue ?
> > >> KahaDB with an embedded AcctiveMQ is already the default for MailQueue
> > >> (ActiveMQMailQueueFactory) for Spring and JPA-guice products.
> > >>
> > >> Regards
> > >>>
> > >>> jean
> > >>>
> > >>
> > >> -
> > >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > >> For additional commands, e-mail: server-dev-h...@james.apache.org
> > >>
> > >>
> > >
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>

Re: Retiring the File mail queue from James 3.7.x

[Jean Helou]

awesome :)

On Mon, Sep 6, 2021 at 1:52 PM btell...@apache.org 
wrote:

> Hello Jean,
>
> On 06/09/2021 18:42, Jean Helou wrote:
> > Hello benoit,
> >
> >
> >> There are some alternatives: embedded activeMQ has zero dependencies.
> >> Furthermore migrating is easy: just migrate with an empty queue - which
> >> can be done easily with a SMTP downtime.
> >>
> >> Also the component is deprecated for 1.5 years, given the many flows it
> >> have, given that no contributors shows up to keep it alive, I advocate
> >> removal.
> >>
> > Sounds fine to me. Maybe make sure the documentation/sample server
> explains
> > how to properly configure activemq to have filesystem based msg
> persistence
> > (https://activemq.apache.org/kahadb) as an inmemory only activemq would
> > reduce the safety of the mailqueue ?
> KahaDB with an embedded AcctiveMQ is already the default for MailQueue
> (ActiveMQMailQueueFactory) for Spring and JPA-guice products.
>
> Regards
> >
> > jean
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Retiring the File mail queue from James 3.7.x

[Jean Helou]

Hello benoit,


> There are some alternatives: embedded activeMQ has zero dependencies.
> Furthermore migrating is easy: just migrate with an empty queue - which
> can be done easily with a SMTP downtime.
>
> Also the component is deprecated for 1.5 years, given the many flows it
> have, given that no contributors shows up to keep it alive, I advocate
> removal.
>

Sounds fine to me. Maybe make sure the documentation/sample server explains
how to properly configure activemq to have filesystem based msg persistence
(https://activemq.apache.org/kahadb) as an inmemory only activemq would
reduce the safety of the mailqueue ?

jean

Re: [VOTE] Retire Apache James HUPA

[Jean Helou]

+1

Le ven. 23 juil. 2021 à 11:28, Antoine Duprat  a écrit :

> +1
>
> Le ven. 23 juil. 2021 à 11:01, btell...@apache.org  a
> écrit :
>
> > Hello all,
> >
> > Following a first email on the topic [1] I would like to call for a
> > formal vote on Apache James Hupa retirement.
> >
> > [1]
> https://www.mail-archive.com/server-dev@james.apache.org/msg70575.html
> >
> > Rationnals:
> >  - The latest release (0.3.0) dates from 2012 which is an eternity in
> > computing.
> >  - The latest tag on Github is 0.0.3
> >  - The pom references 0.0.5-SNAPSHOT suggesting that 0.0.4 release is
> > lost :-(
> >  - This repository is crippled by multiple CVEs (quick dependabot
> review):
> >   - CVE-2021-29425 (commons-io)
> >   - GHSA-m6cp-vxjx-65j6 CVE-2017-7656 CVE-2015-2080 CVE-2017-7657
> > CVE-2019-10241 CVE-2019-10247 (Jetty server)
> >   - CVE-2020-9447 (gwtupload)
> >   - GHSA-g3wg-6mcf-8jj6 (jetty-webapp)
> >   - CVE-2019-17571 (log4j)
> >   - CVE-2016-131 CVE-2016-3092 (commons-fileupload)
> >  - Sporadic activity since 2012
> >  - Zero to no exchanges for several years on the mailing lists.
> >
> > Given that alternatives exists, given that the project is
> > likely not mature, unmaintained and unsecure, I propose to retire this
> > Apache James subproject.
> >
> > |Voting rules: - This is a majority vote as stated in [2] for procedural
> > issues. - The vote starts at Friday 23rd of July 2021, 4pm UTC+7 - The
> > vote ends at Friday 30th of July 2021, 4pm UTC+7 [2]
> > https://www.apache.org/foundation/voting.html Following this retirement,
> > follow up steps are to be taken as described in [3] [3]
> > https://www.mail-archive.com/server-dev@james.apache.org/msg70585.html
> > | - 1. Get a formal vote on server-dev mailing list
> >  - 2. Place a RETIRED_PROJECT file marker in the git
> >  - 3. Add a note in the project README
> >  - 4. Retire the ISSUE trackers (Project names HUPA and POSTAGE)
> >  - 5. Announce it on gene...@james.apache.org and announce@apache
> >  - 6. Add a notice to the Apache website, if present
> >  - 7. Remove releases from downloads.apache.org
> >  - 8. Add notices on the Apache release archives (example
> > https://archive.apache.org/dist/ant/antidote/ <
> > https://archive.apache.org/dist/ant/antidote/>)
> >
> > Best regards,
> >
> > Benoit Tellier
> > ||
> >
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>

Re: [VOTE] Retire Apache James Postage

[Jean Helou]

+1

Le ven. 23 juil. 2021 à 11:28, Antoine Duprat  a écrit :

> +1
>
> Le ven. 23 juil. 2021 à 11:03, btell...@apache.org  a
> écrit :
>
> > Hello all,
> >
> > Following a first email on the topic [1] I would like to call for a
> > formal vote on Apache James Postage retirement.
> >
> > [1]
> https://www.mail-archive.com/server-dev@james.apache.org/msg70576.html
> >
> > Rationnals: this project...
> >  - Have no website page (not deployed)
> >  - Have no README
> >  - Have no formal release, but a tag named "james-2_20120613" dating
> > from 2012 which is quite old already...
> >  - Their exists some alternatives both for JMETER, and Gatling
> > performance testing tools.
> >  - Lack of support for recent mail protocols like IMAP and JMAP
> >  - Hard to scale blocking architecture (from what I understood?)
> >  - No development activity since 2013.
> >  - 5 forks in total on github, none of them did extra developments.
> >  - Relies on 3.0-beta5-SNAPSHOT which is quite old but also unreleased.
> > Proting postage to a released version would likely be already quite of a
> > fight...
> >  - Affected by CVE-2021-29425 (commons-io)||
> > Given the maturity of the project, the presence of alternatives, and the
> > absence of development, in the absence of mainteners, it could be wise
> > to consider retiring it.
> > |Voting rules: - This is a majority vote as stated in [2] for procedural
> > issues. - The vote starts at Friday 23rd of July 2021, 4pm UTC+7 - The
> > vote ends at Friday 30th of July 2021, 4pm UTC+7 [2]
> > https://www.apache.org/foundation/voting.html Following this retirement,
> > follow up steps are to be taken as described in [3] [3]
> > https://www.mail-archive.com/server-dev@james.apache.org/msg70585.html
> > | - 1. Get a formal vote on server-dev mailing list
> >  - 2. Place a RETIRED_PROJECT file marker in the git
> >  - 3. Add a note in the project README
> >  - 4. Retire the ISSUE trackers (Project names POSTAGE)
> >  - 5. Announce it on gene...@james.apache.org and announce@apache
> >  - 6. Add a notice to the Apache website, if present
> >  - 7. Remove releases from downloads.apache.org
> >  - 8. Add notices on the Apache release archives (example
> > https://archive.apache.org/dist/ant/antidote/ <
> > https://archive.apache.org/dist/ant/antidote/>)
> >
> > Best regards,
> >
> > Benoit Tellier
> > ||
> >
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>

Re: Retire Apache James Postage ?

[Jean Helou]

Hi benoit

While looking at unmaintained pieces of code that might have now better
> alternatives, being hosted under the Apache James umbrella, I cannot but
> think about Apache Postage.
>
> Given the maturity of the project, the presence of alternatives, and the
> absence of development, in the absence of mainteners, it could be wise
> to consider retiring it.
>
> Thoughts?
>

I had no idea it even existed :) I feel there is no greater bane than
dead/unused/unmaintained code
+1 for dropping it

jean

Re: Retire Apache James Hupa ?

[Jean Helou]

I think this is an excellent idea ! +1

thank you benoit !
jean


On Mon, Jul 19, 2021 at 10:16 AM btell...@apache.org 
wrote:

> Hello all,
>
> While fixing our download pages following some infra bot complains, I
> ended up fixing the downloads for Apache James Hupa.
>
>  - The latest release (0.3.0) dates from 2012 which is an eternity in
> computing.
>  - The latest tag on Github is 0.0.3
>  - The pom references 0.0.5-SNAPSHOT suggesting that 0.0.4 release is
> lost :-(
>  - This repository is crippled by multiple CVEs (quick dependabot review):
>   - CVE-2021-29425 (commons-io)
>   - GHSA-m6cp-vxjx-65j6 CVE-2017-7656 CVE-2015-2080 CVE-2017-7657
> CVE-2019-10241 CVE-2019-10247 (Jetty server)
>   - CVE-2020-9447 (gwtupload)
>   - GHSA-g3wg-6mcf-8jj6 (jetty-webapp)
>   - CVE-2019-17571 (log4j)
>   - CVE-2016-131 CVE-2016-3092 (commons-fileupload)
>  - Sporadic activity since 2012
>  - Zero to no exchanges for several years on the mailing lists.
>
> From the Readme:
>
> > Hupa is able to discover most of the imap/smtp configuration based on
> the email domain part. When you are prompted to login, type your email
> address and wait few seconds, if you click on the gear button you can
> see the configuration discovered by Hupa, you can modify it if it does
> not match your email provider configuration. Then type your inbox
> password and you will be logged into your email provider servers.
>
> > Hupa is compatible with most email providers, gmail, yahoo, hotmail,
> outlook, exchange, james, etc.
>
> I fail to see the value added compared to other webmails like roundcube,
> rainloops to quote a few...
>
> As such, given that alternatives exists, given that the project is
> likely not mature, unmaintained and unsecure, I propose to retire this
> Apache James subproject.
>
> I will do research on procedures and best practices to do so. I guess a
> formal vote would be necessary. Likely contact Apache Labs were the
> project originated from in 2009...
>
> Best regards,
>
> Benoit TELLIER
>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Should we have a checkstyle for Scala?

[Jean Helou]

> I have a pull request to upgrade scalafix-maven-plugin version [1]. I hope
> it will be approvable by the maintainer.
>

I think there is little chance for it to be approved when there is
https://github.com/evis/scalafix-maven-plugin/pull/12 but you never know
If you are going to upgrade scalafix any why did you chose to stop at 0.9.25
instead of going for the latest release (0.9.29) ?


> I also have another request to James [2], that implement scalafix tool.
>

awesome !


> [1] https://github.com/evis/scalafix-maven-plugin/pull/15
> [2] https://github.com/apache/james-project/pull/538
>
> Cheers,
> Tung
>
> On Tue, Jul 13, 2021 at 6:59 PM Jean Helou  wrote:
>
> > > Yes, I did. I forked this plugin and manual change scala version on
> > local.
> > > As I wrote
> > >
> >
> > sorry I didn't catch that in your original email
> >
> > > "I got another issue: java.lang.NoSuchMethodError:
> > > scala.reflect.internal.util.ReusableInstance,
> > > scala/collection/IterableOnce"
> > > Maybe need more time to check.
> > >
> >
> > just a thought but the maven plugin hasn't been updated in a while and
> > pulls scalafix 0.9.23
> > https://github.com/scalacenter/scalafix/releases/tag/v0.9.26 mentions
> > scala
> > 2.13.5 with link to a PR[1] which explicitely mentions the rule you are
> > trying to use :)
> >
> > can you try upgrading scalafix version from 9.23 at least to 9.26 (even
> > better to 9.29  in your local build ?
> >
> > [1] https://github.com/scalacenter/scalafix/pull/1334/files
> >
> > cheers,
> > jean
> >
> > Thank you,
> > >
> > > On Tue, Jul 13, 2021 at 6:34 PM Jean Helou 
> wrote:
> > >
> > > > This is my configuration
> > > > >
> > > >
> > > > looks good to me, so it looks like the plugin is indeed pinned on
> scala
> > > > 2.12, you could clone the repo, change the plugin version to a custom
> > > > version, change references to scala 2.12 to scala 2.13 and run an
> > install
> > > > locally to check that having the plugin itself compiled against 2.13
> > > works.
> > > > if it does you can chime in on the PR to try and get it merged in an
> > > > official manner.
> > > > What I usually do in such cases is publish an unofficial version of
> the
> > > > tool on an unofficial repo and use that until the PR is merged. not
> > sure
> > > > how that flies for apache projects though especially since the plugin
> > > uses
> > > > a BSD3 license.
> > > > Since it's not a published artifact of the project maybe we are
> allowed
> > > to
> > > > do that as long as we publish the custom build to a public repo ? I
> > used
> > > to
> > > > publish on bintray not sure where to go now ..
> > > >
> > > > Regards,
> > > > Jean
> > > >
> > > >
> > > > > On Tue, Jul 13, 2021 at 5:21 PM Jean Helou 
> > > wrote:
> > > > >
> > > > > > Hi Tung,
> > > > > >
> > > > > >
> > > > > > > Very interesting with the rewrite feature of the scalafix tool.
> > > > > > > I tried it for the `jmap-rfc-8621` module. I got an
> incompatible
> > > > issue
> > > > > > with
> > > > > > > the scala version when trying Semantic Rules.
> > > > > > >
> > > > > >
> > > > > > According to the issue you link to, the plugin only works with
> > scala
> > > > 2.12
> > > > > > and according to the error message you report, the james code
> base
> > > uses
> > > > > > scala 2.13.
> > > > > > In this case it won't work unless the plugin is cross compiled
> for
> > > > 2.13.
> > > > > > What's intriguing is that the tips and tricks section [1] has a
> > > sample
> > > > > > configuration for the semantic db compiler  which seems ot hint
> > that
> > > > the
> > > > > > scala.version can be configured. but i actually doubt it would
> work
> > > > > >
> > > > > > can you share the configuration you used ?
> > > > > > [1]
> https://github.com/evis/scalafix-maven-plugin#tips-and-tricks
> > > > > >
> > > > > > I don't use the maven plugin itself, all my sc

Re: Should we have a checkstyle for Scala?

[Jean Helou]

> Yes, I did. I forked this plugin and manual change scala version on local.
> As I wrote
>

sorry I didn't catch that in your original email

> "I got another issue: java.lang.NoSuchMethodError:
> scala.reflect.internal.util.ReusableInstance,
> scala/collection/IterableOnce"
> Maybe need more time to check.
>

just a thought but the maven plugin hasn't been updated in a while and
pulls scalafix 0.9.23
https://github.com/scalacenter/scalafix/releases/tag/v0.9.26 mentions scala
2.13.5 with link to a PR[1] which explicitely mentions the rule you are
trying to use :)

can you try upgrading scalafix version from 9.23 at least to 9.26 (even
better to 9.29  in your local build ?

[1] https://github.com/scalacenter/scalafix/pull/1334/files

cheers,
jean

Thank you,
>
> On Tue, Jul 13, 2021 at 6:34 PM Jean Helou  wrote:
>
> > This is my configuration
> > >
> >
> > looks good to me, so it looks like the plugin is indeed pinned on scala
> > 2.12, you could clone the repo, change the plugin version to a custom
> > version, change references to scala 2.12 to scala 2.13 and run an install
> > locally to check that having the plugin itself compiled against 2.13
> works.
> > if it does you can chime in on the PR to try and get it merged in an
> > official manner.
> > What I usually do in such cases is publish an unofficial version of the
> > tool on an unofficial repo and use that until the PR is merged. not sure
> > how that flies for apache projects though especially since the plugin
> uses
> > a BSD3 license.
> > Since it's not a published artifact of the project maybe we are allowed
> to
> > do that as long as we publish the custom build to a public repo ? I used
> to
> > publish on bintray not sure where to go now ..
> >
> > Regards,
> > Jean
> >
> >
> > > On Tue, Jul 13, 2021 at 5:21 PM Jean Helou 
> wrote:
> > >
> > > > Hi Tung,
> > > >
> > > >
> > > > > Very interesting with the rewrite feature of the scalafix tool.
> > > > > I tried it for the `jmap-rfc-8621` module. I got an incompatible
> > issue
> > > > with
> > > > > the scala version when trying Semantic Rules.
> > > > >
> > > >
> > > > According to the issue you link to, the plugin only works with scala
> > 2.12
> > > > and according to the error message you report, the james code base
> uses
> > > > scala 2.13.
> > > > In this case it won't work unless the plugin is cross compiled for
> > 2.13.
> > > > What's intriguing is that the tips and tricks section [1] has a
> sample
> > > > configuration for the semantic db compiler  which seems ot hint that
> > the
> > > > scala.version can be configured. but i actually doubt it would work
> > > >
> > > > can you share the configuration you used ?
> > > > [1] https://github.com/evis/scalafix-maven-plugin#tips-and-tricks
> > > >
> > > > I don't use the maven plugin itself, all my scala projects use either
> > SBT
> > > > or mill so I don't have this layer of issues.
> > > >
> > > > Regards,
> > > > Jean
> > > >
> > > > [1] https://github.com/evis/scalafix-maven-plugin/pull/12
> > > > >
> > > > > Regards,
> > > > > Tung
> > > > >
> > > > > On Fri, Jun 25, 2021 at 2:44 PM Jean Helou 
> > > wrote:
> > > > >
> > > > > > Hi tung,
> > > > > >
> > > > > > Have you looked at scalafix[1]? it seems to be much more actively
> > > > > > maintained than scalastyle and I feel that it's scope overlaps
> > > > > scalatyle's
> > > > > > quite a bit...
> > > > > >
> > > > > > [1]https://github.com/scalacenter/scalafix
> > > > > >
> > > > > > Regards,
> > > > > > jean
> > > > > >
> > > > > >
> > > > > > On Fri, Jun 25, 2021 at 8:14 AM tungtv...@gmail.com <
> > > > tungtv...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello guys,
> > > > > > >
> > > > > > > I realize the current checkstyle.xml file doesn't support the
> > Scala
> > > > > > > convention.
> > > > > > > Should we have a checkstyle for that?
> > > > > > >
> > > > > > > My suggestion:
> > > > > > > - https://github.com/scalastyle/scalastyle
> > > > > > >
> > > > > > > - https://github.com/scalastyle/scalastyle-maven-plugin
> > > > > > >
> > > > > > >
> > > > > > > Best Regards,
> > > > > > >
> > > > > > > Tung, Van TRAN
> > > > > > >
> > > > > > >
> > > > > > >
> > > -
> > > > > > > To unsubscribe, e-mail:
> server-dev-unsubscr...@james.apache.org
> > > > > > > For additional commands, e-mail:
> > server-dev-h...@james.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Tung, Tran Van
> > > > > *Phone:* (+84) 35 757 6258
> > > > > *Skype:* tung.tv202
> > > > >
> > > >
> > >
> > >
> > > --
> > > Tung, Tran Van
> > > *Phone:* (+84) 35 757 6258
> > > *Skype:* tung.tv202
> > >
> >
>
>
> --
> Tung, Tran Van
> *Phone:* (+84) 35 757 6258
> *Skype:* tung.tv202
>

Re: Should we have a checkstyle for Scala?

[Jean Helou]

This is my configuration
>

looks good to me, so it looks like the plugin is indeed pinned on scala
2.12, you could clone the repo, change the plugin version to a custom
version, change references to scala 2.12 to scala 2.13 and run an install
locally to check that having the plugin itself compiled against 2.13 works.
if it does you can chime in on the PR to try and get it merged in an
official manner.
What I usually do in such cases is publish an unofficial version of the
tool on an unofficial repo and use that until the PR is merged. not sure
how that flies for apache projects though especially since the plugin uses
a BSD3 license.
Since it's not a published artifact of the project maybe we are allowed to
do that as long as we publish the custom build to a public repo ? I used to
publish on bintray not sure where to go now ..

Regards,
Jean


> On Tue, Jul 13, 2021 at 5:21 PM Jean Helou  wrote:
>
> > Hi Tung,
> >
> >
> > > Very interesting with the rewrite feature of the scalafix tool.
> > > I tried it for the `jmap-rfc-8621` module. I got an incompatible issue
> > with
> > > the scala version when trying Semantic Rules.
> > >
> >
> > According to the issue you link to, the plugin only works with scala 2.12
> > and according to the error message you report, the james code base uses
> > scala 2.13.
> > In this case it won't work unless the plugin is cross compiled for 2.13.
> > What's intriguing is that the tips and tricks section [1] has a sample
> > configuration for the semantic db compiler  which seems ot hint that the
> > scala.version can be configured. but i actually doubt it would work
> >
> > can you share the configuration you used ?
> > [1] https://github.com/evis/scalafix-maven-plugin#tips-and-tricks
> >
> > I don't use the maven plugin itself, all my scala projects use either SBT
> > or mill so I don't have this layer of issues.
> >
> > Regards,
> > Jean
> >
> > [1] https://github.com/evis/scalafix-maven-plugin/pull/12
> > >
> > > Regards,
> > > Tung
> > >
> > > On Fri, Jun 25, 2021 at 2:44 PM Jean Helou 
> wrote:
> > >
> > > > Hi tung,
> > > >
> > > > Have you looked at scalafix[1]? it seems to be much more actively
> > > > maintained than scalastyle and I feel that it's scope overlaps
> > > scalatyle's
> > > > quite a bit...
> > > >
> > > > [1]https://github.com/scalacenter/scalafix
> > > >
> > > > Regards,
> > > > jean
> > > >
> > > >
> > > > On Fri, Jun 25, 2021 at 8:14 AM tungtv...@gmail.com <
> > tungtv...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hello guys,
> > > > >
> > > > > I realize the current checkstyle.xml file doesn't support the Scala
> > > > > convention.
> > > > > Should we have a checkstyle for that?
> > > > >
> > > > > My suggestion:
> > > > > - https://github.com/scalastyle/scalastyle
> > > > >
> > > > > - https://github.com/scalastyle/scalastyle-maven-plugin
> > > > >
> > > > >
> > > > > Best Regards,
> > > > >
> > > > > Tung, Van TRAN
> > > > >
> > > > >
> > > > >
> -
> > > > > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > > > > For additional commands, e-mail: server-dev-h...@james.apache.org
> > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Tung, Tran Van
> > > *Phone:* (+84) 35 757 6258
> > > *Skype:* tung.tv202
> > >
> >
>
>
> --
> Tung, Tran Van
> *Phone:* (+84) 35 757 6258
> *Skype:* tung.tv202
>

Re: Should we have a checkstyle for Scala?

[Jean Helou]

Hi Tung,


> Very interesting with the rewrite feature of the scalafix tool.
> I tried it for the `jmap-rfc-8621` module. I got an incompatible issue with
> the scala version when trying Semantic Rules.
>

According to the issue you link to, the plugin only works with scala 2.12
and according to the error message you report, the james code base uses
scala 2.13.
In this case it won't work unless the plugin is cross compiled for 2.13.
What's intriguing is that the tips and tricks section [1] has a sample
configuration for the semantic db compiler  which seems ot hint that the
scala.version can be configured. but i actually doubt it would work

can you share the configuration you used ?
[1] https://github.com/evis/scalafix-maven-plugin#tips-and-tricks

I don't use the maven plugin itself, all my scala projects use either SBT
or mill so I don't have this layer of issues.

Regards,
Jean

[1] https://github.com/evis/scalafix-maven-plugin/pull/12
>
> Regards,
> Tung
>
> On Fri, Jun 25, 2021 at 2:44 PM Jean Helou  wrote:
>
> > Hi tung,
> >
> > Have you looked at scalafix[1]? it seems to be much more actively
> > maintained than scalastyle and I feel that it's scope overlaps
> scalatyle's
> > quite a bit...
> >
> > [1]https://github.com/scalacenter/scalafix
> >
> > Regards,
> > jean
> >
> >
> > On Fri, Jun 25, 2021 at 8:14 AM tungtv...@gmail.com  >
> > wrote:
> >
> > > Hello guys,
> > >
> > > I realize the current checkstyle.xml file doesn't support the Scala
> > > convention.
> > > Should we have a checkstyle for that?
> > >
> > > My suggestion:
> > > - https://github.com/scalastyle/scalastyle
> > >
> > > - https://github.com/scalastyle/scalastyle-maven-plugin
> > >
> > >
> > > Best Regards,
> > >
> > > Tung, Van TRAN
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > > For additional commands, e-mail: server-dev-h...@james.apache.org
> > >
> > >
> >
>
>
> --
> Tung, Tran Van
> *Phone:* (+84) 35 757 6258
> *Skype:* tung.tv202
>

Re: Should we have a checkstyle for Scala?

[Jean Helou]

Hi tung,

Have you looked at scalafix[1]? it seems to be much more actively
maintained than scalastyle and I feel that it's scope overlaps scalatyle's
quite a bit...

[1]https://github.com/scalacenter/scalafix

Regards,
jean


On Fri, Jun 25, 2021 at 8:14 AM tungtv...@gmail.com 
wrote:

> Hello guys,
>
> I realize the current checkstyle.xml file doesn't support the Scala
> convention.
> Should we have a checkstyle for that?
>
> My suggestion:
> - https://github.com/scalastyle/scalastyle
>
> - https://github.com/scalastyle/scalastyle-maven-plugin
>
>
> Best Regards,
>
> Tung, Van TRAN
>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Code organisation: makes guice application easier to discover

[Jean Helou]

Thank you benoit for clarifying the goals you are reaching for, it's very
clear for me now
I agree with the goal, the plan and the first step :D

regards,
Jean


On Wed, Jun 9, 2021 at 7:15 AM btell...@apache.org 
wrote:

> Hello Jean!
>
> On 08/06/2021 17:21, Jean Helou wrote:
> > Hi benoit !
> >
> > On my crusade to reorganize James code related to Guice apps (and
> >> promote their adoption), I come to the next item of the tick list (after
> >> ZIP packaging, JIB packaging to enable distribution).
> >>
> > These were great improvements
> >
> >
> >> I would like to make those application easier to find in the source
> tree.
> >>
> > and this one promises to be too !
> >
> >
> >> Here would be the principles I would like to enforce:
> >>  - All server applications should be collocated under the same directory
> >>  - Server application modules should be clearly separated from guice
> >> module declarations
> >>
> >> I do not intend to fully reorder James directories just now but rather
> >> take small steps in a globally consensual direction. As such I propose
> >> the following directory layout as a first step:
> >>
> > This first step would be nice and at least would make it easy to add a
> link
> > to all the provided james-server builds in the readme, The instructions
> for
> > running each provided build could then go in the readme of each build and
> > reduce the clutter in the primary readme, I'm all for it :D
> >
> > But if you have a `first step` then you must also have a goal, it would
> be
> > easier to assess the usefulness of the first step if we shared the vision
> > of the goal.
> The goal would be that the directory structure :
>  - reflects the project goals
>
>  - is honest toward the progress toward these goal
>
>  - allow easily understanding the project architecture
>
>  - and allow to easily find the project deliveries, tests, etc...
>
>  - [add your goals here]
>
> I am starting with this last one, likely the easiest and less
> controversial to start with.
>
> >
> > I am partly aware of the history of the project and that some of the top
> > level directories are projects which used to live in different
> repositories
> > and have been merged back. so james git is essentially a mono-repo and
> > that's fine
> >
> > According to the website (which seems to have lost part of its colorful
> > landing page by the way) the primary components are :
> > - Server
> +1
> > - Mailets
>
> +1, intended to be a generic API just like servlets are. Their use was
> intended outside of James. (source Dany Angus at ACEU 2019)
>
> This goal and its results could be re-assessed but I am not wishing to
> do this.
>
> > - Mailbox
>
> Regarding that one I do think mailbox do not make sense without a server
> to use it. I would be in favor to move it in the /server directory. A
> topic for another day.
>
> This module is tightly coupled with the protocols consuming it,
> underwent heavy changes during the JMAP implementations, and overall is
> a critical server component we need full control upon (eg performance).
>
> > - Protocols
> I do think protocols without a hard dependency on mailbox or server
> could be considered top level, but protocols like IMAP with a hard
> dependency on mailbox likely can't.
> > - MPT
> MPT is a weird thing. It mixes a set of tools along side with a test
> suite exercising other components. I think mpt/impl would deserve to be
> relocated.
>
> Maybe to /server/protocols/imap-tests /server/protocols/smtp-tests &
> /server/protocols/managesive-tests ?
>
> >
> > So I would expect these to be the top level directories in the project,
> > along with a couple supporting directories such as
> > - docs
> > - shared-libraries
> > maybe
> > - examples
> >
> > I included shared-libraries  because I assume a reason for some of the
> > directories being top level is that they are used by multiple top level
> > modules :
> > my *guess* would be
> > - json
> > - core
> > - metrics
> > - testing/base
> >
> > Others sound like they should be reintegrated in the server directory
> tree
> > (guesstimate again) :
> > - backends-common
> Project with this dependency would also need to be in the server
> directory tree.
>
> This match the move of mailbox into server.
> > - event-bus
> > - event-sourcing
> > - benchmarks
> > - dockerfiles
> With the exception that it now mostly contai

Re: What's the reason for having Username's constructor private?

[Jean Helou]

Hello Andreas,

We have an environment where usernames can be *any* character thus the
> rules enforced by Username does not apply to us. We were thinking about
> subclassing it, until we saw it has a private constructor signaling that it
> isn't meant for extending.
>

Do you mean org.apache.james.core.Username ?

If I understand correctly, the only character restriction there is you
can't put an @ in the local part ...  even
org.apache.james.core.Username#fromLocalPartWithoutDomain[1] won't bypass
that ( and of course the usual not null, not empty)
As you can see in
https://github.com/apache/james-project/blob/master/core/src/main/java/org/apache/james/core/Username.java#L132,
a Username has a mapping to a org.apache.james.core.MailAddress and this
mapping is used in multiple places. Allowing @ in the username would break
that conversion.

Can you provide a more detailed picture of what you are trying to achieve
and why you would want @ as part of the username ?

(I'll admit that the current check is overly restrictive as it doesn't
account for quoting in the local part [2] which allows using @ in a quoted
string or escaping @ using a \)

[1]
https://github.com/apache/james-project/blob/master/core/src/main/java/org/apache/james/core/Username.java#L71
[2] https://datatracker.ietf.org/doc/html/rfc3696#section-3

Cheers,
Jean


>
>

> I propose to change this constructor to be protected so one may extend
> this class with custom validation-logic for the userName. How are others
> handling this problem?
>
> --
> *Andreas Joseph Krogh*
> CTO / Partner - Visena AS
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Code organisation: makes guice application easier to discover

[Jean Helou]

Hi benoit !

On my crusade to reorganize James code related to Guice apps (and
> promote their adoption), I come to the next item of the tick list (after
> ZIP packaging, JIB packaging to enable distribution).
>

These were great improvements


> I would like to make those application easier to find in the source tree.
>

and this one promises to be too !


> Here would be the principles I would like to enforce:
>  - All server applications should be collocated under the same directory
>  - Server application modules should be clearly separated from guice
> module declarations
>
> I do not intend to fully reorder James directories just now but rather
> take small steps in a globally consensual direction. As such I propose
> the following directory layout as a first step:
>

This first step would be nice and at least would make it easy to add a link
to all the provided james-server builds in the readme, The instructions for
running each provided build could then go in the readme of each build and
reduce the clutter in the primary readme, I'm all for it :D

But if you have a `first step` then you must also have a goal, it would be
easier to assess the usefulness of the first step if we shared the vision
of the goal.

I am partly aware of the history of the project and that some of the top
level directories are projects which used to live in different repositories
and have been merged back. so james git is essentially a mono-repo and
that's fine

According to the website (which seems to have lost part of its colorful
landing page by the way) the primary components are :
- Server
- Mailets
- Mailbox
- Protocols
- MPT

So I would expect these to be the top level directories in the project,
along with a couple supporting directories such as
- docs
- shared-libraries
maybe
- examples

I included shared-libraries  because I assume a reason for some of the
directories being top level is that they are used by multiple top level
modules :
my *guess* would be
- json
- core
- metrics
- testing/base

Others sound like they should be reintegrated in the server directory  tree
(guesstimate again) :
- backends-common
- event-bus
- event-sourcing
- benchmarks
- dockerfiles
- grafana-reporting

I have not taken the time to do a proper dependency analysis, this is just
how I expect to find the dependencies so I may be wrong :D

If this organization is close to what you have in mind then the first step
is indeed the best.

However If the goal is to push server in the spotlight  then we should move
the apps directory to the top level and move mpt, mailets, mailbox,
protocols and mpt etc inside server
Note that this doesn't prevent documenting them, nor publishing the
artifacts nor enforcing proper dependencies

jean

Re: Removal of stale branches for apache/james projects

[Jean Helou]

Thank you Benoit !

On Mon, Jun 7, 2021 at 4:01 AM btell...@apache.org 
wrote:

> Hey there,
>
> I deleted the before mentioned branches.
>
> I opened https://issues.apache.org/jira/browse/INFRA-21969 for the empty
> project removal.
>
> Cheers,
>
> Benoit
>
> On 02/06/2021 20:04, Jean Helou wrote:
> > On Wed, Jun 2, 2021 at 9:54 AM btell...@apache.org 
> > wrote:
> >
> >> A bit of house cleaning might be beneficial Without objections, I
> >> will clean these brnches in a few days.
> >>
> > +9001 (cuz it's over 9000) !
> >
> >
> >> Furthermore the following projects do not have code associated to them:
> >>
> >> - https://github.com/apache/james-protocols
> >> - https://github.com/apache/james-mpt
> >> - https://github.com/apache/james-mailbox
> >> - https://github.com/apache/james-mailet
> >> - https://github.com/apache/james-imap
> >>
> >> Maybe we should open a ticket on the INFRA to plan for their removal?
> >> (these projects had been fused into apache/james-project...)
> >>
> > +1
> >
> > jean
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Removal of stale branches for apache/james projects

[Jean Helou]

On Wed, Jun 2, 2021 at 9:54 AM btell...@apache.org 
wrote:

>
> A bit of house cleaning might be beneficial Without objections, I
> will clean these brnches in a few days.
>

+9001 (cuz it's over 9000) !


> Furthermore the following projects do not have code associated to them:
>
> - https://github.com/apache/james-protocols
> - https://github.com/apache/james-mpt
> - https://github.com/apache/james-mailbox
> - https://github.com/apache/james-mailet
> - https://github.com/apache/james-imap
>
> Maybe we should open a ticket on the INFRA to plan for their removal?
> (these projects had been fused into apache/james-project...)
>

+1

jean

Re: Github token for Apache JAMES ASF build

[Jean Helou]

> I just set up additional builds for JSPF, JSPF, JSIEVE and MIME4J
> projects, and successfully used the asf-ci tokens.
>
> Is there any reasons not to switch these two projects to use the ASF
> tokens too?
>

Not that I know of, but it is possible that the tokens were unavailable
when the builds were first created ( I think I read something like that in
some infra tickets when I initially investigated rebooting the apache CI)

Jean

[jira] [Commented] (JAMES-3569) RecipientRewriteTable sometimes drops attributes from emails

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17330234#comment-17330234
 ] 

Jean Helou commented on JAMES-3569:
---

Hmm  I don't know, the processing for rewrites to local addresses vs remote 
addresses is really very different : 

the RewriteTableProcessors splits local adresses from remote addresses, it sets 
the local addresses on the existing email and creates a new mail with the 
remote recipients which it then sends through the mailContext. this behaviour 
is explicitely asserted in the processor tests so it seems to be intentional 
but I can't really tell why it is/was necessary ...

The fix we made with  [~matthieu] (PR incoming) doesn't change this behaviour 
(maybe matthieu has a better undertanding why it was necessary)

(our fix also contains a bit of boyscouting and removes mockito usage in the 
processor test along with adding an assertion that the attributes are not lost )

> RecipientRewriteTable sometimes drops attributes from emails
> 
>
> Key: JAMES-3569
> URL: https://issues.apache.org/jira/browse/JAMES-3569
> Project: James Server
>  Issue Type: Bug
>  Components: SMTPServer
>Affects Versions: 3.6.0
>    Reporter: Jean Helou
>Priority: Major
>
> When a mail has a recipient with a mapping to a remote email address, 
> RecipientRewriteTable creates a new mail and copies over a few fields.
> Unfortunately it doesn't copy all the fields and in particular it drops the 
> mail attributes that have been computed by the pipeline up to this point.  
> For recipients which are rewritten to a local address there is no information 
> loss.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Created] (JAMES-3569) RecipientRewriteTable sometimes drops attributes from emails

[Jean Helou (Jira)]

Jean Helou created JAMES-3569:
-

 Summary: RecipientRewriteTable sometimes drops attributes from 
emails
 Key: JAMES-3569
 URL: https://issues.apache.org/jira/browse/JAMES-3569
 Project: James Server
  Issue Type: Bug
  Components: SMTPServer
Affects Versions: 3.6.0
Reporter: Jean Helou


When a mail has a recipient with a mapping to a remote email address, 
RecipientRewriteTable creates a new mail and copies over a few fields.

Unfortunately it doesn't copy all the fields and in particular it drops the 
mail attributes that have been computed by the pipeline up to this point.  For 
recipients which are rewritten to a local address there is no information loss. 
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Commented] (JAMES-3330) Clarify Recipient Rewrite Table Configuration (recipientrewritetable.xml)

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329408#comment-17329408
 ] 

Jean Helou commented on JAMES-3330:
---

it feels like this has been completed, can it be closed ?

> Clarify Recipient Rewrite Table Configuration (recipientrewritetable.xml)
> -
>
> Key: JAMES-3330
> URL: https://issues.apache.org/jira/browse/JAMES-3330
> Project: James Server
>  Issue Type: Sub-task
>Reporter: David Leangen
>Priority: Major
>
> See [JAMES-3235]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Commented] (JAMES-3565) Publish and use apache/james docker images

[Jean Helou (Jira)]

[ 
https://issues.apache.org/jira/browse/JAMES-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17322671#comment-17322671
 ] 

Jean Helou commented on JAMES-3565:
---

Here is what I had in mind when I answered on the mailing list: 
 * switch image creation to jib so that it becomes a part of the build and is 
easier to automate in jenkins
 * add a step that automatically builds the images for master in jenkins (to 
make sure that it always builds correctly so we don't get stuck on the day of 
the release) 
 * (optional or maybe only for tags) push the images to a *staging* images 
repository [1] but not on docker hub,
 * (only for tags) add a step with a manual trigger that retags the tag images 
and pushes them to docker hub

 

What I mean by staging images repository is an image repository not on docker 
hub, possibly hosted by apache if there is one already or to look into finding 
one. this would enable having `pre-release` or even `snapshot` images for 
various kind of automation while making it clear that these are unreleased 
possibly unfinished products. the equivalent of the SNAPSHOT jars on maven 
central.

> Publish and use apache/james docker images
> --
>
> Key: JAMES-3565
> URL: https://issues.apache.org/jira/browse/JAMES-3565
> Project: James Server
>  Issue Type: Task
>  Components: docker
>Reporter: Benoit Tellier
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> https://www.mail-archive.com/server-dev@james.apache.org/msg70107.html
> {code:java}
> Hello there,
> Following the 3.6.0 release I experimented with dockerhub and pushed the
> following images:
> - apache/james:memory-3.x.x built from
> https://github.com/apache/james-project/tree/master/dockerfiles/run/guice/memory
> - apache/james:jpa-3.x.x built from
> https://github.com/apache/james-project/tree/master/dockerfiles/run/guice/jpa
> - apache/james:demo-3.x.x built from
> https://github.com/apache/james-project/tree/master/dockerfiles/run/guice/provisioned
> - apache/james:cassandra-3.x.x build from
> https://github.com/apache/james-project/tree/master/dockerfiles/run/guice/cassandra
> - apache/james:distributed-3.x.x built from
> https://github.com/apache/james-project/tree/master/dockerfiles/run/guice/cassandra-rabbitmq
> I however do not have the rights to manage the repository, hence can't
> add a description / overview just like apache/tika repo did. I opened
> INFRA-21718 regarding this.
> I also propose to timely edit the existing configuration to:
>  - No longer use linagora repository for Apache James docker images
> distribution
>  - Only rely on released docker images (no longer branch master) to
> better fit in Apache release policies
> [... the JIB proposition is out of scope here ...]
> {code}
> https://issues.apache.org/jira/browse/INFRA-21718



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: apache/james on docker hub.

[Jean Helou]

Hi benoit,


> This would look like what Juhan proposed in
> https://issues.apache.org/jira/browse/JAMES-3565 ...
>
> Thoughts?
>

interesting, I'll comment with ideas over there thanks


jean

Re: [Results] [VOTE] Call for vote: Apache James 3.6.0 (round 3)

[Jean Helou]

>
> I am about to conclude the 3.6.0 release (publish the website & announce)
>

 :rocket:

However I would like the website to advertise the maintained, apache
> images rather than the unmaintained ones of an undisclosed third party.
>
> I imagine the changes in
> https://github.com/apache/james-project/pull/389 to be consensual.
>
> If the feedback is positive, I can finish the 3.6.0 process as off
> today.


I don't see how 389 blocks the release but I'm ok with the PR anyway :)

Cheers,
Jean


> Otherwise I will proceed, likely on Friday, if not next week when
> I have time... - I must confess being impatient to be done with it.
>
> Cheers,
>
> Benoit
>
> Le 14/04/2021 à 10:34, Tellier Benoit a écrit :
> > Hi all,
> >
> > I am happy to announce you the vote for the 3.6.0 release did succeed.
> >
> > The release received 8 positive votes, 3 of them being binding.
> >
> > Thanks to all contributors, developers and committers who made this
> > possible!
> >
> > In the coming hours, I will finalize the release process, namely:
> >
> > - Publish the maven artifacts
> > - Upgrade the download page and the (old) website
> > - Announce the release
> > - Apache docker images (convenience downloads, non official release
> > artifacts CF Apache policy regarding docker images)
> >
> > Voting thread:
> > https://www.mail-archive.com/server-dev@james.apache.org/msg70025.html
> >
> > Cheers,
> >
> > PMC member name
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: apache/james on docker hub.

[Jean Helou]

> Following the 3.6.0 release I experimented with dockerhub and pushed the
> following images:
>

 THANK YOU ! (yes with caps because that's how grateful I am !)

I also propose to timely edit the existing configuration to:
>  - No longer use linagora repository for Apache James docker images
> distribution
>
 - Only rely on released docker images (no longer branch master) to
> better fit in Apache release policies
>

I saw the corresponding PR :+1:


> To be fairly honest building these images took me too much time. I would
> highly appreciate automation of docker images construction as part of
> the maven build. A way to do it is JIB. [1]


That's what I used to build my pure SMTP relay server assembly, it's being
polished but jib works great :)

demonstrate building a james
> server image variation for a tierce project so I propose to do something
> similar, except without requiring a docker daemon. It includes CLI,
> glowroot, etc... We had been using this JIB generated image on
> production without problems. I propose we target something similar for
> James docker image generation. As part of the release process we would
> then need to load, then push the maven build results. This would lead to
> the removal of /dockerfiles folder (more or less).
>

Since  docker images are not considered release artifacts under apache
policy (if I understand correctly) would it make sense to try and fully
automate the image building post release ?
add a piece to the jenkinsfile that monitors only tags, and builds the jib
image downloading the artifacts from maven central


> [1]
>
> https://github.com/linagora/openpaas-james/tree/master/openpaas-james/apps/memory
> README and pom.xml are of interest.
>
> Cheers,
>
> Benoit
>
> Le 11/12/2020 à 16:29, Tellier Benoit a écrit :
> > You're right,
> >
> > I did not follow up on this.
> >
> > I just created the INFRA ticket.
> > https://issues.apache.org/jira/browse/INFRA-21180
> >
> > As far as I understand, this is not an official release channel but
> > rather a convenience one.
> >
> > Cheers,
> >
> > Benoit
> >
> > Le 11/12/2020 à 15:18, Jean Helou a écrit :
> >>> I propose to ask for rights on apache/james repository
> >>
> >> I'm revisiting this thread after seeing
> >> https://www.docker.com/blog/supporting-open-source-projects-at-docker
> which
> >> reminded me of this effort and I was wondering if an INFRA ticket has
> been
> >> opened for james (of if there is a trace of the progress somewhere)
> >> I'm not sur exactly how the program works but if it whitelists prefixes,
> >> Apache is most likely already in it however in light of the work on CI
> >> builds on apache infrastructure, I am wondering if/how the credentials
> for
> >> pushing form CI can be setup and I was hoping to drop a comment on the
> >> corresponding ticket.
> >>
> >> jean
> >>
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: [Vote] Call for vote: Apache Mime4J 0.8.4

[Jean Helou]

Hi benoit,

The changelog has not been updated which is a bit sad (and i know it's
painful to do but it's s helpful for developers who use the library)
On a semi-related note : the release notes have not been updated either but
that last file actually seems to be totally out of sync with the repo : it
mentions an inexistant 0.9.0 and has not been modified for the last 4 years
... maybe delete it ?

jean

On Wed, Apr 14, 2021 at 9:09 AM Tellier Benoit  wrote:

> Hi,
>
> Following a user request (https://issues.apache.org/jira/browse/MIME4J-297
> ),
>
> I would like to propose a vote for 0.8.4 release of the MIME4J library.
>
> You can find the maven release staged in repository.apache.org as the
> artifact #1053:
> https://repository.apache.org/content/repositories/orgapachejames-1053/
>
>
> Voting rules:
>  - This is a majority approval: this release may not be vetoed.
>  - A quorum of 3 binding votes is required
>  - The vote starts at Wedneday 14th of April 2021, 2pm UTC+7
>  - The vote ends at Wednesday 21th of April 2021, 2pm UTC+7
>
> You can answer to it just with +1 and -1. Down-votes may be motivated.
>
> 3 binding votes are expected move forward on this release.
>
> Cheers,
>
> Benoit Tellier
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: [VOTE] Call for vote: Apache James 3.6.0 (round 3)

[Jean Helou]

go ! +1 :)

On Tue, Apr 6, 2021 at 10:27 AM Antoine Duprat  wrote:

> +1
>
> Le mar. 6 avr. 2021 à 09:04, Tellier Benoit  a écrit
> :
>
> > I would like to propose a new vote 3.6.0 release of the Apache James
> > server.
> >
> > After an initial mistake (sorry), a failed vote (could not achieve a PMC
> > quorum), and more issues I did not yet complain about (closing the maven
> > artifact #1051 failed for a missing MD5 signature - network issue?),
> > here we are with one more vote !!!
> >
> > You can find:
> >
> > - The maven release staged in repository.apache.org as the artifact
> #1052:
> > https://repository.apache.org/content/repositories/orgapachejames-1052/
> > - The changelog for 3.6.0:
> >  https://github.com/apache/james-project/blob/master/CHANGELOG.md
> > - The compatibility instructions/upgrade recommendation:
> >
> >
> https://github.com/apache/james-project/blob/master/upgrade-instructions.md#360-version
> >
> > Voting rules:
> >  - This is a majority approval: this release may not be vetoed.
> >  - A quorum of 3 binding votes is required
> >  - The vote starts at Tuesday 6th of April 2021, 2pm UTC+7
> >  - The vote ends at Tuesday 13th of March 2021, 2pm UTC+7
> >
> > You can answer to it just with +1 and -1. Down-votes shall be motivated.
> >
> > 3 binding votes are expected move forward on this release.
> >
> > Cheers,
> >
> > Benoit Tellier
> >
> > Le 30/03/2021 à 17:16, Tellier Benoit a écrit :
> > > The voting period is over.
> > >
> > > Only one upvote had been collected this time. We did not achieve the 3
> > > PMC quorum, and other contributors did not vote as well.
> > >
> > > I will re-trigger a vote, this is a good opportunity to integrate
> > > https://github.com/apache/james-project/pull/349
> > >
> > > Cheers,
> > >
> > > Benoit
> > >
> > > Le 22/03/2021 à 15:16, Tellier Benoit a écrit :
> > >> Hi,
> > >>
> > >> I would like to propose a new vote 3.6.0 release of the Apache James
> > server.
> > >>
> > >> You can find:
> > >>
> > >>  - The maven release staged in repository.apache.org as the
> > >> artifact #1050:
> > >>
> https://repository.apache.org/content/repositories/orgapachejames-1050/
> > >>  - The changelog for
> > >> 3.6.0:
> > >> https://github.com/apache/james-project/blob/master/CHANGELOG.md
> > >>  - The compatibility instructions/upgrade
> > >> recommendation:
> > >>
> >
> https://github.com/apache/james-project/blob/master/upgrade-instructions.md#360-version
> > >>
> > >> Voting rules:
> > >>  - This is a majority approval: this release may not be vetoed.
> > >>  - A quorum of 3 binding votes is required
> > >>  - The vote starts at Monday 22th of March 2021, 4pm UTC+7
> > >>  - The vote ends at Monday 29th of March 2021, 4pm UTC+7
> > >>
> > >> You can answer to it just with +1 and -1. Down-votes may be motivated.
> > >>
> > >> 3 binding votes are expected move forward on this release.
> > >>
> > >> Cheers,
> > >>
> > >> Benoit Tellier
> > >>
> > >> -
> > >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > >> For additional commands, e-mail: server-dev-h...@james.apache.org
> > >>
> > >
> > > -
> > > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > > For additional commands, e-mail: server-dev-h...@james.apache.org
> > >
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>

Re: [BUILD-UNSTABLE]: Job 'james/ApacheJames-Website/PR-3 [PR-3] [5]'

[Jean Helou]

Hi all,

Sorry about the spam, I opened a PR to try and improve the build flow for
james documentation to try and get the staged documentation published
automatically after each build of james project.

Unfortunately the Jenkinsfile currently in the staging branch of james-site
is out of date (the JDK name has changed) and It took me these 5 tries to
realize that none of the changes I was making to the Jenkinsfile were
accounted for because:

‘Jenkinsfile’ has been modified in an untrusted revision


I'm not sure what the best way to proceed is : this limitation can be
disabled (would be easiest since I may have further adjustments to make),
or an official apache committer can push to an alternate PR to get it to
build with the changes.
Note that in this change I have also changed the email notifications to
only notify the list for the live and staging branches and notify the
requester for other branches.

best regards
jean

On Thu, Apr 1, 2021 at 9:43 PM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> BUILD-UNSTABLE: Job 'james/ApacheJames-Website/PR-3 [PR-3] [5]':
>
> Check console output at "https://ci-builds.apache.org/job/james/job/ApacheJames-Website/job/PR-3/5/;>james/ApacheJames-Website/PR-3
> [PR-3] [5]"
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Remove functionality that saves James metrics into Elasticsearch

[Jean Helou]

Hi Benoit,


> Regarding Prometheus setup, we have implemented an optional HTTP
> endpoint. I would recommend you to include the following lines in your
> webadmin.properties:
>
> extensions.routes=org.apache.james.webadmin.dropwizard.MetricsRoutes
>

Thanks!  I'll have a look.


> We should then:
>  - better document this setup
>  - (provide a docker-compose?)
>  - Update the metric boards in /grafana-reporting
>

Definitely since this is currently very hard to discover without being
told. Nothing in the name remotely suggests that this is potentially
compatible with Prometheus pulls and the name MetricsRoute doesn't seem to
be mentioned anywhere in the documentation.

With that said my target deployment[1] would have separate instances for :
 - relay SMTP ( with read access to the DB )
 - webadmin (administrative VPN )
 - mailbox and normal SMTP (user VPN, with read access to the user/domain
db)
because I like to keep my attack surfaces small and the relay must be
exposed to the general internet to be of any use so the less code it runs
the better.
I will have to see if I can configure my assembly to run a
restricted webmin with only the metrics routes.

So my ideal world would have documentation for both a pull and a push :p
I'll try to contribute the push side if I manage to make it work

Would you agree with such an approach?
>

Definitely, as I said I don't like ES for metrics either. A better
alternative is definitely ok.

> Having the metrics output to logs doesn't help much because of the amount
> > of processing required to get anything useful out of it (or I failed to
> > find how to easily ingest it please correct me)
>
> Logs for metrics had been introduced by Matthieu to get metrics out in
> the logs of a performance test platform, without the need to analyses
> the metrics "live".
>

That's interesting information ! Not the part about matthieu (no offense to
either of you), but the part about it not being intended for production
monitoring and possibly useful for bench post-mortems. I would definitely
like to see this mentioned in
https://james.apache.org/server/metrics.html or maybe in
https://james.apache.org/server/monitor-logging.html

We are moving away from JMX for the CLI, I see little reason to
> encourage its use here too.
>
> See https://www.cvedetails.com/cve/CVE-2017-12628/
>

I was under the impression that the prometheus jmx_exporter was deployed as
an agent to be able to access the MBeans directly but it seems to be
scraping from an exposed JMX tcp endpoint and I concur that this is not a
good idea.

Glowroot is an APM, allowing to :
>  - capture slow traces
>  - capture DB queries (globally and on slow traces)
>  - do some flame graphs captures (that might be more or less relevant
> based on the number of captures performed)


Yes and also
* Responsive UI with mobile support
* MBean attribute capture and **charts**
* Configurable alerting
* Historical rollup of all data (1m, 5m, 30m, 4h) with configurable
retention
not to mention a centralized collector which is why I said it overlaps
quite a bit with prometheus

It provides some insight that regular monitoring tools will not give.
>

> I would categorize it more as a developer diagnosis tool (which to me is
> invaluable for performance issues).
>

Do you run it in your production environments in addition to whatever
metrics reporter you otherwise use or only on dev/bench ? Are there any
good practices around it, is there any specific support for it in james ?

Jean

[1] This is not currently the case in my terraform/helm recipe but it is my
target.

Re: Remove functionality that saves James metrics into Elasticsearch

[Jean Helou]

Hello all,

I agree that ES is not the best tool for metrics but it's implemented,
documented and demonstrated
- https://james.apache.org/server/metrics.html
- https://james.apache.org/server/config-elasticsearch.html
-
https://github.com/apache/james-project/blob/master/server/container/guice/cassandra-guice/src/main/java/org/apache/james/CassandraJamesServerMain.java#L141
and that's only part of the mentions made throughout the documentation

So today it's the most discoverable[1] way to collect metrics for james
especially for people who are not expert "ops". If this is removed, I feel
documenting at least one recommended way to do it is necessary.
Having the metrics output to logs doesn't help much because of the amount
of processing required to get anything useful out of it (or I failed to
find how to easily ingest it please correct me) which leaves JMX.

Since JMX is a JVM standard, people who know the JVM well enough will
either know or easily deduce that there is a generic way to export JMX
metrics to whatever tool they want. However a mention of  jmx_exporter[2]
for prometheus and jmxtrans [3] for other tools in the monitoring/jmx
sections of the documentation would be kind to non-jvm people who stumble
on apache james. Ideally having a demo setup/tutorial would be nice [4]

One other thing regarding monitoring is that the README mentions Glowroot
[5] but there is no additional mention in the documentation. I am not
an ops expert so I can't help but wonder how that fits with the ES exported
metric ...
The glowroot documentation mentions a lot of features some of which overlap
with prometheus. The project doesn't seem very mature:  even if it was
started 6 years ago, there have been very few contributions in the last
year, the highest released version is 0.13.6 (james uses 0.13.4) and the
documentation is quite sparse. I am not sure this is a good recommendation
to put in the readme of the james project.

Jean

[1]
https://www.google.com/search?q=monitoring+apache+james+metrics=monitoring+apache+james+metrics
[2] https://github.com/prometheus/jmx_exporter
[3] https://github.com/jmxtrans/jmxtrans
[4] on that note, one reason I have been very quiet lately is that I am
working on a helm chart and a terraform recipe to deploy james to a k8s
cluster including all the associated software dependencies. I'm getting
there and I intend to contribute this back to the project. One of the
things I don't have yet is ... monitoring ;) Since k8s' monitoring tool of
choice is prometheus that's what I intended to use but I never used it
before and I'm only a casual contributor so I progress slowly
[5] https://glowroot.org/


On Thu, Mar 11, 2021 at 2:35 PM Juhan Aasaru  wrote:

> If the community agrees then I propose the following.
>
> * we deprecate this functionality in the next release (3.6.0) with a note
> that it would be completely removed in the next release (3.7.0)
> * this functionality (James storing history of its metrics to
> Elasticsearch) would continue to work with Elasticsearch 6 while rest of
> the Elasticsearch-related functionality would be upgraded to Elasticsearch
> 7
> * anyone still using it would have to run two different Elasticsearch
> instances (Elasticsearch 6 for history of metrics and Elasticsearch 7 for
> rest of James)
>
> Juhan
>
>
>
> Kontakt Juhan Aasaru () kirjutas kuupäeval N, 11. märts
> 2021 kell 15:17:
>
> > Hi!
> >
> > There is currently ongoing work to upgrade James to Elasticsearch 7.x
> > See https://github.com/apache/james-project/pull/328
> >
> > Current James can be configured to save James metrics to a separate index
> > in Elasticsearch.
> > And then Grafana dashboards can be configured to display the history of
> > these saved metrics.
> > For more detailed documentation refer the configuration parameter
> > "elasticsearch.metrics.reports.enabled" here:
> > https://james.apache.org/server/config-elasticsearch.html
> >
> > This functionality that gathers and publish metrics is provided by an
> > unmaintained library:
> > https://github.com/linagora/elasticsearch-metrics-reporter-java
> > This library is not compatible with Elasticsearch 7.x
> >
> > We are proposing to remove this functionality from James as
> > the industry standard is to use external tools that are purpose built for
> > pulling and storing the metrics.
> >
> > Users currently relying on this functionality would have to configure
> some
> > monitoring tool (like Prometheus) to regularly pull and store these
> metrics
> > if they want to continue displaying history of various James related
> > metrics over time.
> >
> > Please respond if you agree or are against removing it.
> >
> > Kind regards
> > Juhan Aasaru
> >
> >
>

Re: Time for a 3.6.0 release

[Jean Helou]

>
>  - First, previous release had been rejected once. And one time where we
> struggle to get the approval on time.
>  - Second, the numerous maven project counts makes the upload (mvn
> deploy / mvn release prepare / perform) long. I generally plan a day,
> and do it on my laptop not to copy my GPG keys around. Which negatively
> impact my other tasks.
>

I have been working with infrastructure provisioning quite a bit recently,
maybe using a terraform project to create a vm or even baremetal computers
(i know at least scaleway offers baremetal instances which are not VM but
actual computers not shared with other tenants) and release from there is
acceptable ?
connect over ssh using gpg socket forwarding (this way the secret key never
leaves your computer) either clone from github or upload the repo from your
machine and trigger the release process.
This at least would let you use your laptop to do stuff while the release
occurs, it also provides access to extremely powerful computers [1] which
could make the release process much faster.

[1] https://www.scaleway.com/fr/tarifs/#serveurs-cloud-bare-metal  GP-BM1-L
or HC-BM1-XS for €.33/hour/instance

Re: The case of javax.mail MimeMessage CopyOnWrite optimization

[Jean Helou]

Hi everyone,

In light of benoit's latest comments on the related PRs, I want to report
that I spent some time setting up an infra to do benchmarks against a full
server (mx only) I'm not an ops at core, I don't know james all that well
and spare time is limited and I got sidetracked quite a few times, so I
still haven't run the benchmarks but I'm getting there.

I am strongly in favor of integrating at least #282.

#285 is nice to have for me but I won't have time to investigate the config
file option rapidly.

Jean

Le jeu. 7 janv. 2021 à 18:04, Raphaël Ouazana  a
écrit :

> Hi,
>
> I completely agree with what Matthieu said.
>
> Correctness is needed, so we have to fix this issue.
>
> But before merging on master and/or deploying this version I would
> expect some Gatling runs to show which potential performance decrease we
> can expect on real use cases.
>
> Thanks,
>
> Raphaël.
>
> Le 04/01/2021 à 14:42, Matthieu Baechler a écrit :
> > Hi there,
> >
> > Thank you for bringing this topic to the mailing-list.
> >
> > To me safety and correctness is much more important than raw
> > performance so I would like the always-copy implementation to replace
> > the COW version.
> >
> > However, keep in mind that the JMH benchmark figures did not told the
> > full story about the consequences of this change and be ready to
> > experience slower real-world performances.
> >
> > Cheers,
> >
> > -- Matthieu Baechler
> >
> > On Tue, 2020-12-29 at 12:54 +0700, Tellier Benoit wrote:
> >> Hello there!
> >>
> >> We had been discussing on GitHub recently about an optimization in
> >> james
> >> core around the usage of MimeMessage.
> >>
> >> Javax.mail MimeMessage is currently used to represent a message of an
> >> email as part of the mail processing in James. It is part of the Mail
> >> interface (mailet-api).
> >>
> >> As Mail envelope is composed of several recipients, mail related
> >> operations are performed once for all these recipients (we enqueue
> >> the
> >> mail one time, we strip bcc one time etc...). Troubles arise when we
> >> need different behaviors as part of mail processing across recipients
> >> (think remote recipients, that needs there mail to be relayed, versus
> >> local recipients that needs to be locally delivered). The email get's
> >> duplicated (in MatcherSplitter) and the processing will then be
> >> distinct
> >> for both entities. The underlying MimeMessage may - or may not be
> >> modified.
> >>
> >> In order to prevent MimeMessage duplication in the event the
> >> underlying
> >> MimeMessage is not modified, a Copy On Write mechanism was introduced
> >> (I
> >> guess... Sorry, I was not there yet).
> >>
> >> Upon his CI effort, Jean Helou with the help of Matthieu Baechler
> >> made
> >> he unpleasant finding that this was not thread safe, that was leading
> >> to
> >> build instabilities. The mailet processing happens in Camel, which is
> >> multi-threaded. Concurrency issues arised between modifications, and
> >> message disposal, when a same MimeMessage instance was shared. [1]
> >>
> >> A first effort was to try to achieve thread-safety, which leaded to a
> >> brittle double reetrant read-write locks in order to govern data
> >> access.
> >> However, another performance enhancement bypassed these lock
> >> mechanism
> >> (MimeMessageWrapper allows accessing the data as an InputStream
> >> instead
> >> of requiring to copy it). The effort seemed overwhelming, not to
> >> metion
> >> possible risks of dead-locks. [2]
> >>
> >> We then came up with an always copy implementation [3]. Simpler,
> >> safer... The underlying logic is to avoid trying being smarter than
> >> mutability, and leverage immutability to achieve thread safety, which
> >> is
> >> a classic functional programming idiom.
> >>
> >> JMH benchmarks were conducted. We highlighter little performance
> >> difference for small messages, in the percent realm for both memory
> >> allocation and compute time. Differences are however higher for
> >> bigger
> >> messages (~10%) for both metrics.
> >>
> >> Please note that above 100KB the MimeMessage would be stored on disk,
> >> thus limiting memory impact (see MimeMessageInputStreamSource). Maybe
> >> we
> >> should make the threshold configurable, via a system property f

Re: Jenkins CI setup

[Jean Helou]

> Could there be some automatic calculation of memory resources that makes
> the build fail on some servers and not on others?
>

maybe but


> Our CI servers have 28GiB of memory. Could Docker allocate an amount
> that is particularly suitable for our test suite?
>

the last memory error failure (
https://builds.apache.org/job/james/job/ApacheJames/job/PR-264 ) ran on H39
according to
https://cwiki.apache.org/confluence/display/INFRA/Jenkins+node+labels this
node has 4TB disk, 96GB RAM,
builds 9->12 & 14 ran on H39
build 6 which also failed on memory ran on H22 for which the wiki doesn"t
have stats
builds 45 48,49 and 50 succeeded on H48
(I'm sticking to failures  on purpose)

I can't really wait for 3-4 hours in front of the jenkins page to see if
another job starts on the worker my build is running on and the runners
don't seem to store build history (not even for 24h
https://builds.apache.org/computer/H39/builds build #13 failed on OOM and
ran between 12 and 15 on 2021-01-13)

so its really hard to correlate.

for this morning build I happened to see a kafka job start on the runner
(H42) while build #15 was running
Kafka » kafka-trunk-jdk15
<https://builds.apache.org/job/Kafka/job/kafka-trunk-jdk15/>
unfortunately that build was hit by the copy on write concurrency bug so no
memory error this time  \o/

However if you look at what I said about build #13

> I get a more classical `java.lang.OutOfMemoryError: Java heap space` a bit
> before (12:14:09.563 vs 12:45:57.988).
>  The last non error line before the fatal Direct buffer memory error is
> [INFO] Running
> org.apache.james.webadmin.integration.rabbitmq.RabbitMQReindexingWithEventDeadLettersTest
>  The last non error line before the nonfatal heap memory error is
> [INFO] Running
> org.apache.james.jmap.memory.cucumber.MemoryDownloadCucumberTest


According to all the documentation I have ever seen the Java heap space
message means that the JVM could not allocate memory for an object within
the heap and is not related to missing outside memory
in my experience which seems to match
https://stackoverflow.com/questions/46801741/jvm-crashes-with-error-cannot-allocate-memory-errno-12
a native memory allocation error while trying to increase the heap will
crash the JVM so the Java heap OOM error is not related to other processes
running on the machine
I also don't see errors related to docker containers used by the tests
failing to allocate memory, (there are some errors about the docker
containers not finding an image once in a while but thats it) the memory
errors are always in the james test code itself
the OOM Direct buffer memory seems to often be triggered by an attempt to
start GuiceJamesServer which in turns starts multiple netty servers (for
the various ports) the default max off heap memory can be related to Xmx
(see https://blog.alexis-hassler.com/2020/05/15/direct-buffer-memory.html
for a recent french resource on the subject or
http://www.mastertheboss.com/other/java-stuff/troubleshooting-outofmemoryerror-direct-buffer-memory
)
also
https://dzone.com/articles/troubleshooting-problems-with-native-off-heap-memo
says that the error seems to hitting a limit of the JVM instead of malloc
being unable to allocate native memory I couldn't find the corresponding
JDK documentation
So all this seems to confirm Matthieu"s intuition of a resource leak in the
test suites instead of a native memory starvation issue. why that leak
doesn't affect your CI is still a mystery to me

I know that I had some pain to reproduce a green build on my local
> computer, while it works pretty smoothly on the CI.
>
> Raphaël.
>
> Le 14/01/2021 à 09:37, Jean Helou a écrit :
> >> My 2 cents trying to bring a little force in this nice project :)
> >>
> > Thanks Raphaël :)
> >
> > All our old CI is open source, so you can just check the source, Luke:
> >> https://github.com/linagora/james-jenkins/blob/master/workflow-job#L643
> >
> > thanks for the pointer, I had not looked for that one. After digging
> > through the repo, I couldn't find any memory specific settings either.
> >
> > and in particular
> >>
> https://github.com/linagora/james-project/blob/master/dockerfiles/compilation/java-11/compile.sh#L65
> >>
> > So I'm sorry there is no magic mvn parameter...
> > well I was wondering if maybe there was something passed in
> > MVN_ADDITIONAL_ARG_LINE  but as I said I was unable to find anything
> > special in the james-jenkins repo.
> >
> > This leaves me even more confused, don't you encounter these random
> looking
> > failures on the linagora ci platform ?
> > I mean I did manage to get a few green builds but overall out of 62
> builds
> > I had 5 successful ones, that's less than 10% !
> >
> > I haven't kept detailed stats (I d

Re: Jenkins CI setup

[Jean Helou]

>
> My 2 cents trying to bring a little force in this nice project :)
>

Thanks Raphaël :)

All our old CI is open source, so you can just check the source, Luke:
> https://github.com/linagora/james-jenkins/blob/master/workflow-job#L643


thanks for the pointer, I had not looked for that one. After digging
through the repo, I couldn't find any memory specific settings either.

and in particular
>
> https://github.com/linagora/james-project/blob/master/dockerfiles/compilation/java-11/compile.sh#L65
>

So I'm sorry there is no magic mvn parameter...
>

well I was wondering if maybe there was something passed in
MVN_ADDITIONAL_ARG_LINE  but as I said I was unable to find anything
special in the james-jenkins repo.

This leaves me even more confused, don't you encounter these random looking
failures on the linagora ci platform ?
I mean I did manage to get a few green builds but overall out of 62 builds
I had 5 successful ones, that's less than 10% !

I haven't kept detailed stats (I didn't think it would be this bad) but out
of gut feeling, the primary causes seem to be:
- copy on write thread safety (which can arguably be explained by slower
computers), hence my impatience to see JAMES-3477 fixed since this would
likely resolve a lot of unstable tests
- out of memory errors which I find much harder to explain by slower
machines

For the out of memory errors I ended up increasing the Xmx of surefire
(from 1g to 2g) in the following pom files :

 
server/protocols/jmap-draft-integration-testing/cassandra-jmap-draft-integration-testing/pom.xml
 
server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/pom.xml

 
server/protocols/webadmin-integration-test/memory-webadmin-integration-test/pom.xml

 server/protocols/webadmin-integration-test/pom.xml

 
server/protocols/jmap-rfc-8621-integration-tests/distributed-jmap-rfc-8621-integration-tests/pom.xml
 server/protocols/webadmin/webadmin-mailbox/pom.xml
 server/protocols/webadmin/webadmin-mailbox/pom.xml
 server/container/guice/cassandra-rabbitmq-guice/pom.xml
 
server/protocols/jmap-draft-integration-testing/rabbitmq-jmap-draft-integration-testing/pom.xml
 
server/protocols/jmap-draft-integration-testing/memory-jmap-draft-integration-testing/pom.xml

Obviously I very much look forward to the removal of the jmap draft module,
since I read in other exchanges that it was deprecated and would be removed

If I still can't get the build to pass, I'll look into matthieu's
suggestion :

The first solution is to recycle JVMs less to mitigate leaks effects
> (with surefire reusefork option).


Cheers,
jean


> And happy new year to you!
>
> Cheers,
>
> Raphaël.
>
> Le 13/01/2021 à 12:50, Jean Helou a écrit :
> > Happy new year fellow jamers !
> >
> > In this thrilling new episode you might learn if 2021 will be the year
> the
> > james project gets a public ci rolling again !
> >
> > CI wars
> > Episode 49e^55
> > The Memory errors strike back
> > The CI Resistance succeeded in configuring jenkins, fixed some tests,
> > exposed some bugs and tagged a lot of unstable tests as being unstable.
> > After such a striking defeat the empire of bugs reacted in the most
> vicious
> > way ever, it deployed "Direct buffer memory" errors throughout the galaxy
> > to find contributors to the CI effort and tear down their hope and
> > motivation. They found the apache jenkins and it will need help from all
> > the CI resistance members to fight them off !
> >
> > On a bit more serious note,
> > I am at a loss as to how to fix this issue. My last four builds have
> failed
> > because a `java.lang.OutOfMemoryError: Direct buffer memory` caused the
> > forked jvm to crash, crashing the surefire plugin and the build with it.
> > and that has been a build failure cause for a lot of the 63 builds on the
> > apache CI. until now I updated the pom files of the corresponding
> projects
> > to increase heap to 2G but the last failure occured in a project where
> the
> > heap was already increased.
> >
> > Looking at a specific log
> >
> https://builds.apache.org/blue/organizations/jenkins/james%2FApacheJames/detail/PR-264/12/pipeline
> > I get a more classical `java.lang.OutOfMemoryError: Java heap space` a
> bit
> > before (12:14:09.563 vs 12:45:57.988).
> >   The last non error line before the fatal Direct buffer memory error is
> > [INFO] Running
> >
> org.apache.james.webadmin.integration.rabbitmq.RabbitMQReindexingWithEventDeadLettersTest
> >   The last non error line before the nonfatal heap memory error is
> > [INFO] Running
> > org.apache.james.jmap.memory.cucumber.MemoryDownloadCucumberTest
> >
> > I will try to increase surefire's heap for the
> > memory-jmap

Re: Jenkins CI setup

[Jean Helou]

Happy new year fellow jamers !

In this thrilling new episode you might learn if 2021 will be the year the
james project gets a public ci rolling again !

CI wars
Episode 49e^55
The Memory errors strike back
The CI Resistance succeeded in configuring jenkins, fixed some tests,
exposed some bugs and tagged a lot of unstable tests as being unstable.
After such a striking defeat the empire of bugs reacted in the most vicious
way ever, it deployed "Direct buffer memory" errors throughout the galaxy
to find contributors to the CI effort and tear down their hope and
motivation. They found the apache jenkins and it will need help from all
the CI resistance members to fight them off !

On a bit more serious note,
I am at a loss as to how to fix this issue. My last four builds have failed
because a `java.lang.OutOfMemoryError: Direct buffer memory` caused the
forked jvm to crash, crashing the surefire plugin and the build with it.
and that has been a build failure cause for a lot of the 63 builds on the
apache CI. until now I updated the pom files of the corresponding projects
to increase heap to 2G but the last failure occured in a project where the
heap was already increased.

Looking at a specific log
https://builds.apache.org/blue/organizations/jenkins/james%2FApacheJames/detail/PR-264/12/pipeline
I get a more classical `java.lang.OutOfMemoryError: Java heap space` a bit
before (12:14:09.563 vs 12:45:57.988).
 The last non error line before the fatal Direct buffer memory error is
[INFO] Running
org.apache.james.webadmin.integration.rabbitmq.RabbitMQReindexingWithEventDeadLettersTest
 The last non error line before the nonfatal heap memory error is
[INFO] Running
org.apache.james.jmap.memory.cucumber.MemoryDownloadCucumberTest

I will try to increase surefire's heap for the
memory-jmap-draft-integration-testing project too in case the inital heap
space OOM triggered the other one.
stackoverflow is not very helpful either
https://stackoverflow.com/search?q=java.lang.OutOfMemoryError%3A+Direct+buffer+memory
or I have not been able to comprehend how the solutions there could help

I have gone through the files in /dockerfiles without finding anything that
looked related to memory configuration of maven itself, If people who run
the build locally with success or on their own CI could check the MVN_OPTS
and let me know if they override maven's Xmx itself I would appreciate it.

thanks for your help
jean



On Tue, Dec 29, 2020 at 9:10 AM Jean Helou  wrote:

> Hi Benoit,
>
> As someone operating another CI, I want to play even unstable test on
>> every runs. Is there some adaptation needed to do this?
>>
>
> Yes you will have to change your CI,
> > mvn -B -e -fae test
> now only runs stable tests, to run unstable tests you need an additional
> step
> > mvn -B -e -fae test -Punstable-tests
>
> I believe your CI is also based on jenkins (because of the stress test
> jenkinsfile at the root of the project) in which case you could configure
> your jenkins to pick up the jenkinsfile and use the same pipeline as we use
> on apache CI
>
> cheers,
> jean
>

Re: IMAP Fetch behavior on not found messages

[Jean Helou]

>
>
> When doing a fetch with some
> non existing messages Cyrus will do a best effort and return the
> existing messages whereas James will return a BAD response.


I would preserve Cyrus's behavior as a defacto standard, not honoring this
incurs the risk of breaking existing client software which relies on this
behavior.
if the preference is for a stricter behavior, then BAD is correct here. I
would definitely suggest to try the stricter behaviour with an outlook
client to make sure it doesn't break the UX too badly


> And in case
> of a fetch on an empty mailbox Cyrus will return a NO response where
> James will return a BAD one.
>

After reading the discussion I feel that NO is more appropriate.
-  NO feels more like  HTTP 404 NOT FOUND
-  BAD feels more like HTTP 400 BAD REQUEST

--
jean