[jira] [Closed] (JAMES-4007) Some IMAP litterals were not cleaned up

[Benoit Tellier (Jira)]

 [ 
https://issues.apache.org/jira/browse/JAMES-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benoit Tellier closed JAMES-4007.
-
Resolution: Fixed

> Some IMAP litterals were not cleaned up
> ---
>
> Key: JAMES-4007
> URL: https://issues.apache.org/jira/browse/JAMES-4007
> Project: James Server
>  Issue Type: Bug
>  Components: IMAPServer
>Reporter: Benoit Tellier
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> h3. What I noticed
> {code:java}
> root@tmail-imap-smtp-9dc75475f-4dfc7:/# ls -lah /tmp
> total 17M
> drwxrwxrwt 3 root root  220 Feb 21 11:47 .
> drwxr-xr-x 1 root root   29 Feb 21 09:49 ..
> -rw--- 1 root root 252K Feb 21 11:47 
> FileBackedOutputStream4796294521529525936.tmp
> -rw--- 1 root root 6.3M Feb 21 11:47 
> FileBackedOutputStream5042970611980201980.tmp
> -rw--- 1 root root 111K Feb 21 11:47 
> FileBackedOutputStream9469184234067839554.tmp
> drwxr-xr-x 2 root root   60 Feb 21 09:49 hsperfdata_root
> -rw--- 1 root root 8.0M Feb 21 11:47 imap-literal14135829094410407687.tmp
> -rw--- 1 root root 253K Feb 21 11:47 imap-literal14825699277716575828.tmp
> -rw--- 1 root root 113K Feb 21 11:47 imap-literal17479090714550035642.tmp
> -rw--- 1 root root 200K Feb 21 11:47 imap-literal3284956234462346798.tmp
> -rw--- 1 root root 1.8M Feb 21 11:17 imap-literal7906819567048433417.tmp
> {code}
> There a 30 min+ file that was not cleaned up.
> This is a very rare occurence.
> Looking at the code this could happend if the subscription is canceled while 
> processing the APPEND command...



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Closed] (JAMES-4008) JMAP - Email/set - Should be able to save a draft with invalid email address

[Benoit Tellier (Jira)]

 [ 
https://issues.apache.org/jira/browse/JAMES-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benoit Tellier closed JAMES-4008.
-
Resolution: Fixed

> JMAP - Email/set - Should be able to save a draft with invalid email address
> 
>
> Key: JAMES-4008
> URL: https://issues.apache.org/jira/browse/JAMES-4008
> Project: James Server
>  Issue Type: Improvement
>Reporter: Tung TRAN
>Priority: Trivial
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> As the Jmap.io [https://jmap.io/spec-mail.html#emailset]
> {code:java}
> The final message generated may be invalid per RFC 5322. For example, if it 
> is a half-finished draft, the To header field may have a value that does not 
> conform to the required syntax for this header. The message will be checked 
> for strict conformance when submitted for sending (see the EmailSubmission 
> object description).{code}
> should be able to save such a draft, with a recipient in the typing
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[BUILD-FAILURE]: Job 'james/ApacheJames/master [master] [1272]'

[Apache Jenkins Server]

BUILD-FAILURE: Job 'james/ApacheJames/master [master] [1272]':
Check console output at "https://ci-builds.apache.org/job/james/job/ApacheJames/job/master/1272/;>james/ApacheJames/master
 [master] [1272]"

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Resolved] (JAMES-3995) Optimize Email/get method

[Jira]

 [ 
https://issues.apache.org/jira/browse/JAMES-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

René Cordier resolved JAMES-3995.
-
Resolution: Fixed

https://github.com/apache/james-project/pull/2005 and 
https://github.com/apache/james-project/pull/2024 have been merged

> Optimize Email/get method
> -
>
> Key: JAMES-3995
> URL: https://issues.apache.org/jira/browse/JAMES-3995
> Project: James Server
>  Issue Type: Improvement
>Affects Versions: 3.8.1
>Reporter: René Cordier
>Priority: Major
> Fix For: 3.9.0
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> For Email/get, we seem to buffer body binary parts using Mime4J to be able to 
> recalculate the size of the part. 
> This causes James to consume a very high amount of memory, which can cause 
> performance issues if there is too many of those.
> We should try to find a way to avoid buffering the binary parts in order to 
> save memory, and boost performance for Email/get JMAP method.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Resolved] (JAMES-4009) Duplicated attachment filename: StripAttachment logs the binaries

[Jira]

 [ 
https://issues.apache.org/jira/browse/JAMES-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

René Cordier resolved JAMES-4009.
-
Fix Version/s: 3.9.0
   Resolution: Fixed

https://github.com/apache/james-project/pull/2043 resolves this and has been 
merged

> Duplicated attachment filename: StripAttachment logs the binaries
> -
>
> Key: JAMES-4009
> URL: https://issues.apache.org/jira/browse/JAMES-4009
> Project: James Server
>  Issue Type: Bug
>  Components: Mailet Contributions
>Reporter: Benoit Tellier
>Priority: Major
> Fix For: 3.9.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Given a mail with two attachemnts with the same file names, StripAttachment 
> would emmit a log with the binary of the attachment as a payload, which can 
> be absurdly large.
> EG
> {code:java}
> java.lang.IllegalArgumentException: Multiple entries with same key: 
> 23-02-2024-a-16h00.ics=AttributeValue{value=[67, 111, 110, 116, 101, 110, 
> 116, 45, 84, 121, 112, 101, 58, 32, 116, 101, 120, 116, 47, 99, 97, 108, 101, 
> 110, 100, 97, 114, 13, 10, 67, 111, 110, 116, 101, 110, 116, 45, 84, 114, 97, 
> 110, 115, 102, 101, 114, 45, 69, 110, 99, 111, 100, 105, 110, 103, 58, 32, 
> 98, 97, 115, 101, 54, 52, 13, 10, 67, 111, 110, 116, 101, 110, 116, 45, 68, 
> 105, 115, 112, 111, 115, 105, 116, 105, 111, 110, 58, 32, 97, 116, 116, 97, 
> 99, 104, 109, 101, 110, 116, 59, 32, 102, 105, 108, 101, 110, 97, 109, 101, 
> 61, 50, 51, 45, 48, 50, 45, 50, 48, 50, 52, 45, 97, 45, 49, 54, 104, 48, 48, 
> 46, 105, 99, 115, 13, 10, 13, 10, 81, 107, 86, 72, 83, 85, 52, 54, 86, 107, 
> 78, 66, 84, 69, 86, 79, 82, 69, 70, 83, 68, 81, 112, 87, 82, 86, 74, 84, 83, 
> 85, 57, 79, 79, 106, 73, 117, 77, 65, 48, 75, 85, 70, 74, 80, 82, 69, 108, 
> 69, 79, 109, 108, 106, 89, 87, 120, 108, 98, 109, 82, 104, 99, 105, 49, 121, 
> 100, 87, 74, 53, 68, 81, 112, 68, 81, 85, 120, 84, 13, 10, 81, 48, 70, 77, 
> 82, 84, 112, 72, 85, 107, 86, 72, 84, 49, 74, 74, 81, 85, 52, 78, 67, 107, 
> 49, 70, 86, 69, 104, 80, 82, 68, 112, 81, 86, 85, 74, 77, 83, 86, 78, 73, 68, 
> 81, 112, 67, 82, 85, 100, 74, 84, 106, 112, 87, 86, 69, 108, 78, 82, 86, 112, 
> 80, 84, 107, 85, 78, 67, 108, 82, 97, 83, 85, 81, 54, 82, 88, 86, 121, 13, 
> 10, 98, 51, 66, 108, 76, 49, 66, 104, 99, 109, 108, 122, 68, 81, 112, 67, 82, 
> 85, 100, 74, 84, 106, 112, 69, 81, 86, 108, 77, 83, 85, 100, 73, 86, 65, 48, 
> 75, 82, 70, 82, 84, 86, 69, 70, 83, 86, 68, 111, 121, 77, 68, 73, 48, 77, 68, 
> 77, 122, 77, 86, 81, 119, 77, 122, 65, 119, 77, 68, 65, 78, 67, 108, 82, 97, 
> 84, 48, 90, 71, 13, 10, 85, 48, 86, 85, 82, 108, 74, 80, 84, 84, 111, 114, 
> 77, 68, 69, 119, 77, 65, 48, 75, 86, 70, 112, 80, 82, 107, 90, 84, 82, 86, 
> 82, 85, 84, 122, 111, 114, 77, 68, 73, 119, 77, 65, 48, 75, 85, 108, 74, 86, 
> 84, 69, 85, 54, 82, 108, 74, 70, 85, 84, 49, 90, 82, 85, 70, 83, 84, 70, 107, 
> 55, 81, 108, 108, 69, 81, 86, 107, 57, 13, 10, 76, 84, 70, 84, 86, 84, 116, 
> 67, 87, 85, 49, 80, 84, 108, 82, 73, 80, 84, 77, 78, 67, 108, 82, 97, 84, 
> 107, 70, 78, 82, 84, 112, 68, 82, 86, 78, 85, 68, 81, 112, 70, 84, 107, 81, 
> 54, 82, 69, 70, 90, 84, 69, 108, 72, 83, 70, 81, 78, 67, 107, 74, 70, 82, 48, 
> 108, 79, 79, 108, 78, 85, 81, 85, 53, 69, 81, 86, 74, 69, 13, 10, 68, 81, 
> 112, 69, 86, 70, 78, 85, 81, 86, 74, 85, 79, 106, 73, 119, 77, 106, 77, 120, 
> 77, 68, 73, 53, 86, 68, 65, 121, 77, 68, 65, 119, 77, 65, 48, 75, 86, 70, 
> 112, 80, 82, 107, 90, 84, 82, 86, 82, 71, 85, 107, 57, 78, 79, 105, 115, 119, 
> 77, 106, 65, 119, 68, 81, 112, 85, 87, 107, 57, 71, 82, 108, 78, 70, 86, 70, 
> 82, 80, 13, 10, 79, 105, 115, 119, 77, 84, 65, 119, 68, 81, 112, 83, 85, 108, 
> 86, 77, 82, 84, 112, 71, 85, 107, 86, 82, 80, 86, 108, 70, 81, 86, 74, 77, 
> 87, 84, 116, 67, 87, 85, 82, 66, 87, 84, 48, 116, 77, 86, 78, 86, 79, 48, 74, 
> 90, 84, 85, 57, 79, 86, 69, 103, 57, 77, 84, 65, 78, 67, 108, 82, 97, 84, 
> 107, 70, 78, 82, 84, 112, 68, 13, 10, 82, 86, 81, 78, 67, 107, 86, 79, 82, 
> 68, 112, 84, 86, 69, 70, 79, 82, 69, 70, 83, 82, 65, 48, 75, 82, 85, 53, 69, 
> 79, 108, 90, 85, 83, 85, 49, 70, 87, 107, 57, 79, 82, 81, 48, 75, 81, 107, 
> 86, 72, 83, 85, 52, 54, 86, 107, 86, 87, 82, 85, 53, 85, 68, 81, 112, 69, 86, 
> 70, 78, 85, 81, 85, 49, 81, 79, 106, 73, 119, 13, 10, 77, 106, 81, 119, 77, 
> 106, 73, 121, 86, 68, 69, 121, 77, 84, 89, 121, 77, 86, 111, 78, 67, 108, 86, 
> 74, 82, 68, 112, 107, 90, 68, 70, 107, 78, 84, 69, 121, 77, 105, 48, 50, 79, 
> 68, 65, 121, 76, 84, 82, 109, 90, 106, 73, 116, 89, 84, 86, 107, 79, 83, 48, 
> 48, 79, 84, 90, 106, 89, 87, 69, 48, 78, 71, 90, 109, 77, 122, 73, 78, 13, 
> 10, 67, 107, 82, 85, 85, 49, 82, 66, 85, 108, 81, 55, 86, 70, 112, 74, 82, 
> 68, 49, 70, 100, 88, 74, 118, 99, 71, 85, 118, 85, 71, 70, 121, 97, 88, 77, 
> 54, 

[jira] [Resolved] (JAMES-4010) Option to disable body indexing in OpenSearch

[Jira]

 [ 
https://issues.apache.org/jira/browse/JAMES-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

René Cordier resolved JAMES-4010.
-
Resolution: Implemented

https://github.com/apache/james-project/pull/2018 implements this and has been 
merged

> Option to disable body indexing in OpenSearch
> -
>
> Key: JAMES-4010
> URL: https://issues.apache.org/jira/browse/JAMES-4010
> Project: James Server
>  Issue Type: Improvement
>  Components: opensearch
>Reporter: Benoit Tellier
>Priority: Major
>
>  -> This allows not reading bodies for this
>  -> This allows saving the costs of indexing large bodies
>  -> Still benefit from searching header
> So defaulting to false, that is a nice "safety trigger" or "cost saver" to 
> get around



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Build metrics

[Rene Cordier]

Hello Jean,

Mistake on my part indeed, was not the intention of making this private, 
so i'm pushing it as public again on the ML. Thanks :)


Thanks for taking a look.

Interesting the diagnoses we can seem to get with develocity !

Regards,

Rene.

On 2/23/24 15:37, Jean Helou wrote:

hello rene,

I'm not sure if you answered privately on purpose or not so I'm 
answering in the same manner but I do thinks this conversation could 
benefit from being public :D


I looked in the testcontainer issue you mention and damn what a shitty 
design. There were mentions of changing it when introducing the 
ImagePullPolicy but I reviewed the MR and it wasn't changed and it 
still isn't. The effort to work around it is quite significant (but 
doable) I might poc it after my vacation.


However, going through the details of the errors using develocity, 
there are a lot more times when the cassandra tests fail because of
Caused by: com.datastax.oss.driver.api.core.DriverTimeoutException: 
Query timed out after PT2S 	


than there are times where it failed with a docker init error

looking around I also found that :
cassandra, opensearch, s3 and ldap are the subsystems that show errors
rabbit or pulsar for instance seem unaffected ( no failures over 3 
weeks of builds) there may be less integration tests for these 2 but 
this questions the volume of integration test done for the others :p



for now the volume of tests is a bit low and I need to include all PR 
builds to get some trends. over time we will be able to target master 
only to get a better view of what's going on


jean

Le jeu. 22 févr. 2024 à 04:12, Rene Cordier  a 
écrit :


Hi Jean,

Thanks for the heads up on your work on this, very interesting.

Regarding the test failures, to add a bit more insight, I never
really
took the time to fully look into it, but yes the issues with a test
containers not starting because could not initialize some class error
log has been going on for a while, and is plaguing our builds.

As I think Quan noticed a while ago, this issue opened on the
testcontainers java backlog seems related:
https://github.com/testcontainers/testcontainers-java/issues/1872

It seems this error is misleading and hides the real one. It could
ber
elated to infra, or something maybe real tricky on our build.
Might need
to take the time to dig deeper at some point, but looks like a tricky
one to me :)

Hope this bit of info helps somehow!

Regards,

Rene.

On 2/21/24 18:01, Jean Helou wrote:
> Hi fellows,
>
> The Develocity  integration of
james-project build
> has now been running for long enough that we can start seeing
interesting
> things beyond the raw build scans.
> First we can give a short look at build trends
>

>
> for the `clean install` command on master (the build stage), its
hard to
> derive actual trends since
> - we only have 15 days of history
> - we run on runners with varying cpu power ( build scans show
T1C go from
> 16 to 24)
> but it can be used to establish a baseline so we can compare
later on
> In average this stage took 9min10s
> the week from feb 5 to feb 11th has had a lot more spread 5th-95th
> percentile was 4m54s to 11m26s
> last week was a bit more homogenous with the same percentiles
being 7m54 -
> 10m18
> this week so far looks more similar to last week than the
previous, 7m12 -
> 9m4
>
> Adding the local build cache added about 14s of overhead and
provided no
> benefits on CI (completely expected, maybe I should disable
local caching
> on CI since it spawns in a fresh env everytime) but this will be
> interesting to compare with the remote build cache.
>
> The tests
>

>
> monitoring is also interesting. This screen monitors 2 things:
> - which tests fail the most often
> - which tests are flaky
> To measure this I enabled retries for surefire in this commit
>  so
that a failing
> test is retried once.
> Failed is both tries failed
> Flakyness is derived by looking at retries (1 fail followed by 1
success).
>
> The results probably won't surprise you too much :
>
> The test suite with most failures (11% of the runs) is
> org.apache.james.blob.cassandra.CassandraBlobStoreTest
>

[BUILD-FAILURE]: Job 'james/ApacheJames/master [master] [1271]'

[Apache Jenkins Server]

BUILD-FAILURE: Job 'james/ApacheJames/master [master] [1271]':
Check console output at "https://ci-builds.apache.org/job/james/job/ApacheJames/job/master/1271/;>james/ApacheJames/master
 [master] [1271]"

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Created] (JAMES-4010) Option to disable body indexing in OpenSearch

[Benoit Tellier (Jira)]

Benoit Tellier created JAMES-4010:
-

 Summary: Option to disable body indexing in OpenSearch
 Key: JAMES-4010
 URL: https://issues.apache.org/jira/browse/JAMES-4010
 Project: James Server
  Issue Type: Improvement
  Components: opensearch
Reporter: Benoit Tellier


 -> This allows not reading bodies for this
 -> This allows saving the costs of indexing large bodies
 -> Still benefit from searching header

So defaulting to false, that is a nice "safety trigger" or "cost saver" to get 
around



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Closed] (JAMES-3986) AttachmentFilenameIs do not sanitize QEncoding

[Benoit Tellier (Jira)]

 [ 
https://issues.apache.org/jira/browse/JAMES-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benoit Tellier closed JAMES-3986.
-
Resolution: Fixed

> AttachmentFilenameIs do not sanitize QEncoding
> --
>
> Key: JAMES-3986
> URL: https://issues.apache.org/jira/browse/JAMES-3986
> Project: James Server
>  Issue Type: Bug
>  Components: Mailet Contributions
>Affects Versions: 3.8.0
>Reporter: Benoit Tellier
>Priority: Major
> Fix For: master
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> AttachmentFileNameIs do not support Q encoding
> EG: =?US-ASCII?Q?IHE=5FXDM.zip?=
> It should sanitize those.
> Also, by providing a decent code coverage we can unmark this matcher as 
> experimental.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[jira] [Closed] (JAMES-3997) Netty backpressure for IMAP Fetch

[Benoit Tellier (Jira)]

 [ 
https://issues.apache.org/jira/browse/JAMES-3997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benoit Tellier closed JAMES-3997.
-
Resolution: Fixed

Could be a candidate for 3.8 backport though...

If somebody in the community feels motivated.

> Netty backpressure for IMAP Fetch
> -
>
> Key: JAMES-3997
> URL: https://issues.apache.org/jira/browse/JAMES-3997
> Project: James Server
>  Issue Type: Improvement
>Affects Versions: 3.8.0, 3.8.1
>Reporter: René Cordier
>Priority: Major
> Fix For: 3.9.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Some OOM IMAP issues on a production environment using James have been 
> detected, regarding the method IMAP FETCH.
> We seem to do to write out:
> {code:java}
> channel.writeAndFlush()Unpooled.wrappedBuffer(buffer)); 
> {code}
>  without any checks. Making the method using intermediate buffers in a slow 
> network, potentially exploding the memory.
> Need to investigate this issue and enhance this processus, maybe by using 
> Netty backpressure instead?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org