Benoit Tellier created JAMES-4235:
-------------------------------------
Summary: mailbox/lucene hardening and adoption for PG app
Key: JAMES-4235
URL: https://issues.apache.org/jira/browse/JAMES-4235
Project: James Server
Issue Type: Bug
Components: lucene
Reporter: Benoit Tellier
h3. Context
As part of a hackaton a colleague gave a try to our pg and memory app in a low
memory setup.
Here is a sum up of the issues:
- A No viable alternatives to OpenSearch for search snippets on top of the PG
app
- B SearchSnippet/get fails with NPE for non matching text attachment
- C Email/query subject / text / body never match strings of more than 4 words
or words after punctuation
- D Email/query hasKeyword / notKeyword without inMailbox matches messages by
UID across mailboxes
- E Expunging two or more consecutive UIDs removes every message of the
mailbox from the search index
2026 being 2026 there's a lengthy AI description for each, I include links to
the original reports:
- A https://github.com/linagora/james-project/issues/5616
- B https://github.com/linagora/james-project/issues/5614
- C https://github.com/linagora/james-project/issues/5613
- D https://github.com/linagora/james-project/issues/5612
- E https://github.com/linagora/james-project/issues/5607
h3. Proposal
Fix B, C, D, E then for A adopt Lucene as an option for pg-app - convenient for
self hosting. I verified it yields no classpath conflits with OpenSearch driver
which now is Rest - only.
If this is consensual we can get some linagorians on the topic.
h3. Details
h4. B SearchSnippet/get fails with NPE for non matching text attachment
{code:Markdown}
### Reproduction
```json
[["Email/query", {"accountId": "…", "filter": {"subject": "Quarterly"}}, "q"],
["SearchSnippet/get", {"accountId": "…", "filter": {"subject": "Quarterly"},
"#emailIds": {"resultOf": "q", "name": "Email/query", "path": "/ids"}}, "s"]]
```
1. Observed: `serverFail` (NPE). Expected: snippet with highlighted subject,
`preview: null`.
###
Cause`mailbox/lucene/src/main/java/org/apache/james/mailbox/lucene/search/LuceneSearchHighlighter.java`:
1. `getHighlightAttachmentTextBody` (:188-194): `getBestFragment` returns
`null` when nothing matches, and `Stream.findFirst()` throws NPE on a null
element. JAMES-4077 fixed subject/body only.
2. `buildQuery` (:162-165): raw value given to the classic `QueryParser`.
3. `highlightSearch` (:118-124): asks for `limit = messageIds.size()`
documents, but a message has one document per mailbox, so duplicates push
others out.
### Fix
1. `.filter(Objects::nonNull)` before `findFirst()`.
2. `QueryParser.escape(value)` (or build the query from analyzer tokens); at
worst catch `ParseException` and return a snippet without highlight.
3. Search with the usual `maxQueryResults` and keep one snippet per message id
(`distinct`).
{code}
h4. C Email/query subject / text / body never match strings of more than 4
words or words after punctuation
{code:Markdown}
### Reproduction
emails created in INBOX with subjects `Quarterly budget review meeting notes`,
`Invitation <Team sync> tomorrow`:
```json
["Email/query", {"accountId": "…", "filter": {"inMailbox": "<INBOX>",
"subject": "Quarterly budget review meeting notes"}}, "0"]
```
- Observed: no result. Expected: the email.
- `subject: "Quarterly budget review meeting"` (4 words) → found.
- `subject: "Team"` / `"Team sync"` → nothing; `text: "brown fox jumps over
the"` on a body `The quick brown fox jumps over the lazy dog` → nothing (4
words → found).
### Cause
At james-project `9aac85900f` (same on apache/james-project `master`), in
`mailbox/lucene/src/main/java/org/apache/james/mailbox/lucene/search/`:
- `LenientImapSearchAnalyzer.java:35-50` (used at
`LuceneMessageSearchIndex.java:245`): whitespace tokenizer, upper-case,
shingles of up to `DEFAULT_MAX_TOKEN_LENGTH = 4` tokens; no indexed term
exceeds 4 words, and `<TEAM` keeps its punctuation.
- `LuceneMessageSearchIndex.java:487-493` (`createTermQuery`, used by
`createHeaderQuery` :498-512 and `createTextQuery` :701-715): the value is not
analyzed, just `new PrefixQuery(new Term(field, value.toUpperCase()))`.
### Fix
- Query side (no reindex): beyond 4 tokens, AND per-token
`PrefixQuery`/`TermQuery` (or a phrase query over unigrams), e.g. via the index
analyzer without the shingle filter.
- Index side (reindex): a tokenizer that strips leading/trailing punctuation
(e.g. `StandardTokenizer`).
- Test: "search a subject of more than 4 words".
{code}
h4. D Email/query hasKeyword / notKeyword without inMailbox matches messages by
UID across mailboxes
{code:Markdown}
### Reproduction
fresh user, created with `Email/set`: INBOX `a1` (UID 1, keyword `label1`),
Sent `s1` (UID 1, no keyword).
```json
["Email/query", {"accountId": "…", "filter": {"hasKeyword": "label1"}}, "0"]
```
- Observed: `[a1, s1]`. Expected: `[a1]`.
- `notKeyword: label1` → `[a1, s1]`, expected `[s1]`. With `inMailbox` results
are correct.
- Not the Message-ID: a Sent message sharing `a2`'s Message-ID but not its UID
is not returned; one sharing its UID is.
### Cause
-
`mailbox/lucene/src/main/java/org/apache/james/mailbox/lucene/search/LuceneIndexableDocument.java:223-253`:
flags live in a separate per-message "flags" document.
-
`mailbox/lucene/src/main/java/org/apache/james/mailbox/lucene/search/LuceneMessageSearchIndex.java:576-622`
(`createFlagQuery`): searches flags documents in all query mailboxes, collects
only their UIDs into a `Set<MessageUid>` (:593-601), and returns a bare UID
query (:612-618) with no mailbox; `buildQuery` (:405-417) ANDs it with "any of
the mailboxes", so any message with a matching UID in any mailbox matches.
### Fix
- In `createFlagQuery`, collect `mailboxId -> uids` and return one `SHOULD`
clause per mailbox (`MAILBOX_ID_FIELD` term `MUST` + UID query `MUST`); mind
that message documents store `MAILBOX_ID_FIELD` upper-cased
(`LuceneIndexableDocument.java:133`).
- Structural alternative: store flags on the message document, removing the
two-step query.
- Test: "flag search across mailboxes having the same uids".
{code}
h4. E Expunging two or more consecutive UIDs removes every message of the
mailbox from the search index
{code:Markdown}
### Reproduction
create drafts d1…d5 (one `Email/set` each, so UIDs 1…5), then:
```json
["Email/set", {"accountId": "…", "destroy": ["<d2>", "<d3>"]}, "0"]
["Email/query", {"accountId": "…", "filter": {"inMailbox": "<Drafts>"}}, "1"]
```
- Expected: d1, d4, d5.
- Observed: empty. (`destroy [d2, d4]` correctly returns d1, d3, d5; postgresql
image returns d1, d4, d5.)
### Cause
-
`mailbox/api/src/main/java/org/apache/james/mailbox/model/MessageRange.java:153-188`:
`toRanges` turns consecutive UIDs into one `RANGE` (`{2, 3}` → `2:3`).
-
`mailbox/lucene/src/main/java/org/apache/james/mailbox/lucene/search/LuceneMessageSearchIndex.java:831-841`:
`createQuery(MessageRange)` has no `RANGE` case; it falls into `default`, the
`ALL` query `[MIN, MAX]`, so `delete(mailboxId, range)` (855-861) deletes every
document of the mailbox.
### Fix
- In `createQuery`, handle `ONE, RANGE` with `[uidFrom, uidTo]` and make the
switch exhaustive (no `default`); add a test "expunging two consecutive uids
keeps the other messages searchable".
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
