dovecot and solr clustering

2024-03-13 Thread Maciej Milaszewski
Hi I have 1 director and ~8 dovecot nodes. I thinking about Solr cluster beacuse one server solr probably is not enough. I thinking about SolrCloud Mode but I don't know if it will work with Dovecot And if not SolrCloud Mode, then what? I don't have much experience in Solr clustering and I want

Re: Solr/Tika patches for dovecot 2.3.21

2024-01-03 Thread John Fawcett
://github.com/dovecot/core/pull/215 All three patches are in a single pull request, but each patch has a separate commit. Also just noticed that pull request 213 contains a fix about solr "rows" query parameter with a different approach. If you intend to merge #213, let me know and I

Re: Solr/Tika patches for dovecot 2.3.21

2024-01-03 Thread Aki Tuomi via dovecot
21-tika-http-auth.patch* > > Allows specification of username and password in the fts_tika setting > for basic auth against tika server. For example > > fts_tika = https://user:password@tika_server:443/tika > > *dovecot-2.3.21-solr-max-size.patch > * > > Th

Solr/Tika patches for dovecot 2.3.21

2023-12-09 Thread John Fawcett
tika server. For example fts_tika = https://user:password@tika_server:443/tika dovecot-2.3.21-solr-max-size.patch This is a simplified version of my previous patch. Sets a size limit (configuration fts_max_size) on message bodies that are to be indexed. Message bodies for messages larger than

Re: dovecot-fts-solr Solr9 support

2023-04-24 Thread deano-dovecot
Shawn - You had mentioned in another email (somewhere) that were hopefully going to do a write-up of setting up Solr 9.x with Dovecot. Any chance you've had time for that ? Thanks - On 2022-09-30 1:52 pm, Shawn Heisey wrote: > On 9/27/22 19:32, Nathanael Anderson wrote: >

Re: How to use dovecot with more recent versions of solr (8x, 9x)?

2023-03-27 Thread Thomas Zajic
* j...@w3.org, 27.03.23 15:28 [...] Could someone confirm me if those 7.7.0 configuration files are compatible with 8.11.2? A diff between those files and the ones from the 8.11.2 dist reveal some changes, starting from the LuceneVersion. [...] Following

How to use dovecot with more recent versions of solr (8x, 9x)?

2023-03-27 Thread jkdv
Hello, dovecot version 2.3.13 (89f716dc2) The dovecot solr installation guide [1] gives links and configuration info for sorl 7.7.0. However, sorl was deprecated some time ago. Those links to 7.7.0 don't work anymore. Sorl is currently available in versions 8.11.2 and 9.1.1 [3]. Specifically

Re: iPhone Mail clients generate high Solr I/O

2023-03-21 Thread Shawn Heisey
On 3/20/23 09:52, Alessio Cecchi wrote: I'm running a mail server with Dovecot 2.3.20 and Apache Solr as FTS backend. It seems that iPhone/iPad Mail clients are generating some IMAP searches, especially on header Message-ID, that are increasing our Solr I/O load, especially during the night

iPhone Mail clients generate high Solr I/O

2023-03-20 Thread Alessio Cecchi
Hi, I'm running a mail server with Dovecot 2.3.20 and Apache Solr as FTS backend. It seems that iPhone/iPad Mail clients are generating some IMAP searches, especially on header Message-ID, that are increasing our Solr I/O load, especially during the night. Are these kind of queries normal

Re: dovecot-fts-solr Solr9 support

2022-09-30 Thread Shawn Heisey
On 9/27/22 19:32, Nathanael Anderson wrote: I was trying a new install of dovecot w/ solr9.   I've manually fixed the file linking to the proper directories, however one plugin is no longer shipped.   Since the solr files aren't updated yet to 9, can anyone tell me if I need the discontinued

Re: dovecot-fts-solr Solr9 support

2022-09-29 Thread Aki Tuomi
> On 28/09/2022 04:32 EEST Nathanael Anderson wrote: > > > I was trying a new install of dovecot w/ solr9.   I've manually fixed > the file linking to the proper directories, however one plugin is no > longer shipped.   Since the solr files aren't updated yet to 9, can

Re: dovecot-fts-solr Solr9 support

2022-09-29 Thread Nathanael Anderson
I was trying a new install of dovecot w/ solr9.   I've manually fixed the file linking to the proper directories, however one plugin is no longer shipped.   Since the solr files aren't updated yet to 9, can anyone tell me if I need the discontinued velocity plugin that was default

Re: solr-jetty package in Debian 11 bullseye

2022-05-25 Thread Shawn Heisey
On 5/25/22 13:30, John Gateley wrote: I am in the process of upgrading the OS on my mailserver, currently Debian 9, to Debian 11. The solr-jetty package does not appear to exist in Debian 11. The Solr package that was included in Debian and derivatives was REALLY ancient -- Solr 3.6

solr-jetty package in Debian 11 bullseye

2022-05-25 Thread John Gateley
Hello, I am in the process of upgrading the OS on my mailserver, currently Debian 9, to Debian 11. The solr-jetty package does not appear to exist in Debian 11. Does anyone know where it is? Thanks John

Re: dovecot-fts-solr Solr9 support

2022-05-23 Thread Shawn Heisey
. CaffeineCache was added in Solr 8.3.0.  So it is not available in all 8.x releases. https://issues.apache.org/jira/browse/SOLR-8241 I know more than a little bit about Solr's caches.  I am the author of LFUCache, added way back in Solr 3.6.0.  That is an extremely naive "intro to progra

Re: FTS Solr -- no commit on message delete

2022-05-18 Thread Shawn Heisey
On 5/18/2022 8:24 AM, Shawn Heisey wrote: Today I discovered a related problem that I think IS in Dovecot.  I did a shift-delete on some messages, and then exited Thunderbird.  Dovecot did delete the messages from Solr, but it did not send a commit to make those changes affect subsequent

FTS Solr -- no commit on message delete

2022-05-18 Thread Shawn Heisey
I've been chasing down oddities in behavior related to FTS Solr for a while now. First the part that isn't a problem with dovecot:  In cases where message deletes bypass the Trash folder (such as shift-delete and normal workflow with the Drafts and Sent folders), Thunderbird does not inform

Re: dovecot-fts-solr Solr9 support

2022-05-17 Thread Shawn Heisey
before.  That was what I was missing to make things really clean.  Before I was deleting all the dovecot.*index* files, and now I don't need to do that.  This is my new script, with the old way commented out: --- #!/bin/sh DOVECOT_SOLR_URL_BASE="http://localhost:8983/solr/dovecot; #MAIL

Re: dovecot-fts-solr Solr9 support

2022-05-17 Thread PGNet Dev
deprecation of legacy cache(s), did I notice these ... /var/log/solr/solr.log.6:2022-05-17 10:08:09.402 ERROR (qtp1165791284-17) [ x:dovecot] o.a.s.s.CacheConfig Error instantiating cache => org.apache.solr.common.SolrException: Error loading class 'solr.search.LRUCache' /var/log/solr/solr.

Re: dovecot-fts-solr Solr9 support

2022-05-17 Thread Shawn Heisey
On 5/17/22 09:02, PGNet Dev wrote: Modifying the config to use CaffeineCache cp solr-config-7.7.0.xml solr-config-9.0.0.xml perl -pi -e '\ s|solr.FastLRUCache|solr.CaffeineCache|g; \ s|solr.LRUCache|solr.CaffeineCache|g; \ s|solr.search.LRUCache|solr.search.CaffeineCache

dovecot-fts-solr Solr9 support

2022-05-17 Thread PGNet Dev
I run dovecot-2.3.18 , & am deploying Solr search with 'dovecot-fts-solr' I run Solr standalone. When creating a dedicated 'dovecot' core, cp'ing from Dovecot's included solr schemas ls -al /usr/share/doc/dovecot/solr-* -rw-r--r-- 1 root root 12K Dec

Re: Different handling of upper and lower case while indexing/searching with Solr

2022-02-10 Thread Patrik Peng
On 10.02.22 12:52, Aki Tuomi wrote: It is applied, but, your sql database overrides it by having non-lowercased usernames. Ah, I see. I was able to adjust our config accordingly. Thanks for your help :) OpenPGP_signature Description: OpenPGP digital signature

Re: Different handling of upper and lower case while indexing/searching with Solr

2022-02-10 Thread Aki Tuomi
> On 10/02/2022 13:46 Patrik Peng wrote: > > > On 10.02.22 11:25, Aki Tuomi wrote: > > You can configure dovecot with > > > > auth_username_format=%Lu > > > > which downcases the username provided by the customer, as well. > > According to the docs [1], is '%Lu' already the default and this

Re: Different handling of upper and lower case while indexing/searching with Solr

2022-02-10 Thread Patrik Peng
On 10.02.22 11:25, Aki Tuomi wrote: You can configure dovecot with auth_username_format=%Lu which downcases the username provided by the customer, as well. According to the docs [1], is '%Lu' already the default and this value is not changed in our config. I guess 'auth_username_format=%Lu'

Re: Different handling of upper and lower case while indexing/searching with Solr

2022-02-10 Thread Aki Tuomi
> On 10/02/2022 11:58 Patrik Peng wrote: > > > On 10.02.22 10:43, Aki Tuomi wrote: > > Probably easiest fix is to fix the users in database to all lowercase, as > > you are likely returning `user` attribute in your SQL queries. > We thought about this as well, but there are 500+ affected

Re: Different handling of upper and lower case while indexing/searching with Solr

2022-02-10 Thread Patrik Peng
On 10.02.22 10:43, Aki Tuomi wrote: Probably easiest fix is to fix the users in database to all lowercase, as you are likely returning `user` attribute in your SQL queries. We thought about this as well, but there are 500+ affected accounts and they are used by our customers which would mean

Re: Different handling of upper and lower case while indexing/searching with Solr

2022-02-10 Thread Aki Tuomi
> On 10/02/2022 11:36 Patrik Peng wrote: > > > On 09.02.22 17:47, Christian Kivalo wrote: > > > How are your users added to your auth backend? > > We use a SQL DB as auth backend. Users are added by an external application. > New accounts are all added as lowercase, but it could be possible

Re: Different handling of upper and lower case while indexing/searching with Solr

2022-02-10 Thread Patrik Peng
On 09.02.22 17:47, Christian Kivalo wrote: How are your users added to your auth backend? We use a SQL DB as auth backend. Users are added by an external application. New accounts are all added as lowercase, but it could be possible that there was a time in the past where accounts were added

Re: Different handling of upper and lower case while indexing/searching with Solr

2022-02-09 Thread Christian Kivalo
On February 9, 2022 12:31:23 PM GMT+01:00, Patrik Peng wrote: >Woops, this time with better formatting. > >On 09.02.22 12:21, Patrik Peng wrote: >> >> Hello there >> >> We stumbled upon an user account with Solr FTS, which returned no >> search results

Re: Different handling of upper and lower case while indexing/searching with Solr

2022-02-09 Thread Patrik Peng
Woops, this time with better formatting. On 09.02.22 12:21, Patrik Peng wrote: Hello there We stumbled upon an user account with Solr FTS, which returned no search results for any given search query. Further investigation revealed an issue between indexing mails and querying the index

Different handling of upper and lower case while indexing/searching with Solr

2022-02-09 Thread Patrik Peng
Hello there We stumbled upon an user account with Solr FTS, which returned no search results for any given search query. Further investigation revealed an issue between indexing mails and querying the index. The user name contains upper and lower case characters (eg. some.u...@domain.net

Patch: enhancements for solr/tika integration

2022-01-08 Thread John Fawcett
Hi here's a patch with some enhancements that I am applying locally for solr/tika integration. Hopefully this can be considered for inclusion. I've tested up to 2.3.16 and this patch applies against latest version 2.3.17.1. The contents are: 1. Allow username and password in tika

Patch: safeguard against too large value for uid being sent to solr for single mailbox searches

2022-01-08 Thread John Fawcett
Hi I'm reposting a patch for solr integration which I have been applying locally. It applies against 2.3.17.1. Dovecot already has a mechanism when doing solr fts searches on multiple mailboxes that prevents a too large value for maximum rows being sent to solr. #define

Re: Can dovecot be leveraged to exploit Solr/Log4shell?

2021-12-15 Thread John Fawcett
On 15/12/2021 08:52, Aki Tuomi wrote: The suggested configuration is good, and although we did some checking to ensure that dovecot escapes the search queries and usernames sent to solr, so it is not trivial to send the JNDI expansion strings to be logged by solr, it is still good idea to set

Re: Can dovecot be leveraged to exploit Solr/Log4shell?

2021-12-15 Thread Jochen Bern
On 15.12.21 08:45, Alessio Cecchi wrote: SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true" and should be enough to prevent this vulnerability. Possibly not anymore, see CVE-2021-45046 ("re-opened" CVE-2021-44228 for v2 prior to 2.16.0) and CVE-2021-4104 (variant for v1, in the meantime

Re: Can dovecot be leveraged to exploit Solr/Log4shell?

2021-12-14 Thread Aki Tuomi
The suggested configuration is good, and although we did some checking to ensure that dovecot escapes the search queries and usernames sent to solr, so it is not trivial to send the JNDI expansion strings to be logged by solr, it is still good idea to set this. Aki > On 15/12/2021 09

Re: Can dovecot be leveraged to exploit Solr/Log4shell?

2021-12-14 Thread Alessio Cecchi
Hi, for Solr you can edit your solr.in.sh file to include: SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true" and should be enough to prevent this vulnerability. Ciao Il 13/12/21 23:43, Joseph Tam ha scritto: I'm surprised I haven't seen this mentioned yet. An internet red

Re: Can dovecot be leveraged to exploit Solr/Log4shell?

2021-12-14 Thread John Fawcett
On 14/12/2021 03:23, Scott wrote: Is this assuming you log at some verbose level ? What if you log at WARN or higher ? For production it seems kind of silly to log search queries anyways. Scott It's a pretty much standard install where most things are at INFO level. Probably could turn it

RE: Can dovecot be leveraged to exploit Solr/Log4shell?

2021-12-13 Thread Aki Tuomi
Dovecot itself has no log4j vulnerability as Dovecot does not use Java or Log4j directly. Solr, however, does use log4j. Please see https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228 for further information on upgrading or mitigating the issue. Aki

RE: Can dovecot be leveraged to exploit Solr/Log4shell?

2021-12-13 Thread Scott
Subject: Re: Can dovecot be leveraged to exploit Solr/Log4shell? On 13/12/2021 23:43, Joseph Tam wrote: > > I'm surprised I haven't seen this mentioned yet. > > An internet red alert went out Friday on a new zero-day exploit. It is > an input validation problem where Java's L

Re: Can dovecot be leveraged to exploit Solr/Log4shell?

2021-12-13 Thread John Fawcett
a remote LDAP server.  It has been designated the Log4shell exploit (CVE-2021-44228). Although I don't use it, I immediately thought of Solr, which provides some dovecot installations with search indexing.  Can dovecot be made to pass on arbitrary loggable strings to affected versions of Solr

Can dovecot be leveraged to exploit Solr/Log4shell?

2021-12-13 Thread Joseph Tam
designated the Log4shell exploit (CVE-2021-44228). Although I don't use it, I immediately thought of Solr, which provides some dovecot installations with search indexing. Can dovecot be made to pass on arbitrary loggable strings to affected versions of Solr (7.4.0-7.7.3, 8.0.0-8.11.0)? Those running

Dovecot, Solr (FTS) and iOS body search

2021-12-13 Thread Ben Howard
I had a question about full text search with Dovecot, Solr with iOS as a client (the built in default mail client). Does anyone happen to know if it's possible to get the iOS mail client to search bodies of email via IMAP with Dovecot and Solr on the server? I've looked at the IMAP queries

Re: Solr FTS - message deletes not working as expected

2021-11-19 Thread William Edwards
> Op 18 nov. 2021 om 17:15 heeft Shawn Heisey het > volgende geschreven: > > On 11/3/21 11:45 PM, Shawn Heisey wrote: >> Manual expunges of existing messages also are not sending a delete request >> to Solr. I waited several minutes for that too. > > Up

Re: Solr FTS - message deletes not working as expected

2021-11-18 Thread Shawn Heisey
On 11/3/21 11:45 PM, Shawn Heisey wrote: Manual expunges of existing messages also are not sending a delete request to Solr.  I waited several minutes for that too. Update on this, since you're all on the edge of your seat waiting. :) I did a Send test with TypeApp, a mail client for Android

Re: Solr FTS - message deletes not working as expected

2021-11-04 Thread Shawn Heisey
and add sent to Solr), and then emptying the trash does delete from Solr.  This is exactly how Thunderbird works with "normal" deletes. I use shift-delete in Thunderbird a lot.  Because when I delete a message, I really want it gone, I do not want another step to remember later -

Re: Solr FTS - message deletes not working as expected

2021-11-03 Thread Shawn Heisey
find both the copy in Drafts and the copy in Sent when I do a manual Solr query for that unique text. I waited several minutes ... I didn't hit Send and then immediately do the query. Manual expunges of existing messages also are not sending a delete request to Solr. I waited several minutes

Re: Solr FTS - message deletes not working as expected

2021-11-03 Thread Shawn Heisey
On 11/3/2021 10:12 PM, Shawn Heisey wrote: Maybe I can do a custom compile of the source code and replace the /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so file with what the compile produces.  I'm going to try that, and see if it explodes. :) Bit of a problem trying to do this. I pulled

Re: Solr FTS - message deletes not working as expected

2021-11-03 Thread Shawn Heisey
On 11/3/21 1:09 PM, Michael Slusarz wrote: For Solr, there's a code path in the FTS expunge code that will silently toss expunge requests: if (ctx->last_indexed_uid == 0 || uid > ctx->last_indexed_uid + 100) { /* don't waste time asking Solr t

Re: Solr FTS - message deletes not working as expected

2021-11-03 Thread Michael Slusarz
eems to be a very good protocol!  I can't say it > definitively without more evidence, but the problem *seems* to be in > FTS, not imap or core dovecot. For Solr, there's a code path in the FTS expunge code that will silently toss expunge requests: if (ctx->last_indexed_uid

Re: Solr FTS - message deletes not working as expected

2021-11-03 Thread Shawn Heisey
On 11/3/21 12:38 PM, Michael Slusarz wrote: Have you tried another client? I only have two clients configured:  Thunderbird 78.13.0 on Linux and Thunderbird 91.2.1 on Windows.  Behavior is the same on both. I will see if I can get another client configured.  Windows Mail is included on

Re: Solr FTS - message deletes not working as expected

2021-11-03 Thread Michael Slusarz
sey wrote: > > Also, when I send a message with Thunderbird, which deletes the > > message in Drafts and adds one to Sent, I am not seeing a delete > > request in the Solr log.  I do see the add. So this isn't limited to > > just the shift-delete workflow. > > > I have

Re: Solr FTS - message deletes not working as expected

2021-11-02 Thread Shawn Heisey
On 10/28/21 8:00 AM, Shawn Heisey wrote: Also, when I send a message with Thunderbird, which deletes the message in Drafts and adds one to Sent, I am not seeing a delete request in the Solr log.  I do see the add. So this isn't limited to just the shift-delete workflow. I have confirmed

Re: Solr FTS - message deletes not working as expected

2021-10-28 Thread Shawn Heisey
not seeing a delete request in the Solr log.  I do see the add.  So this isn't limited to just the shift-delete workflow. Thanks, Shawn

Solr FTS - message deletes not working as expected

2021-10-26 Thread Shawn Heisey
I have Solr FTS enabled in dovecot.  It is behaving differently than I would expect. When I delete messages in Thunderbird normally, using IMAP to talk to dovecot, the deletion hits Solr right away.  If I then ask Thunderbird to empty the trash, that delete also hits Solr immediately

Re: Dovecot - FTS Solr: disk usage & position information?

2021-09-06 Thread Shawn Heisey
On 9/6/2021 12:58 AM, Vincent Brillault wrote: Hi Alessio, this optimization also produce a less RAM requirements on Solr server? Unfortunately we didn't measure this before/after the change. Since we are removing features (position information), I wouldn't expect the memory requirement

Re: Dovecot - FTS Solr: disk usage & position information?

2021-09-06 Thread Vincent Brillault
Hi Alessio, this optimization also produce a less RAM requirements on Solr server? Unfortunately we didn't measure this before/after the change. Since we are removing features (position information), I wouldn't expect the memory requirement to increase, but I'm no expert. To be honest

Re: Solr FTS - when does indexing happen?

2021-09-05 Thread Steve Dondley
Since most people will want fts_autoindex, the wiki page should include it in its example configuration that goes into 90-plugin.conf. Possibly better ... maybe it should default to "yes". It's probably a safe bet the developers, who are experts on these systems, probably have good reason

Re: Solr FTS - when does indexing happen?

2021-09-04 Thread Shawn Heisey
option, I see those deletes in Solr's log immediately. But if I press Shift-Del in Thunderbird (which immediately deletes the message, bypassing Trash), then it takes about 15 seconds before the Solr log shows the delete request. Is that expected? It's not causing me any problems, as

Re: Solr FTS - when does indexing happen?

2021-09-04 Thread Shawn Heisey
/configuration_manual/fts/solr/?highlight=fts%20user%20plugin There is an autoindex setting that neeeds to be set to "yes". I see something talking about autoindex, but it does not have an example so that I can see where it needs to go. I cannot work it out from what is there. With a little g

Re: Solr FTS - when does indexing happen?

2021-09-04 Thread Steve Dondley
On 2021-09-03 12:43 PM, Shawn Heisey wrote: I have Solr FTS on my dovecot install.  I followed the instructions on the dovecot wiki. How long a delay should I expect to see between new mail being delivered with the dovecot LDA and an indexing request sent to Solr?  Because I get a LOT of email

Solr FTS - when does indexing happen?

2021-09-03 Thread Shawn Heisey
I have Solr FTS on my dovecot install.  I followed the instructions on the dovecot wiki. How long a delay should I expect to see between new mail being delivered with the dovecot LDA and an indexing request sent to Solr?  Because I get a LOT of email from various mailing lists, and I do

Re: Dovecot - FTS Solr: disk usage & position information?

2021-09-02 Thread Alessio Cecchi
overloading the systems). Before the re-indexing we had 1.33 TiB in our Solr Indexes. After re-indexation, we had only 542 GiB, that's a 60% of our storage requirements for our FTS indexes :) this optimization also produce a less RAM requirements on Solr server? So far, we haven't been reported any

Re: Dovecot - FTS Solr: disk usage & position information?

2021-09-01 Thread Vincent Brillault
Dear all, Just a status update, in case this can help others. We went forward and disabled the position information indexing and the re-indexed of our mail data (over a couple of days to avoid overloading the systems). Before the re-indexing we had 1.33 TiB in our Solr Indexes. After re

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-26 Thread Aki Tuomi
> On 25/08/2021 23:05 Steve Dondley wrote: > > > > The search time was no better with it on than off. > > > > So I'm thinking I got something misconfigured somewhere. It seems IMAP > > may not be using solr to fetch results. But this would be odd si

Re: What kind of search response time are you setting with solr full text search?

2021-08-26 Thread azurit
Citát Steve Dondley : On 2021-08-24 08:53 PM, Steve Dondley wrote: MY SETUP: I have apache solr full text search enabled with dovecot. I have an inbox with about 40 subfolders. I'm using the roundcube web-based mail client. The find command is showing 15823 email files and apache solr

Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley
On 2021-08-24 08:53 PM, Steve Dondley wrote: MY SETUP: I have apache solr full text search enabled with dovecot. I have an inbox with about 40 subfolders. I'm using the roundcube web-based mail client. The find command is showing 15823 email files and apache solr reports the same number. I'm

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley
On 2021-08-25 04:32 PM, Shawn Heisey wrote: On 8/25/2021 2:10 PM, Steve Dondley wrote: And it looks like I'm running into a major bug in the slightly dated version of dovecot debian uses: https://www.mail-archive.com/dovecot@dovecot.org/msg78825.html Recently I did a fairly major

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Shawn Heisey
On 8/25/2021 2:10 PM, Steve Dondley wrote: And it looks like I'm running into a major bug in the slightly dated version of dovecot debian uses: https://www.mail-archive.com/dovecot@dovecot.org/msg78825.html Recently I did a fairly major upgrade.  I had an older Ubuntu release with

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley
I think this will be nailed once I figure out this issue. And it looks like I'm running into a major bug in the slightly dated version of dovecot debian uses: https://www.mail-archive.com/dovecot@dovecot.org/msg78825.html And this: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970692

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley
On 2021-08-25 04:05 PM, Steve Dondley wrote: The search time was no better with it on than off. So I'm thinking I got something misconfigured somewhere. It seems IMAP may not be using solr to fetch results. But this would be odd since I definitely do see a big improvements in times with fts

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley
The search time was no better with it on than off. So I'm thinking I got something misconfigured somewhere. It seems IMAP may not be using solr to fetch results. But this would be odd since I definitely do see a big improvements in times with fts plugins turned on when using roundcube. OK

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley
I'm inclined to believe the problem is not that high up the food chain. Because when I query IMAP on a single folder over telnet following the instructions found here: https://doc.dovecot.org/configuration_manual/fts/solr/, imap reports that it's taking 3 to 4 seconds to return results

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley
Random guess... Buffering? Whatever is sending to the browser isn't sending enough bytes to flush the buffer so the data is left in limbo until enough time goes by the buffer gets flushed anyways. Maybe a apache/nginx thing, php thing or browser thing. Remember its solr > dovecot > php

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread dovecot
So this looks really good and fast. So I think we can say with confidence solr is doing its job. So why is roundcube/dovecot taking so long to show the results? Random guess... Buffering? Whatever is sending to the browser isn't sending enough bytes to flush the buffer so the data is left

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley
On 2021-08-25 02:05 PM, Steve Dondley wrote: Try this in on the commandline of the Solr server: time curl "http://localhost:YYY/solr/dovecot/select?q=maynez=edismax=body+to+subject+cc+from; OK I had to modify the query path slightly to get it to work with my core to: time curl

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley
That query should search ALL emails that dovecot has indexed to Solr.  There is no restriction for mailbox or folder. OK. Try replacing "maynez" with something else that you know will be in the index. Did a search on "the". Still nothing. Very, very weird. What w

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Shawn Heisey
On 8/25/2021 12:05 PM, Steve Dondley wrote: OK I had to modify the query path slightly to get it to work with my core to: time curl http://localhost:8983/solr/dondley/select?q=maynez=edismax=body+to+subjec:t+ccfrom But it didn't return any results: I only have emails for this person

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley
Try this in on the commandline of the Solr server: time curl "http://localhost:YYY/solr/dovecot/select?q=maynez=edismax=body+to+subject+cc+from; OK I had to modify the query path slightly to get it to work with my core to: time curl http://localhost:8983/solr/dondley/select?q=m

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Shawn Heisey
On 8/25/2021 11:33 AM, Steve Dondley wrote: OK, I take this back. I did an imap search via telnet and solr reports the search takes about 3 to 4 seconds. Here's the output: a search text "maynez" Try this in on the commandline of the Solr server: time curl "http://loc

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley
This is a search for "a" which I had run several times, so Solr was serving it from its cache, and this time it only took 6 milliseconds.  It also shows what a facet can do.  The longest time I got for the "a" search was 15 milliseconds, before the query was in

Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley
One other data point from my experimenting that might shed some light on the problem: If I limit a search to a single folder instead of across all folders, it still takes 5 or 6 seconds for the results to appear. So that kind of destroys my theory that the problem might be caused by having

[OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Shawn Heisey
hundred emails are the ones that are take a much longer time. This is offtopic for this list, but I will try to help you.  If I am unsuccessful, you should raise the issue on the solr-users mailing list. How much of the total server memory of 4GB did you give to Solr for its heap

Re: What kind of search response time are you setting with solr full text search?

2021-08-24 Thread Gedalya
On 8/25/21 9:19 AM, Steve Dondley wrote: > I did some experimenting. I noticed that if the word I'm searching on is > fairly rare, results will pop up quickly, like in around 3 to 5 seconds. > Words that don't exist at all in any email returns nothing almost instantly. > > But words that appear

Re: What kind of search response time are you setting with solr full text search?

2021-08-24 Thread Steve Dondley
THE PROBLEM: When I do a full text search through all my inbox and all subfolders on a single word, search results are returned in about 10 to 15 seconds. This is better than the 40 seconds or so I'm getting when I turn off the fts and fts_solr plugins but still a little disappointing. I did

What kind of search response time are you setting with solr full text search?

2021-08-24 Thread Steve Dondley
MY SETUP: I have apache solr full text search enabled with dovecot. I have an inbox with about 40 subfolders. I'm using the roundcube web-based mail client. The find command is showing 15823 email files and apache solr reports the same number. I'm running a dedicated mail server with a 1 GB

Re: Dovecot - FTS Solr: disk usage & position information?

2021-08-05 Thread Shawn Heisey
On 8/5/2021 1:00 AM, Vincent Brillault wrote: Indeed, I should have. I'm using Solr 8.6, which is clearly not the same as Solr 6.2.0, but when looking at more recent versions of the documentation, no information about the use of each file appeared. That's why I was mentioning it was slightly

Re: Dovecot - FTS Solr: disk usage & position information?

2021-08-05 Thread Vincent Brillault
Dear Shawn, Thanks for your very complete answer! > This is completely off-topic for the dovecot list.  I am involved with > the Solr project, so I can discuss it.  My message will also be off > topic here. Sorry, maybe I didn't explain myself properly. I asked on the dovecot mai

Re: Dovecot - FTS Solr: disk usage & position information?

2021-08-04 Thread Shawn Heisey
On 8/4/2021 1:24 AM, Vincent Brillault wrote: On a local dovecot cluster currently hosting roughly 2.1TB of data, using Solr as its FTS backend, we now have 256GB of data in Solr, split in 12 shard (to which replication adds 256GB of data through 12 additional cores). I'm now trying to see

Dovecot - FTS Solr: disk usage & position information?

2021-08-04 Thread Vincent Brillault
Dear all, On a local dovecot cluster currently hosting roughly 2.1TB of data, using Solr as its FTS backend, we now have 256GB of data in Solr, split in 12 shard (to which replication adds 256GB of data through 12 additional cores). I'm now trying to see if we can optimize that data. Looking

Re: Panic on indexer-worker and Dovecot stop to search on Solr

2021-06-03 Thread Alessio Cecchi
Il 31/05/21 21:00, Aki Tuomi ha scritto: On 31/05/2021 21:58 Alessio Cecchi wrote: Il 31/05/21 18:17, Aki Tuomi ha scritto: On 31/05/2021 19:09 Alessio Cecchi wrote: Hi, I have setup on a (little busy) Dovecot server FTS with Solr and Virtual folder to enable "search in all fo

Re: Panic on indexer-worker and Dovecot stop to search on Solr

2021-05-31 Thread Aki Tuomi
> On 31/05/2021 21:58 Alessio Cecchi wrote: > > > Il 31/05/21 18:17, Aki Tuomi ha scritto: > > >> On 31/05/2021 19:09 Alessio Cecchi wrote: > >> > >> > >> Hi, > >> I have setup on a (little busy) Dovecot server FTS with So

Re: Panic on indexer-worker and Dovecot stop to search on Solr

2021-05-31 Thread Alessio Cecchi
Il 31/05/21 18:17, Aki Tuomi ha scritto: On 31/05/2021 19:09 Alessio Cecchi wrote: Hi, I have setup on a (little busy) Dovecot server FTS with Solr and Virtual folder to enable "search in all folders" for users. All works fine until for some users the indexer-worker pro

Re: Panic on indexer-worker and Dovecot stop to search on Solr

2021-05-31 Thread Aki Tuomi
> On 31/05/2021 19:09 Alessio Cecchi wrote: > > > Hi, > I have setup on a (little busy) Dovecot server FTS with Solr and Virtual > folder to enable "search in all folders" for users. > > All works fine until for some users the indexer-worker process cras

Panic on indexer-worker and Dovecot stop to search on Solr

2021-05-31 Thread Alessio Cecchi
Hi, I have setup on a (little busy) Dovecot server FTS with Solr and Virtual folder to enable "search in all folders" for users. All works fine until for some users the indexer-worker process crash. After this crash Dovecot stop to query Solr for new search in BODY, returning

Re: Only INBOX is searched when using gmail with dovecot FTS and solr

2021-04-16 Thread Sami Ketola
s://github.com/docker-mailserver/docker-mailserver to setup >> my mailserver and added solr for full-text search. Nearly everything is >> working as expected. But I have a problem with the full-text search, as I >> have a lot of folders in my mailbox and I just can't find any ma

Re: Only INBOX is searched when using gmail with dovecot FTS and solr

2021-04-12 Thread Marcel
Any ideas on this? I am sure that I can't be the only one having issues with the search function while using Gmail. Hi, I am using https://github.com/docker-mailserver/docker-mailserver to setup my mailserver and added solr for full-text search. Nearly everything is working as expected

Re: Incorrect value sent to solr

2021-04-08 Thread John Fawcett
On 08/04/2021 08:02, Aki Tuomi wrote: > Hi! > Thanks for reminding us, I'll make a ticket about this to avoid forgetting it > again. > > Aki Thanks to you and the team for taking it into consideration! PS is this list the best way to post patch proposals or is it easier for you to get code from

Re: Incorrect value sent to solr

2021-04-08 Thread Aki Tuomi
> On 07/04/2021 22:58 John Fawcett wrote: > > > On 07/04/2021 13:36, Łukasz Szczepański wrote: > > I'm not as familiar with C, but I don't see in solr backed in dovecot > > any  clue of subsequent queries for single mailbox lookup (which most > > mail clien

Re: Incorrect value sent to solr

2021-04-07 Thread John Fawcett
On 07/04/2021 13:36, Łukasz Szczepański wrote: > I'm not as familiar with C, but I don't see in solr backed in dovecot > any  clue of subsequent queries for single mailbox lookup (which most > mail client uses). There is a hard limit of 10 rows for multiple > ma

  1   2   3   4   5   6   7   8   >