date:20190104

Re: Solr -> Xapian ?

2019-01-04 Thread Joan Moreau via dovecot

Thank Stephan 


I basically need to know the role/description of each of the functions
of the fts_backend: 


struct fts_backend fts_backend_xapian = {
.name = "xapian",
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT, -> WHAT OTHER FLAGS ? 


{
fts_backend_xapian_alloc,
fts_backend_xapian_init,
fts_backend_xapian_deinit,
fts_backend_xapian_get_last_uid,
fts_backend_xapian_update_init,
fts_backend_xapian_update_deinit,
fts_backend_xapian_update_set_mailbox,
fts_backend_xapian_update_expunge,
fts_backend_xapian_update_set_build_key,
fts_backend_xapian_update_unset_build_key,
fts_backend_xapian_update_build_more,
fts_backend_xapian_refresh,
fts_backend_xapian_rescan,
fts_backend_xapian_optimize,
fts_backend_default_can_lookup,
fts_backend_xapian_lookup,
fts_backend_xapian_lookup_multi,
fts_backend_xapian_lookup_done
}
}; 

THank you 


On 2019-01-05 08:49, Stephan Bosch wrote:

Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot: 


Why not, but please guide me about the core structure (mandatory funcitons, 
etc..) of a typical Dovecot FTS plugin

The Dovecot API documentation is not exhaustive everywhere, but the basics are 
documented. The remaining questions can be answered by looking at examples 
found in similar plugins or the relevant API sources.

I know of one FTS plugin not written by Dovecot developers:

https://github.com/atkinsj/fts-elasticsearch

If you really wish to do something like this, just go ahead. It will not be a 
small effort though. As soon as you have concrete questions, we can help you 
(don't expect rapid responses though).

Regards,

Stephan.

Re: Solr - complete setup (update)

2019-01-04 Thread Stephan Bosch


Hi,

Op 04/01/2019 om 05:36 schreef Joan Moreau via dovecot:


Hi

This is the summary of my work with SOLR-Dovecot, in my *quest to 
reproduce the previoulsy excellent work of fts_squat*



@Aki : Based on the time I have spent on this, I would love to see you 
updating the Wiki with those improvements, and adding my name somewhere


@All : Hope it helps

I'll be going through the description below soon. I've recently 
independently installed fts-solr from scratch. Although this wasn't a 
flawless effort, I managed to get some basic indexing going. From this 
mail thread I understand that there are quite a few more problems than 
I've seen myself so far. Then again, I didn't perform extensive tests 
with actual searches.


Maybe we can turn all this into a test suite that we can run internally 
here at Dovecot. At the very least, the described Dovecot bugs need to 
be addressed and the wiki needs to be updated.


I'll get back to you.


Regards,

Stephan.





*- Installation:*

-> Create a clean install using the default, (at least in the 
Archlinux package), and do a "sudo -u solr solr create -c dovecot ". 
The config files are then in /opt/solr/server/solr/dovecot/conf and 
datafiles in /opt/solr/server/solr/dovecot/data


-> In /opt/solr/server/solr/dovecot/conf/solrconfig.xml:

 * around line 313, change false to 
true


 * around line 147, set 2000 
(or above)


 * around line 696 : uncomment hdr

 * around line 1127, before class="solr.UUIDUpdateProcessorFactory" name="uuid"/>, add 



 * around line 1161, delete the whole class="solr.AddSchemaFieldsUpdateProcessorFactory" 
name="add-schema-fields">


    * around line 1192, remove the whole name="add-unknown-fields-to-the-schema" ... />


-> Remove /opt/solr/server/solr/dovecot/conf/managed-schema

-> Change "schema.xml" by the one below to reproduce fts_squat 
behavior  (equivalent to " fts_squat = partial=3 full=25" in 
dovecot.conf) (note : such a huge trouble to replace a single line 
setup, anyway...)


-> Move /opt/solr/server/solr (or the subfolder data) to a partition 
with *space*, ideally ext4 or faster file system (it looks like Solr 
is not considering using a simple mysql database, which would make 
sense to avoid all the fuzz and let it transit to a non-java state, 
but that is another story)


-> Config of dovecot.conf is as below

-> The systemd unit shall specify high ulimit for files and proc (see 
below)


-> Increase the memory available for the JavaVM (I put 12Gb as I have 
quite a space on my server, but you may adapt it as per your specs) : 
in /opt/solr/bin/solr.in.sh, set SOLR_HEAP="12288m"


-> As Solr is complaining a lot, you may consider a filter for it in 
your syslog-ng or journald as it pollutes greatly your audit files


-> (re)Start solr (first) and dovecot by systemctl

-> Launch redindex ( doveadm fts rescan -u  )

-> wait for a big while to let the system re-index all your mail boxes


*- Bugs so far*

-> Line 620 of fts_solr dovecot plugin : the size oof header is 
improperly calculated ("huge header" warning for a simple email, which 
kilss the index of that considered email, so basically MOST emails as 
the calculation is wrong)


-> The UID returned by SOlr is to be considered as a STRING (and that 
is maybe the source of problem of the "out of bound" errors in 
fts_solr dovecot, as "long" is not enough)


-> Java errors : A lot of non sense for me, I am not expert in Java. 
But, with increased memory, it seems not crashing, even if complaining 
quite a lot in the logs





*---SCHEMA.XML in /opt/solr/server/solr/dovecot/conf*



id
autoGeneratePhraseQueries="true" positionIncrementGap="100">



catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" 
generateWordParts="1" splitOnNumerics="1" catenateAll="1" 
catenateWords="1" preserveOriginal="1"/>













autoGeneratePhraseQueries="true">




















stored="true"/>




stored="true"/>



stored="true"/>
stored="true"/>




*-- DOVECOT.CONF*

mail_plugins = fts fts_solr

plugin {
plugin = fts fts_solr managesieve sieve

fts = solr
fts_autoindex = yes
fts_enforced = yes
fts_solr = url=http://127.0.0.1:8983/solr/dovecot/

(replace 127.0.0.1 by your solr server if you want to use an external 
server)

(...)

}



*-- /etc/systemd/system/multi-user.target.wants/solr.service*

[Unit]
Description=Solr full text search engine
After=network.target

[Service]
Type=simple
User=solr
Group=solr
PrivateTmp=yes
WorkingDirectory=/opt/solr
*LimitNOFILE=65000*
*LimitNPROC=65000*
ExecStart=/opt/solr/bin/solr start -f

[Install]
WantedBy=multi-user.target

Re: Solr -> Xapian ?

2019-01-04 Thread Stephan Bosch




Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:


Why not, but please guide me about the core structure (mandatory 
funcitons, etc..) of a typical Dovecot FTS plugin


The Dovecot API documentation is not exhaustive everywhere, but the 
basics are documented. The remaining questions can be answered by 
looking at examples found in similar plugins or the relevant API sources.


I know of one FTS plugin not written by Dovecot developers:

https://github.com/atkinsj/fts-elasticsearch

If you really wish to do something like this, just go ahead. It will not 
be a small effort though. As soon as you have concrete questions, we can 
help you (don't expect rapid responses though).


Regards,

Stephan.

Re: Solr

2019-01-04 Thread Stephan Bosch




Op 02/01/2019 om 11:00 schreef Joan Moreau via dovecot:


Another bug appearing today:

Jan 02 09:59:08 
indexer-worker(j...@grosjo.net)<6777>: 
Warning:*fts-solr(j...@grosjo.net)*: Mailbox XX UID=121635*header 
size is huge, truncating*


header of the said email has nothing of "huge"


Does this happen consistently with that same message? Can you show it to me?

Regards,

Stephan.





On 2019-01-02 15:22, Joan Moreau via dovecot wrote:


Refinement of the schema.xml (below)


THis however does not solve the "no results" and "Out of range" 
errors in Dovecot and Solr





id
sortMissingLast="true" multiValued="true"/>
positionIncrementGap="0"/>
autoGeneratePhraseQueries="true" positionIncrementGap="100">



catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" 
generateWordParts="1" splitOnNumerics="1" catenateAll="1" 
catenateWords="1" preserveOriginal="1"/>









autoGeneratePhraseQueries="true">


minGramSize="3" />












stored="true"/>




stored="true"/>



stored="true"/>
stored="true"/>

Feature request: client bind address for replication

2019-01-04 Thread John Fawcett

Hi

would it be possible to consider a new parameter for replication:
doveadm_local_ip which allows the source ip address to be set when
connection to a remote dovecot for replication?

It could be useful when the network interface has multiple ips and a
specific one is used for mail services. See attached proposal. I tested
against 2.2.36 only. It applies correctly against 2.3.4 with a warning,.

John


--- dovecot-2.2.36/src/doveadm/doveadm-settings.c   2018-04-30 
15:52:05.0 +0200
+++ dovecot-2.2.36-new/src/doveadm/doveadm-settings.c   2019-01-04 
14:59:11.556270077 +0100
@@ -62,6 +62,7 @@
DEF(SET_UINT, doveadm_worker_count),
DEF(SET_IN_PORT, doveadm_port),
{ SET_ALIAS, "doveadm_proxy_port", 0, NULL },
+   DEF(SET_STR, doveadm_local_ip),
DEF(SET_STR, doveadm_username),
DEF(SET_STR, doveadm_password),
DEF(SET_STR, doveadm_allowed_commands),
@@ -91,6 +92,7 @@
.doveadm_socket_path = "doveadm-server",
.doveadm_worker_count = 0,
.doveadm_port = 0,
+   .doveadm_local_ip = "",
.doveadm_username = "doveadm",
.doveadm_password = "",
.doveadm_allowed_commands = "",
--- dovecot-2.2.36/src/doveadm/doveadm-settings.h   2017-10-05 
19:09:55.0 +0200
+++ dovecot-2.2.36-new/src/doveadm/doveadm-settings.h   2019-01-04 
14:57:32.906269791 +0100
@@ -19,6 +19,7 @@
const char *doveadm_socket_path;
unsigned int doveadm_worker_count;
in_port_t doveadm_port;
+   const char *doveadm_local_ip;
const char *doveadm_username;
const char *doveadm_password;
const char *doveadm_allowed_commands;
--- dovecot-2.2.36/src/doveadm/doveadm-util.c   2018-04-30 15:52:05.0 
+0200
+++ dovecot-2.2.36-new/src/doveadm/doveadm-util.c   2019-01-04 
15:26:09.326903786 +0100
@@ -100,19 +100,29 @@
 }
 
 static int
-doveadm_tcp_connect_port(const char *host, in_port_t port)
+doveadm_tcp_connect_port(const char *host, in_port_t port, char * my_ip)
 {
struct ip_addr *ips;
unsigned int ips_count;
int ret, fd;
-
+   struct ip_addr my_net_ip;
+   int use_my_ip = 0;
+if (strcmp(my_ip,"") == 0) {
+i_info("No doveadm_local_ip setting, local ip supplied by 
operating system");
+   }
+   if (net_addr2ip(my_ip, &my_net_ip)) {
+i_error("error using doveadm_local_ip setting %s",my_ip);
+   } else {
+i_info("Using doveadm_local_ip setting: %s",my_ip);
+   use_my_ip = 1;
+   }
alarm(DOVEADM_TCP_CONNECT_TIMEOUT_SECS);
ret = net_gethostbyname(host, &ips, &ips_count);
if (ret != 0) {
i_fatal("Lookup of host %s failed: %s",
host, net_gethosterror(ret));
}
-   fd = net_connect_ip_blocking(&ips[0], port, NULL);
+   fd = net_connect_ip_blocking(&ips[0], port, use_my_ip ? &my_net_ip : 
NULL);
if (fd == -1) {
i_fatal("connect(%s:%u) failed: %m",
net_ip2addr(&ips[0]), port);
@@ -121,7 +131,7 @@
return fd;
 }
 
-int doveadm_tcp_connect(const char *target, in_port_t default_port)
+int doveadm_tcp_connect(const char *target, in_port_t default_port, char * 
my_ip)
 {
const char *host;
in_port_t port;
@@ -130,18 +140,18 @@
i_fatal("Port not known for %s. Either set proxy_port "
"or use %s:port", target, target);
}
-   return doveadm_tcp_connect_port(host, port);
+   return doveadm_tcp_connect_port(host, port, my_ip);
 }
 
 int doveadm_connect_with_default_port(const char *path,
- in_port_t default_port)
+ in_port_t default_port, char * my_ip)
 {
int fd;
 
/* we'll assume UNIX sockets typically have an absolute path,
   or at the very least '/' somewhere. */
if (strchr(path, '/') == NULL)
-   fd = doveadm_tcp_connect(path, default_port);
+   fd = doveadm_tcp_connect(path, default_port, my_ip);
else {
fd = net_connect_unix(path);
if (fd == -1)
@@ -152,7 +162,7 @@
 
 int doveadm_connect(const char *path)
 {
-   return doveadm_connect_with_default_port(path, 0);
+   return doveadm_connect_with_default_port(path, 0, "");
 }
 
 int i_strccdascmp(const char *a, const char *b)
--- dovecot-2.2.36/src/doveadm/doveadm-util.h   2017-10-05 19:09:55.0 
+0200
+++ dovecot-2.2.36-new/src/doveadm/doveadm-util.h   2019-01-04 
14:55:52.263289702 +0100
@@ -13,9 +13,9 @@
 const char *unixdate2str(time_t timestamp);
 const char *doveadm_plugin_getenv(const char *name);
 int doveadm_connect(const char *path);
-int doveadm_tcp_connect(const char *target, in_port_t default_port);
+int doveadm_tcp_connect(const char *target, in_port_t default_port, char * 
my_ip);
 int doveadm_connect_with_default_port(const char *path,

Re: Solr -> Xapian ?

2019-01-04 Thread Joan Moreau via dovecot

Also, a description of the "to be" functions of the backend:

struct fts_backend fts_backend_xapian = {
.name = "xapian",
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT, -> WHAT OTHER FLAGS ?

{
fts_backend_xapian_alloc,
fts_backend_xapian_init,
fts_backend_xapian_deinit,
fts_backend_xapian_get_last_uid,
fts_backend_xapian_update_init,
fts_backend_xapian_update_deinit,
fts_backend_xapian_update_set_mailbox,
fts_backend_xapian_update_expunge,
fts_backend_xapian_update_set_build_key,
fts_backend_xapian_update_unset_build_key,
fts_backend_xapian_update_build_more,
fts_backend_xapian_refresh,
fts_backend_xapian_rescan,
fts_backend_xapian_optimize,
fts_backend_default_can_lookup,
fts_backend_xapian_lookup,
fts_backend_xapian_lookup_multi,
fts_backend_xapian_lookup_done
}
};

On 2019-01-04 20:33, Joan Moreau via dovecot wrote:

Yes but:

1 - is there a documentation of the main object ? (fts_backend, mail_user, mailbox, etc..)

2 - What are the mandatory functions ?

3 - Search : Supposedly, the FTS shall have several parameters : the keyword(s), the user & mailbox, and the fields (to, from, body, etc..) to be includude in the search. What is the function called in the plugin ?

4 - Indexing : Somehow, what is the logic ? fts core just ask to "index me this email of this mailbox" ? or this is delegated to the plugin to sort out which emails it has indexed yet or not ?

Thank you

On 2019-01-04 18:49, admin wrote:
A starting point would be to have a look at the current FTS plugins:

https://github.com/dovecot/core/tree/master/src/plugins/fts-solr
and
https://github.com/dovecot/core/tree/master/src/plugins/fts-squat

-M

Am Freitag, den 04.01.2019, 18:17 +0800 schrieb Joan Moreau via dovecot:

Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin

On 2019-01-04 17:20, Aki Tuomi wrote:
I hope you are aware that "linking with Xapian" requires somewhat more work than just -lxapian in linker? If you or someone feels like writing fts_xapian, go for it.

Aki

On 04 January 2019 at 08:20 Joan Moreau via dovecot wrote:

What about consedering linking Dovecot with Xapian librairies instead of
going to nightmare Solr ?

https://xapian.org/features

On 2019-01-02 17:10, John Tulp wrote:

On Wed, 2019-01-02 at 00:59 -0800, M. Balridge wrote: The main problem is :
After some time of indexing from Dovecot, Dovecot
returns errors (invalid SID, etc...) and Solr return "out of range
indexes" errors
I've been watching the progress of this thread with no small concern, mainly

because I've been tasked with providing a server-side email search facility
with a budget and manpower level that comes down to mainly *1*, i.e., me.

I was expecting, given the strongly worded language about "just use
lucene/SOLR" and "ignore squat", that I should invest time + effort into this
JAVA nightmare that is SOLR.

I started with squat and another word-indexor system that used out-of-band
(not a dovecot plugin) software to provide rapid (sub-second) searches through
tens-of-GB-scale mailboxes.

Unlike what I was led to believe, the squat indexes worked surprisingly well,
once you sorted out the odd resource size (ulimit-related) issues (vsz &
friends) limitations. I did notice the "worst-case" search performance have
worryingly high O(x) increases in time, but I'd not seen anything that was a
dealbreaker. It goes without saying that various substring searches worked as
expected, for the most part.

My experiences with SOLR were similar to Messr. Moreau's: lots of startup
errors with provided schemata files. Lots of JAVA nonsense issues. Lots of
sensitivity to WHICH Java runtime, etc, etc. I finally fixated a specific JVM,
version of SOLR, and dovecot to find the "best" working combination, only to
find that the searches didn't work out as expected. I expected to be able to
do date-ranging based searches. Didn't work. I expected to search CONTENTS of
emails, and despite many days of tweaks, I couldn't get it to index even the
basics like filenames/types of attachments, so I could exposed
attachment-based searching to my users.

So, without rancour or antipathy, I ask the entire list: has ANYONE gotten a
Dovecot/solr-fts-plugin setup to work that provides as a BASELINE, all of the
following functionality:

1) The ability to search for a string within any of the structured fields
(from/subject) that returns correct results?

2) The ability to search for any string within the BODY of emails, including
the MIME attachment boundaries?

3) The ability to do "ranging" searches for structures within emails that
decompose to "dates" or other simple-numeric data?

OPTIONALLY, and this is probably way outside of the scope of the above,
despite the fact that it's listed as a "selling point" of SOLR versus other
full text search engines:

4) The ability to do searches against any attachments that are able to be
post-processed and hyper-indexed by SOLR+Tika?

SOLR seems

Re: Solr -> Xapian ?

2019-01-04 Thread Joan Moreau via dovecot

Yes but:

1 - is there a documentation of the main object ? (fts_backend,
mail_user, mailbox, etc..)

2 - What are the mandatory functions ?

3 - Search : Supposedly, the FTS shall have several parameters : the
keyword(s), the user & mailbox, and the fields (to, from, body, etc..)
to be includude in the search. What is the function called in the plugin
?

4 - Indexing : Somehow, what is the logic ? fts core just ask to "index
me this email of this mailbox" ? or this is delegated to the plugin to
sort out which emails it has indexed yet or not ?

Thank you

On 2019-01-04 18:49, admin wrote:

A starting point would be to have a look at the current FTS plugins: