Re: Initial use of Xapian

[ellie timoney]

On Mon, Feb 19, 2018, at 11:38 AM, ellie timoney wrote:
> Hmm.  I suspect the -R and -n options are mutually exclusive.  -n is a 
> one-shot run of the named channel (as you see); -R is the rolling mode 
> (with no way to specify the channel).  And it looks like, as you've 
> discovered, the -n invocation takes precedence, and the -R gets ignored.

I am wrong about this: -n is the option for specifying which channel -R should 
roll on.  It should work the way I would like it to.  I don't yet know what's 
going on in your case.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Initial use of Xapian

[ellie timoney]

On Fri, Feb 16, 2018, at 4:02 AM, Lists Nethead wrote:
> Some more observations,
> 
> - When placed in START section, "squatter cmd="squatter -R -n  
> squatter", the process does the log and then just vanishes and does  
> nothing more while the "squatter" log just grows.

Hmm.  I suspect the -R and -n options are mutually exclusive.  -n is a one-shot 
run of the named channel (as you see); -R is the rolling mode (with no way to 
specify the channel).  And it looks like, as you've discovered, the -n 
invocation takes precedence, and the -R gets ignored.

I don't like this: I think it ought to support rolling mode on a channel name 
you decide.  I'll raise a github issue for this.  I'm not sure if the change 
will be included in a 3.0 release though, it depends on how intrusive the 
implementation ends up being.

In the meantime, since you're using the default squatter channel name of 
"squatter" anyway, it should be adequate to just run "squatter -R" from the 
START section.  But note that when invoked from START, you may need to kill the 
squatter process manually when shutting down Cyrus, depending on your 
environment.  Also, if the squatter process crashes or otherwise exits, master 
won't know how to restart it.

> - When under DAEMON with the -d option added to the above, it keeps running.
> 
> Is this the expected behaviour?

There is a known bug in 3.0.0..3.0.5, where squatter doesn't run correctly with 
-R under the DAEMON section unless the -d option is also provided.  There is a 
fix for this in progress, which will be included in 3.0.6.

When squatter -R is invoked from the DAEMON section, the master process is able 
to keep track of it, restart it if it dies, and shut it down cleanly when Cyrus 
is shut down.  This is the proper modern way to do it... once 3.0.6 is 
released. :)

> Further, while poking around, I found in cyrusdb_lmdb.c:
> "cryusdb_lmdb(%s): %s", db->fname, mdb_strerror(mr));
> which presumably is a speling error :-)

Heh wow, good find!

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Initial use of Xapian

[Lists Nethead]

Quoting Hajimu UMEMOTO :


Hi,


On Wed, 14 Feb 2018 11:25:12 -0600
Nic Bernstein  said:



I do understand that managing a project like this is quite a task
and v.3 is still young. Apart from the log issue and xapian I also
stumbled over other undocumented changes related to FreeBSD only I
think(?) in that binaries that before existed in bin/ are now
separated into sbin/ and libexec/, meaning that your scripts will of
course fail after the upgrade. I should produce a short writeup for
the FreeBSD camp.


nic> As for the FreeBSD path changing issues, that's up to the package
nic> creator. Please let them know.

I'm a maintainer of the FreeBSD ports of Cyrus IMAPd.
My understanding is that Cyrus IMAPd 3.X changed the directory layout
to separate sbin and libexec by default.  I did nothing special for
FreeBSD around here.  So, if the directory layout is same as 2.X, it
was changed by that distro, IMHO.



Yes, absolutely correct. It was a TL;DR from my side, sorry.

//per


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Initial use of Xapian

[Hajimu UMEMOTO]

Hi,

> On Wed, 14 Feb 2018 11:25:12 -0600
> Nic Bernstein  said:

> I do understand that managing a project like this is quite a task
> and v.3 is still young. Apart from the log issue and xapian I also
> stumbled over other undocumented changes related to FreeBSD only I
> think(?) in that binaries that before existed in bin/ are now
> separated into sbin/ and libexec/, meaning that your scripts will of
> course fail after the upgrade. I should produce a short writeup for
> the FreeBSD camp.

nic> As for the FreeBSD path changing issues, that's up to the package
nic> creator.  Please let them know.

I'm a maintainer of the FreeBSD ports of Cyrus IMAPd.
My understanding is that Cyrus IMAPd 3.X changed the directory layout
to separate sbin and libexec by default.  I did nothing special for
FreeBSD around here.  So, if the directory layout is same as 2.X, it
was changed by that distro, IMHO.

Sincerely,

--
Hajimu UMEMOTO
u...@mahoroba.org  u...@freebsd.org
http://www.mahoroba.org/~ume/

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Initial use of Xapian

[Lists Nethead]

Quoting Nic Bernstein :

[...]


Per,
I've created an issue #2248 for this on GitHub, and have just  
created PR #2249 which addresses it. Thanks for your feedback.


As for the FreeBSD path changing issues, that's up to the package  
creator. Please let them know.


Cheers,
 -nic

1: https://github.com/cyrusimap/cyrus-imapd/issues/2248
2: https://github.com/cyrusimap/cyrus-imapd/pull/2249


Thanks.

Some more observations,

- When placed in START section, "squatter cmd="squatter -R -n  
squatter", the process does the log and then just vanishes and does  
nothing more while the "squatter" log just grows.


- When under DAEMON with the -d option added to the above, it keeps running.

Is this the expected behaviour?

Further, while poking around, I found in cyrusdb_lmdb.c:
"cryusdb_lmdb(%s): %s", db->fname, mdb_strerror(mr));
which presumably is a speling error :-)



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Initial use of Xapian

[ellie timoney]

(re-sending this for the list, because I sent from the wrong address and it got 
eaten, doh!)

On Thu, Feb 15, 2018, at 1:32 PM, ellie timoney wrote:
> The change in the install directories for binaries occurred at the start 
> of the 3.0 series, and is documented in the 3.0.0 release notes:
> 
> https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.0.html
> 
> > * The handling of configure options has changed, dropping support for 
> > --with-cyrus-prefix and --with-service-path in favour of the more 
> > conventional --prefix, --bindir, --sbindir, and --libexecdir.
> > * Binaries executed by the master(8) service process are now installed to 
> > --libexecdir. Other binaries are installed to --bindir and/or --sbindir.
> > * The make installbinsymlinks target can be used to set up symlinks to 
> > everything in --bindir, if you need that in your environment.
> 
> I don't know whether the FreeBSD packager(s) have done anything 
> additional to this.
> 
> Cheers,
> 
> ellie
> 
> On Thu, Feb 15, 2018, at 4:25 AM, Nic Bernstein wrote:
> > On 02/14/2018 11:04 AM, Per olof Ljungmark wrote:
> > >
> > > Nic, thank you for the clarification, it makes real sense.
> > >
> > > I do understand that managing a project like this is quite a task and 
> > > v.3 is still young. Apart from the log issue and xapian I also 
> > > stumbled over other undocumented changes related to FreeBSD only I 
> > > think(?) in that binaries that before existed in bin/ are now 
> > > separated into sbin/ and libexec/, meaning that your scripts will of 
> > > course fail after the upgrade. I should produce a short writeup for 
> > > the FreeBSD camp.
> > >
> > > Cheers,
> > >
> > > //per
> > 
> > Per,
> > I've created an issue #2248 for this on GitHub, and have just created PR 
> > #2249 which addresses it.  Thanks for your feedback.
> > 
> > As for the FreeBSD path changing issues, that's up to the package 
> > creator.  Please let them know.
> > 
> > Cheers,
> >      -nic
> > 

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Initial use of Xapian

[Nic Bernstein]

On 02/14/2018 11:04 AM, Per olof Ljungmark wrote:

Quoting Nic Bernstein :


On 02/14/2018 10:35 AM, Lists Nethead wrote:

Quoting Nic Bernstein :

The advice to move it to START is actually based on a recently 
discovered bug, referred to in that issue report (#2234). It 
/should/ be in DAEMON, but for that bug, which has been 
fixed. The fix will be in the next release.


In general, the mailing is a grand place to start, and IRC is also 
your friend (#cyrus on freenode).


Cheers,
 -nic

On 02/14/2018 04:58 AM, Lists Nethead wrote:

Quoting Sebastian Hagedorn :

--On 14. Februar 2018 um 11:13:15 +0100 Lists Nethead 
 wrote:



I am testing Xapian in a 3.0.5 setup on FreeBSD using most default
setting. If I start imapd with "squatter cmd="squatter -R", more 
and more
"squatter" processes are created and eventually grabs all 
resources.


Where did you put that statement? You can't put it in the DAEMON 
section with 3.0.5. Put it in START instead. See this issue for 
more information:





A-ha. So it is better to look at Github instead of the mailing 
list apparently.


https://www.cyrusimap.org/imap/reference/manpages/systemcommands/squatter.html 
still advises DAEMON, that is why it ended up there.


Thanks to the advise above and I believe I got it working now.

One more thing though, if replication is involved, it appears one 
needs one log for squatter and another for replication, am I 
correct? I got replication to work only after I added another log, 
like,


sync_log_channels: squatter replog
and
syncclient   cmd="/usr/local/cyrus/sbin/sync_client -r -n replog"

Not sure if this is mentioned in the docs somewhere.

Cheers,

//per


In general, for each consumer of a sync_log, a new log should be 
defined in `sync_log_channels:`.  I use the word "consumer" there 
intentionally, as the processes which use these logs actually consume 
them, and leave nothing behind (assuming it all works as expected).  
So if more than one process tries to eat up the same log, you'll have 
problems.


The overloading of the sync_log framework for purposes beyond 
replication is new in 3.0, so we're still getting the documentation 
up to snuff in that regard.  However, the documentation already makes 
this concept clear in that when using more than one replica you need 
to specify more than one sync_log via the `sync_log_channels:` 
directive (see 
https://www.cyrusimap.org/imap/reference/admin/sop/replication.html#channels 
for details).


We obviously do need to produce more generalized documentation for 
this whole scheme, and I'll be using this discussion as a guide in 
that regard.  sync_log, as the name implies, started life as a way 
for the "master" server to provide a list of "units" -- either users 
or mailboxes -- it has touched, so that a replica knows what to 
request in updates.  This is such a useful concept, however, that it 
is spreading to other subsystems which need to know what might have 
changed in a potentially large data set (the typical mail store) and 
so we need to explain this not just in the Replication documentation, 
but in a more general way.


Note also that there is a `sync_log_unsuppressable_channels:` 
directive, which defaults to "squatter".  This is defined as:


   If specified, the named channels are exempt from the effect of
   setting sync_log_chain:off, i.e. they are always logged to by
   the sync_server process. This is only really useful to allow
   rolling search indexing on a replica.

If you are going to use a name other than "squatter" for your rolling 
indexing sync_log channel, then you need to update this as well.




Nic, thank you for the clarification, it makes real sense.

I do understand that managing a project like this is quite a task and 
v.3 is still young. Apart from the log issue and xapian I also 
stumbled over other undocumented changes related to FreeBSD only I 
think(?) in that binaries that before existed in bin/ are now 
separated into sbin/ and libexec/, meaning that your scripts will of 
course fail after the upgrade. I should produce a short writeup for 
the FreeBSD camp.


Cheers,

//per


Per,
I've created an issue #2248 for this on GitHub, and have just created PR 
#2249 which addresses it.  Thanks for your feedback.


As for the FreeBSD path changing issues, that's up to the package 
creator.  Please let them know.


Cheers,
    -nic

1: https://github.com/cyrusimap/cyrus-imapd/issues/2248
2: https://github.com/cyrusimap/cyrus-imapd/pull/2249
<>
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Initial use of Xapian

[Nic Bernstein]

On 02/14/2018 10:35 AM, Lists Nethead wrote:

Quoting Nic Bernstein :

The advice to move it to START is actually based on a recently 
discovered bug, referred to in that issue report (#2234). It 
/should/ be in DAEMON, but for that bug, which has been fixed. 
The fix will be in the next release.


In general, the mailing is a grand place to start, and IRC is also 
your friend (#cyrus on freenode).


Cheers,
 -nic

On 02/14/2018 04:58 AM, Lists Nethead wrote:

Quoting Sebastian Hagedorn :

--On 14. Februar 2018 um 11:13:15 +0100 Lists Nethead 
 wrote:



I am testing Xapian in a 3.0.5 setup on FreeBSD using most default
setting. If I start imapd with "squatter cmd="squatter -R", more 
and more

"squatter" processes are created and eventually grabs all resources.


Where did you put that statement? You can't put it in the DAEMON 
section with 3.0.5. Put it in START instead. See this issue for 
more information:





A-ha. So it is better to look at Github instead of the mailing list 
apparently.


https://www.cyrusimap.org/imap/reference/manpages/systemcommands/squatter.html 
still advises DAEMON, that is why it ended up there.


Thanks to the advise above and I believe I got it working now.

One more thing though, if replication is involved, it appears one 
needs one log for squatter and another for replication, am I correct? 
I got replication to work only after I added another log, like,


sync_log_channels: squatter replog
and
syncclient   cmd="/usr/local/cyrus/sbin/sync_client -r -n replog"

Not sure if this is mentioned in the docs somewhere.

Cheers,

//per


In general, for each consumer of a sync_log, a new log should be defined 
in `sync_log_channels:`.  I use the word "consumer" there intentionally, 
as the processes which use these logs actually consume them, and leave 
nothing behind (assuming it all works as expected).  So if more than one 
process tries to eat up the same log, you'll have problems.


The overloading of the sync_log framework for purposes beyond 
replication is new in 3.0, so we're still getting the documentation up 
to snuff in that regard.  However, the documentation already makes this 
concept clear in that when using more than one replica you need to 
specify more than one sync_log via the `sync_log_channels:` directive 
(see 
https://www.cyrusimap.org/imap/reference/admin/sop/replication.html#channels 
for details).


We obviously do need to produce more generalized documentation for this 
whole scheme, and I'll be using this discussion as a guide in that 
regard.  sync_log, as the name implies, started life as a way for the 
"master" server to provide a list of "units" -- either users or 
mailboxes -- it has touched, so that a replica knows what to request in 
updates.  This is such a useful concept, however, that it is spreading 
to other subsystems which need to know what might have changed in a 
potentially large data set (the typical mail store) and so we need to 
explain this not just in the Replication documentation, but in a more 
general way.


Note also that there is a `sync_log_unsuppressable_channels:` directive, 
which defaults to "squatter".  This is defined as:


   If specified, the named channels are exempt from the effect of
   setting sync_log_chain:off, i.e. they are always logged to by
   the sync_server process. This is only really useful to allow
   rolling search indexing on a replica.

If you are going to use a name other than "squatter" for your rolling 
indexing sync_log channel, then you need to update this as well.


Cheers,
    -nic
<>
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Initial use of Xapian

[Lists Nethead]

Quoting Nic Bernstein :

The advice to move it to START is actually based on a recently  
discovered bug, referred to in that issue report (#2234). It  
/should/ be in DAEMON, but for that bug, which has been fixed.  
The fix will be in the next release.


In general, the mailing is a grand place to start, and IRC is also  
your friend (#cyrus on freenode).


Cheers,
 -nic

On 02/14/2018 04:58 AM, Lists Nethead wrote:

Quoting Sebastian Hagedorn :

--On 14. Februar 2018 um 11:13:15 +0100 Lists Nethead  
 wrote:



I am testing Xapian in a 3.0.5 setup on FreeBSD using most default
setting. If I start imapd with "squatter cmd="squatter -R", more and more
"squatter" processes are created and eventually grabs all resources.


Where did you put that statement? You can't put it in the DAEMON  
section with 3.0.5. Put it in START instead. See this issue for  
more information:





A-ha. So it is better to look at Github instead of the mailing list  
apparently.


https://www.cyrusimap.org/imap/reference/manpages/systemcommands/squatter.html still advises DAEMON, that is why it ended up  
there.


Thanks to the advise above and I believe I got it working now.

One more thing though, if replication is involved, it appears one  
needs one log for squatter and another for replication, am I correct?  
I got replication to work only after I added another log, like,


sync_log_channels: squatter replog
and
syncclient   cmd="/usr/local/cyrus/sbin/sync_client -r -n replog"

Not sure if this is mentioned in the docs somewhere.

Cheers,

//per


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Initial use of Xapian

[Nic Bernstein]

The advice to move it to START is actually based on a recently 
discovered bug, referred to in that issue report (#2234).  It /should/ 
be in DAEMON, but for that bug, which has been fixed.  The fix will be 
in the next release.


In general, the mailing is a grand place to start, and IRC is also your 
friend (#cyrus on freenode).


Cheers,
    -nic

On 02/14/2018 04:58 AM, Lists Nethead wrote:

Quoting Sebastian Hagedorn :

--On 14. Februar 2018 um 11:13:15 +0100 Lists Nethead 
 wrote:



I am testing Xapian in a 3.0.5 setup on FreeBSD using most default
setting. If I start imapd with "squatter cmd="squatter -R", more and 
more

"squatter" processes are created and eventually grabs all resources.


Where did you put that statement? You can't put it in the DAEMON 
section with 3.0.5. Put it in START instead. See this issue for more 
information:





A-ha. So it is better to look at Github instead of the mailing list 
apparently.


https://www.cyrusimap.org/imap/reference/manpages/systemcommands/squatter.html 



still advises DAEMON, that is why it ended up there.

Thanks,




Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


--
Nic Bernstein n...@onlight.com
Onlight Inc.  www.onlight.com
6525 W Bluemound Rd., Ste 24  v. 414.272.4477
Milwaukee, Wisconsin  53213-4073  f. 414.290.0335


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Initial use of Xapian

[Lists Nethead]

Quoting Sebastian Hagedorn :

--On 14. Februar 2018 um 11:13:15 +0100 Lists Nethead  
 wrote:



I am testing Xapian in a 3.0.5 setup on FreeBSD using most default
setting. If I start imapd with "squatter cmd="squatter -R", more and more
"squatter" processes are created and eventually grabs all resources.


Where did you put that statement? You can't put it in the DAEMON  
section with 3.0.5. Put it in START instead. See this issue for more  
information:





A-ha. So it is better to look at Github instead of the mailing list  
apparently.


https://www.cyrusimap.org/imap/reference/manpages/systemcommands/squatter.html

still advises DAEMON, that is why it ended up there.

Thanks,




Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Initial use of Xapian

[Sebastian Hagedorn]

--On 14. Februar 2018 um 11:13:15 +0100 Lists Nethead  
wrote:



I am testing Xapian in a 3.0.5 setup on FreeBSD using most default
setting. If I start imapd with "squatter cmd="squatter -R", more and more
"squatter" processes are created and eventually grabs all resources.


Where did you put that statement? You can't put it in the DAEMON section 
with 3.0.5. Put it in START instead. See this issue for more information:



--
   .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:.
.:.Regionales Rechenzentrum (RRZK).:.
  .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.

pgp1W0y769zI7.pgp
Description: PGP signature

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Initial use of Xapian

[Lists Nethead]

Hi,

I am testing Xapian in a 3.0.5 setup on FreeBSD using most default  
setting. If I start imapd with "squatter cmd="squatter -R", more and  
more "squatter" processes are created and eventually grabs all  
resources.


The mail store is only 136G but it seems like the system wants to  
index all mailboxes/messages at the same time instead of walking over  
the store one by one like the old squatter process.


So, is this normal as a first time use of Xapian? Can I limit the  
numer of squatter processes somehow?


Thanks,

//per


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus