Re: Xapian partition definition when using archiving?
Ellie,
Thanks for weighing in on this, and for doing so on a weekend! :-)
I've just pushed some documentation edits in PR #2224 as a first attempt
at clarifying this stuff.
Cheers,
-nic
On 12/28/2017 09:22 PM, ellie timoney wrote:
> It's really the default search tier to use, and has no direct
relationship to the "default" partition.
Yep, or at least, this is my understanding too.
> In other words, this is just another circumstance where seemingly
obvious partition names (like default-default) get us into
documentation trouble. Right?
We trip over this all the time... and it's hard to document because
the trivial case and the complicated cases are so divergent. Even at
FM, where our setup is pretty advanced, it's still relatively simple,
in that we only have a single partition-/name/ entry ("default") per
Cyrus instance. So all of our /foo/partition-/name/ settings are
/foo/partition-default!
I guess what I'm getting at is there's a few dimensions of complexity
here. I think the "multiple-named-partitions" dimension of complexity
might be reasonably well documented, but then each additional
dimension (e.g. archive partitions, search tiers, etc) is only
documented for the single-named-default-partition case.
It'd be great if the documentation could move away from the "single
partition is named 'default'" scheme -- I think that would ease a LOT
of confusion for people trying to understand the different dimensions
all at once. But I don't really have any ideas for what might be a
better name for it. I haven't seen real-world multi-partition Cyrus
instances, so I don't know the use cases for having multiple
partitions, so it's hard to suggest a good name for someone's first
and maybe only partition.
Cheers,
ellie
On Fri, Dec 29, 2017, at 4:47 AM, Nic Bernstein wrote:
Bron, et al.,
Okay, one more time around on this. When I try a version of the
partition layout listed below, in my original post, and run "cyr_info
conf-lint" against it, I get all of the "/yadda/searchtier: blah"
lines thrown back at me. I had assumed that "/default/searchtier:
blah" was to specify to which tier to index the default partition,
but that's wrong, isn't it? It's really the default search tier to
use, and has no direct relationship to the "default" partition.
Am I correct in this new interpretation? It is implied by the man
page for imapd.conf (derived from lib/imapoptions):
defaultsearchtier:
Name of the default tier that messages will be indexed to.
Search indexes can be organized in tiers to allow index
storage in different directories and physical media. See the man
page of squatter for details. The default search
tier also requires the definition of an according
searchtierpartition-name entry.
This option MUST be specified for xapian search.
...
searchtierpartition-name:
The pathname where to store the xapian search indexes of
searchtier for mailboxes of partition name. This must be
configured for the defaultsearchtier and any additional search
tier (see squatter for details).
For example: if defaultpartition is defined as part1 and
defaultsearchtier as tier1 then the configuration must
contain an entry tier1partitionname-part1 that defines the
path where to store this tier1's search index for the
part1 partition.
This option MUST be specified for xapian search.
This is all so muddled, due to the use of the word "default" as an
embedded string within so many settings, some of which refer to a
default value and some of which refer to a partition called
"default". I really think we need to go through the documentation
from top to bottom and weed out such confusing language. The same is
true for the various uses of the word "archive" and some others.
This sort of thing is clearly leading to some folks just giving up on
complex Cyrus features, the implementation of which depend upon
piercing the veil of confusion surround this language (he says in
frustration).
Oh, and that imapd.conf(5) comment "(see sqautter for details)" is
useless, as the squatter(8) man page says nothing about this (damn
manpage writers!).
Thanks in advance,
-nic
On 12/20/2017 04:04 PM, Bron Gondwana wrote:
Totally! The name "archive" is overused. It could be called
something else easily enough.
Bron.
On Thu, 21 Dec 2017, at 01:05, Nic Bernstein wrote:
Bron,
Thanks for the swift reply. So if I understand this correctly, the
"archivesearchpartition-default" is named such because it's for the
archive location of Xapian search data, not because it's Xapian
search data from an Archive partition. Is that correct? In other
words, this is just another circumstance where seemingly obvious
partition names (like default-default) get us into documentation
trouble. Right?
Thanks
Re: Xapian partition definition when using archiving?
> It's really the default search tier to use, and has no direct
> relationship to the "default" partition.
Yep, or at least, this is my understanding too.
> In other words, this is just another circumstance where seemingly
> obvious partition names (like default-default) get us into
> documentation trouble. Right?
We trip over this all the time... and it's hard to document because
the trivial case and the complicated cases are so divergent. Even
at FM, where our setup is pretty advanced, it's still relatively
simple, in that we only have a single partition-*name* entry
("default") per Cyrus instance. So all of our *foo*partition-*name*
settings are *foo*partition-default!
I guess what I'm getting at is there's a few dimensions of complexity
here. I think the "multiple-named-partitions" dimension of complexity
might be reasonably well documented, but then each additional dimension
(e.g. archive partitions, search tiers, etc) is only documented for the
single-named-default-partition case.
It'd be great if the documentation could move away from the "single
partition is named 'default'" scheme -- I think that would ease a LOT of
confusion for people trying to understand the different dimensions all
at once. But I don't really have any ideas for what might be a better
name for it. I haven't seen real-world multi-partition Cyrus instances,
so I don't know the use cases for having multiple partitions, so it's
hard to suggest a good name for someone's first and maybe only
partition.
Cheers,
ellie
On Fri, Dec 29, 2017, at 4:47 AM, Nic Bernstein wrote:
> Bron, et al.,
> Okay, one more time around on this. When I try a version of the
> partition layout listed below, in my original post, and run "cyr_info
> conf-lint" against it, I get all of the "*yadda*searchtier: blah"
> lines thrown back at me. I had assumed that "*default*searchtier:
> blah" was to specify to which tier to index the default partition, but
> that's wrong, isn't it? It's really the default search tier to use,
> and has no direct relationship to the "default" partition.>
> Am I correct in this new interpretation? It is implied by the man
> page for imapd.conf (derived from lib/imapoptions):>
> defaultsearchtier: Name of the default tier
> that messages will be indexed to. Search indexes can be
> organized in tiers to allow index storage in different
> directories and physical media. See the man page of squatter
> for details. The default search tier also requires the
> definition of an according searchtierpartition-name entry.
> This option MUST be specified for xapian search. ...
> searchtierpartition-
> name: The pathname where to store the xapian search
> indexes of searchtier for mailboxes of partition name. This
> must be configured for the defaultsearchtier and any
> additional search tier (see squatter for details). For
> example: if defaultpartition is defined as part1 and
> defaultsearchtier as tier1 then the configuration must
> contain an entry tier1partitionname-part1 that defines the
> path where to store this tier1's search index for the part1
> partition. This option MUST be specified for xapian search.
>> This is all so muddled, due to the use of the word "default" as an
> embedded string within so many settings, some of which refer to a
> default value and some of which refer to a partition called "default".
> I really think we need to go through the documentation from top to
> bottom and weed out such confusing language. The same is true for the
> various uses of the word "archive" and some others. This sort of
> thing is clearly leading to some folks just giving up on complex Cyrus
> features, the implementation of which depend upon piercing the veil of
> confusion surround this language (he says in frustration).>
> Oh, and that imapd.conf(5) comment "(see sqautter for details)" is
> useless, as the squatter(8) man page says nothing about this (damn
> manpage writers!).>
> Thanks in advance,
> -nic
>
>
> On 12/20/2017 04:04 PM, Bron Gondwana wrote:
>> Totally! The name "archive" is overused. It could be called
>> something else easily enough.>>
>> Bron.
>>
>>
>> On Thu, 21 Dec 2017, at 01:05, Nic Bernstein wrote:
>>> Bron,
>>> Thanks for the swift reply. So if I understand this correctly, the
>>> "archivesearchpartition-default" is named such because it's for the
>>> archive location of Xapian search data, not because it's Xapian
>>> search data from an Archive partition. Is that correct? In other
>>> words, this is just another circumstance where seemingly obvious
>>> partition names (like default-default) get us into documentation
>>> trouble. Right?>>>
>>> Thanks again,
>>> -nic
>>>
>>>
>>> On 12/20/2017 05:39 AM, Bron Gondwana wrote:
Hi Nic,
The Xapian partitions are entirely separate from archive
Re: Xapian partition definition when using archiving?
Bron, et al., Okay, one more time around on this. When I try a version of the partition layout listed below, in my original post, and run "cyr_info conf-lint" against it, I get all of the "/yadda/searchtier: blah" lines thrown back at me. I had assumed that "/default/searchtier: blah" was to specify to which tier to index the default partition, but that's wrong, isn't it? It's really the default search tier to use, and has no direct relationship to the "default" partition. Am I correct in this new interpretation? It is implied by the man page for imapd.conf (derived from lib/imapoptions): defaultsearchtier: Name of the default tier that messages will be indexed to. Search indexes can be organized in tiers to allow index storage in different directories and physical media. See the man page of squatter for details. The default search tier also requires the definition of an according searchtierpartition-name entry. This option MUST be specified for xapian search. ... searchtierpartition-name: The pathname where to store the xapian search indexes of searchtier for mailboxes of partition name. This must be configured for the defaultsearchtier and any additional search tier (see squatter for details). For example: if defaultpartition is defined as part1 and defaultsearchtier as tier1 then the configuration must contain an entry tier1partitionname-part1 that defines the path where to store this tier1's search index for the part1 partition. This option MUST be specified for xapian search. This is all so muddled, due to the use of the word "default" as an embedded string within so many settings, some of which refer to a default value and some of which refer to a partition called "default". I really think we need to go through the documentation from top to bottom and weed out such confusing language. The same is true for the various uses of the word "archive" and some others. This sort of thing is clearly leading to some folks just giving up on complex Cyrus features, the implementation of which depend upon piercing the veil of confusion surround this language (he says in frustration). Oh, and that imapd.conf(5) comment "(see sqautter for details)" is useless, as the squatter(8) man page says nothing about this (damn manpage writers!). Thanks in advance, -nic On 12/20/2017 04:04 PM, Bron Gondwana wrote: Totally! The name "archive" is overused. It could be called something else easily enough. Bron. On Thu, 21 Dec 2017, at 01:05, Nic Bernstein wrote: Bron, Thanks for the swift reply. So if I understand this correctly, the "archivesearchpartition-default" is named such because it's for the archive location of Xapian search data, not because it's Xapian search data from an Archive partition. Is that correct? In other words, this is just another circumstance where seemingly obvious partition names (like default-default) get us into documentation trouble. Right? Thanks again, -nic On 12/20/2017 05:39 AM, Bron Gondwana wrote: Hi Nic, The Xapian partitions are entirely separate from archive partitions. The indexing code will find the file in the correct location if it needs to read it. Here's an example from one of our servers: defaultpartition: default defaultsearchtier: temp partition-default: /mnt/ssd21d2/sloti21d2t40/store254/spool tempsearchpartition-default: /tmpfs-search/sloti21d2t40 metasearchpartition-default: /mnt/ssd21d2/sloti21d2t40/store254/search datasearchpartition-default: /mnt/i21d2search/sloti21d2t40/store254/search archivesearchpartition-default: /mnt/i21d2search/sloti21d2t40/store254/search-archive archivepartition-default: /mnt/i21d2t40/sloti21d2t40/store254/spool-archive (clearly auto-generated!) Bron. On Wed, 20 Dec 2017, at 07:32, Nic Bernstein wrote: Bron, et al., We're about to set up a whole new bunch of partitions in support of Xapian indexing, for a 2.5.10-to-3.0.4 upgrade, and then will be introducing archive functionality, too. How do archive partitions and Xapian partitions interact? For example, the server currently has the following in imapd.conf: defaultpartition: default partition-default: /var/mailstores/default metapartition-default: /var/imapmeta/default partition-1: /var/mailstores/1 partition-2: /var/mailstores/2 partition-3: /var/mailstores/3 partition-4: /var/mailstores/4 ... partition-29: /var/mailstores/29 partition-30: /var/mailstores/30 partition-100: /var/mailstores/100 partition-temp: /var/mailstores/temp ... # non-default metapartitions metapartition-1: /var/imapmeta/1 metapartition-2: /var/imapmeta/2 metapartition-3: /var/imapmeta/3 metapartition-4: /var/imapmeta/4 ... metapartition-29: /var/imapmeta/29 metapartition-30:
Re: Xapian partition definition when using archiving?
Totally! The name "archive" is overused. It could be called something else easily enough. Bron. On Thu, 21 Dec 2017, at 01:05, Nic Bernstein wrote: > Bron, > Thanks for the swift reply. So if I understand this correctly, the > "archivesearchpartition-default" is named such because it's for the > archive location of Xapian search data, not because it's Xapian > search data from an Archive partition. Is that correct? In other > words, this is just another circumstance where seemingly obvious > partition names (like default-default) get us into documentation > trouble. Right?> > Thanks again, > -nic > > > On 12/20/2017 05:39 AM, Bron Gondwana wrote: >> Hi Nic, >> >> The Xapian partitions are entirely separate from archive partitions. >> The indexing code will find the file in the correct location if it >> needs to read it.>> >> Here's an example from one of our servers: >> >> defaultpartition: default >> defaultsearchtier: temp >> partition-default: /mnt/ssd21d2/sloti21d2t40/store254/spool >> tempsearchpartition-default: /tmpfs-search/sloti21d2t40 >> metasearchpartition-default: >> /mnt/ssd21d2/sloti21d2t40/store254/search>> datasearchpartition-default: >> /mnt/i21d2search/sloti21d2t40/store254/search>> >> archivesearchpartition-default: >> /mnt/i21d2search/sloti21d2t40/store254/search- >> archive>> archivepartition-default: >> /mnt/i21d2t40/sloti21d2t40/store254/spool- >> archive>> >> (clearly auto-generated!) >> >> Bron. >> >> >> On Wed, 20 Dec 2017, at 07:32, Nic Bernstein wrote: >>> Bron, et al., >>> We're about to set up a whole new bunch of partitions in support of >>> Xapian indexing, for a 2.5.10-to-3.0.4 upgrade, and then will be >>> introducing archive functionality, too. How do archive partitions >>> and Xapian partitions interact?>>> >>> For example, the server currently has the following in imapd.conf: >>> defaultpartition: default partition-default: /var/mailstores/default metapartition-default: /var/imapmeta/default partition-1: /var/mailstores/1 partition-2: /var/mailstores/2 partition-3: /var/mailstores/3 partition-4: /var/mailstores/4 ... partition-29: /var/mailstores/29 partition- 30: /var/mailstores/30 partition-100: /var/mailstores/100 partition- temp: /var/mailstores/temp ... # non-default metapartitions metapartition- 1: /var/imapmeta/1 metapartition-2: /var/imapmeta/2 metapartition- 3: /var/imapmeta/3 metapartition-4: /var/imapmeta/4 ... metapartition- 29: /var/imapmeta/29 metapartition-30: /var/imapmeta/30 metapartition- 100: /var/imapmeta/100 metapartition-temp: /var/imapmeta/temp >>> Going by the documentation, which I wrote with help from you good >>> folk at Fastmail, the Archive partition scheme might look something >>> like this:>>> >>> https://www.cyrusimap.org/imap/reference/admin/locations/archive-partitions.html>>> archivepartition-default: /var/mailarchives/default archivepartition- 1: /var/mailarchives/1 archivepartition-2: /var/mailarchives/2 archivepartition- 3: /var/mailarchives/3 archiveartition-4: /var/mailarchives/4 ... archivepartition-29: /var/mailarchives/29 archivepartition-30: /var/mailarchives/30 archivepartition-100: /var/mailarchives/100 archivepartition-temp: /var/mailarchives/temp >>> And the Xapian partition structure to mate with this would look >>> something like this (again, from the docs):>>> >>> https://www.cyrusimap.org/imap/developer/install-xapian.html >>> defaultpartition: default partition-default: /var/mailstores/default search_engine: xapian search_index_headers: no search_batchsize: 8192 defaultsearchtier: t1 1searchtier: t1 2searchtier: t1 3searchtier: t1 4searchtier: t1 ... 29searchtier: t1 30searchtier: t1 100searchtier: t1 tempsearchtier: t1 ... t1searchpartition- default: /var/search/default t1searchpartition-1: /var/search/1 t1searchpartition- 2: /var/search/2 t1searchpartition-3: /var/search/3 t1searchpartition- 4: /var/search/4 ... t1searchpartition-29: /var/search/29 t1searchpartition- 30: /var/search/30 t1searchpartition-100: /var/search/100 t1searchpartition- temp: /var/search/temp >>> First question, since there's no examples to work from; Is this >>> Xapian layout correct?>>> >>> Do I need to define & create Xapian partitions for the metadata >>> partitions, as is indirectly implied in Bron's original email on >>> this topic:>>> >>> https://lists.tartarus.org/pipermail/xapian-discuss/2014-October/009112.html>>> >>> >>> Also, how do these Xapian and Archive, interact? Do I need to add a >>> separate Xapian partition for each Archive partition, or will the >>> Archive partition be treated like a child of the non-Archive >>> partition? (again, implied but not directly addressed in that >>> email).>>> >>> Any guidance gladly accepted, and whatever I learn will be >>> repackaged into more complete documentation
Re: Xapian partition definition when using archiving?
Hi Nic, The Xapian partitions are entirely separate from archive partitions. The indexing code will find the file in the correct location if it needs to read it. Here's an example from one of our servers: defaultpartition: default defaultsearchtier: temp partition-default: /mnt/ssd21d2/sloti21d2t40/store254/spool tempsearchpartition-default: /tmpfs-search/sloti21d2t40 metasearchpartition-default: /mnt/ssd21d2/sloti21d2t40/store254/search datasearchpartition-default: /mnt/i21d2search/sloti21d2t40/store254/searcharchivesearchpartition-default: /mnt/i21d2search/sloti21d2t40/store254/search-archivearchivepartition-default: /mnt/i21d2t40/sloti21d2t40/store254/spool-archive (clearly auto-generated!) Bron. On Wed, 20 Dec 2017, at 07:32, Nic Bernstein wrote: > Bron, et al., > We're about to set up a whole new bunch of partitions in support of > Xapian indexing, for a 2.5.10-to-3.0.4 upgrade, and then will be > introducing archive functionality, too. How do archive partitions > and Xapian partitions interact?> > For example, the server currently has the following in imapd.conf: > >> defaultpartition: default partition-default: /var/mailstores/default >> metapartition-default: /var/imapmeta/default partition-1: >> /var/mailstores/1 partition-2: /var/mailstores/2 partition-3: >> /var/mailstores/3 partition-4: /var/mailstores/4 ... partition-29: >> /var/mailstores/29 partition-30: /var/mailstores/30 partition-100: >> /var/mailstores/100 partition-temp: /var/mailstores/temp ... # non- >> default metapartitions metapartition-1: /var/imapmeta/1 metapartition- >> 2: /var/imapmeta/2 metapartition-3: /var/imapmeta/3 metapartition-4: >> /var/imapmeta/4 ... metapartition-29: /var/imapmeta/29 metapartition- >> 30: /var/imapmeta/30 metapartition-100: /var/imapmeta/100 metapartition- >> temp: /var/imapmeta/temp >>> Going by the documentation, which I wrote with help from you good > folk at Fastmail, the Archive partition scheme might look something > like this:> > https://www.cyrusimap.org/imap/reference/admin/locations/archive-partitions.html> >> archivepartition-default: /var/mailarchives/default archivepartition- >> 1: /var/mailarchives/1 archivepartition-2: /var/mailarchives/2 >> archivepartition- >> 3: /var/mailarchives/3 archiveartition-4: /var/mailarchives/4 ... >> archivepartition- >> 29: /var/mailarchives/29 archivepartition-30: /var/mailarchives/30 >> archivepartition-100: /var/mailarchives/100 archivepartition-temp: >> /var/mailarchives/temp >>> And the Xapian partition structure to mate with this would look > something like this (again, from the docs):> > https://www.cyrusimap.org/imap/developer/install-xapian.html > >> defaultpartition: default partition-default: /var/mailstores/default >> search_engine: xapian search_index_headers: no search_batchsize: 8192 >> defaultsearchtier: t1 1searchtier: t1 2searchtier: t1 3searchtier: t1 >> 4searchtier: t1 ... 29searchtier: t1 30searchtier: t1 100searchtier: >> t1 tempsearchtier: t1 ... t1searchpartition-default: >> /var/search/default t1searchpartition-1: /var/search/1 t1searchpartition- >> 2: /var/search/2 t1searchpartition-3: /var/search/3 t1searchpartition- >> 4: /var/search/4 ... t1searchpartition-29: /var/search/29 t1searchpartition- >> 30: /var/search/30 t1searchpartition-100: /var/search/100 t1searchpartition- >> temp: /var/search/temp> First question, since there's no examples to work >> from; Is this Xapian > layout correct?> > Do I need to define & create Xapian partitions for the metadata > partitions, as is indirectly implied in Bron's original email on > this topic:> > https://lists.tartarus.org/pipermail/xapian-discuss/2014-October/009112.html> > Also, how do these Xapian and Archive, interact? Do I need to add a > separate Xapian partition for each Archive partition, or will the > Archive partition be treated like a child of the non-Archive > partition? (again, implied but not directly addressed in that > email).> > Any guidance gladly accepted, and whatever I learn will be repackaged > into more complete documentation on same.> > Cheers, > -nic > > -- Nic Bernstein n...@onlight.com > Onlight, Inc. www.onlight.com 6525 W > Bluemound Road, Suite 24 v. 414.272.4477 Milwaukee, > Wisconsin 53213-4073 >> Email had 1 attachment: > * nic.vcf > 1k (text/x-vcard) -- Bron Gondwana br...@fastmail.fm
Xapian partition definition when using archiving?
Bron, et al., We're about to set up a whole new bunch of partitions in support of Xapian indexing, for a 2.5.10-to-3.0.4 upgrade, and then will be introducing archive functionality, too. How do archive partitions and Xapian partitions interact? For example, the server currently has the following in imapd.conf: defaultpartition: default partition-default: /var/mailstores/default metapartition-default: /var/imapmeta/default partition-1: /var/mailstores/1 partition-2: /var/mailstores/2 partition-3: /var/mailstores/3 partition-4: /var/mailstores/4 ... partition-29: /var/mailstores/29 partition-30: /var/mailstores/30 partition-100: /var/mailstores/100 partition-temp: /var/mailstores/temp ... # non-default metapartitions metapartition-1: /var/imapmeta/1 metapartition-2: /var/imapmeta/2 metapartition-3: /var/imapmeta/3 metapartition-4: /var/imapmeta/4 ... metapartition-29: /var/imapmeta/29 metapartition-30: /var/imapmeta/30 metapartition-100: /var/imapmeta/100 metapartition-temp: /var/imapmeta/temp Going by the documentation, which I wrote with help from you good folk at Fastmail, the Archive partition scheme might look something like this: https://www.cyrusimap.org/imap/reference/admin/locations/archive-partitions.html archivepartition-default: /var/mailarchives/default archivepartition-1: /var/mailarchives/1 archivepartition-2: /var/mailarchives/2 archivepartition-3: /var/mailarchives/3 archiveartition-4: /var/mailarchives/4 ... archivepartition-29: /var/mailarchives/29 archivepartition-30: /var/mailarchives/30 archivepartition-100: /var/mailarchives/100 archivepartition-temp: /var/mailarchives/temp And the Xapian partition structure to mate with this would look something like this (again, from the docs): https://www.cyrusimap.org/imap/developer/install-xapian.html defaultpartition: default partition-default: /var/mailstores/default search_engine:xapian search_index_headers: no search_batchsize: 8192 defaultsearchtier: t1 1searchtier: t1 2searchtier: t1 3searchtier: t1 4searchtier: t1 ... 29searchtier: t1 30searchtier: t1 100searchtier: t1 tempsearchtier: t1 ... t1searchpartition-default: /var/search/default t1searchpartition-1: /var/search/1 t1searchpartition-2: /var/search/2 t1searchpartition-3: /var/search/3 t1searchpartition-4: /var/search/4 ... t1searchpartition-29: /var/search/29 t1searchpartition-30: /var/search/30 t1searchpartition-100: /var/search/100 t1searchpartition-temp: /var/search/temp First question, since there's no examples to work from; Is this Xapian layout correct? Do I need to define & create Xapian partitions for the metadata partitions, as is indirectly implied in Bron's original email on this topic: https://lists.tartarus.org/pipermail/xapian-discuss/2014-October/009112.html Also, how do these Xapian and Archive, interact? Do I need to add a separate Xapian partition for each Archive partition, or will the Archive partition be treated like a child of the non-Archive partition? (again, implied but not directly addressed in that email). Any guidance gladly accepted, and whatever I learn will be repackaged into more complete documentation on same. Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 6525 W Bluemound Road, Suite 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 <>
