On Wednesday, 11 September 2024 16:08:05 CEST Jean Charles Delépine via Info
wrote:
> Hello,
>
> Quoting [email protected]:
> > An obvious addition.
> > Yes, index size decreases to expected 50% of source if I remove old
> > index (rm *) and create completely new instead. Is it the indended
> > maintenance method?
>
> You can read this mail from Bron Gondwana at Fastmail on how they use
> the Xapian index, it's the better doc I find and it helps me
> configuring it.
>
> https://lists.tartarus.org/pipermail/xapian-discuss/2014-October/009112.html
>
> Sincerly,
> Jean Charles Delépine
>
As addition to that mail, which I used as a basis as well, here is how I
configured mine. Hope this helps others.
- settings in imapd.conf:
tempsearchpartition-main: /var/spool/index/temp
activesearchpartition-main: /var/spool/index/active
archivesearchpartition-main: /var/spool/index/archive
- In cyrus.conf (added the 'reindex' section):
START {
# do not delete this entry!
recover cmd="ctl_cyrusdb -r"
statscleanup cmd="promstatsd -c"
reindex cmd="squatter -v -i"
}
- The filesystems used (from fstab):
tmpfs /var/spool/index/temp tmpfs
defaults,uid=cyrus,gid=mail,mode=0755,noatime,size=12G
/dev/xvdd1 /var/spool/index/active ext4 defaults,noatime
/dev/xvdd2 /var/spool/index/archive ext4 defaults,noatime
- My regularly running script to handle the indexes:
see file "xapian_index_management.txt"
My script can be adjusted to limit sizes and usage-levels
--
Joost
------------------------------------------
Cyrus: Info
Permalink:
https://cyrus.topicbox.com/groups/info/Tdcc91f096c0df53c-M40317592fe04e62b73b4e5bb
Delivery options: https://cyrus.topicbox.com/groups/info/subscription
#!/bin/bash
#
maxtempperc=10
maxactiveperc=30
# 100MB = 100*1024*1024 = 104857600
maxtemp=104857600
# 500MB = 500 * 1024 * 1024 = 524288000
maxactive=524288000
tempperc=`/bin/df /var/spool/index/temp | /bin/grep temp | /usr/bin/awk '{
USED=($3*1000); SIZE=($2*1000); USAGE=(USED/SIZE)*100; print USAGE}'`
activeperc=`/bin/df /var/spool/index/active | /bin/grep active | /usr/bin/awk
'{ USED=($3*1000); SIZE=($2*1000); USAGE=(USED/SIZE)*100; print USAGE}'`
/bin/echo "tempperc is [ ${tempperc} ]"
/bin/echo "activeperc is [ ${activeperc} ]"
if /usr/bin/awk -v n1="${tempperc}" -v n2="${maxtempperc}" 'BEGIN { exit ( n1 >
n2 ) }' /dev/null; then
/bin/echo "Temp usage [ ${tempperc} ] is smaller than max temp usage [
${maxtempperc} ]"
else
/bin/echo "Temp usage [ ${tempperc} ] is larger than max temp usage [
${maxtempperc} ]"
/bin/echo "REDUCING maxtemp"
maxtemp=1048576
fi
if /usr/bin/awk -v n1="${activeperc}" -v n2="${maxactiveperc}" 'BEGIN { exit (
n1 > n2 ) }' /dev/null; then
/bin/echo "Active usage [ ${activeperc} ] is smaller than max active usage [
${maxactiveperc} ]"
else
/bin/echo "Active usage [ ${activeperc} ] is larger than max active usage [
${maxactiveperc} ]"
/bin/echo "REDUCING maxarchive"
maxarchive=1048576
fi
/bin/echo ""
/bin/echo "Checking individual temp usage"
if /bin/ls /var/spool/index/temp/*/user/*/xapian* 1> /dev/null 2>&1; then
/usr/bin/du -bs /var/spool/index/temp/*/user/*/xapian* |
while read size name
do
username=`/bin/echo ${name} | /bin/sed
's/\(.*\)\/user\/\(.*\)\/xapia\(.*\)/\2/'`
if [ "${size}" -le "${maxtemp}" ]; then
/bin/echo "username [ ${username} ] with size [ ${size} ] is smaller
than [ ${maxtemp} ]"
else
/bin/echo "username [ ${username} ] with size [ ${size} ] is greater
than [ ${maxtemp} ]"
/bin/echo "EXEC : squatter -v -o -z active -t temp -u ${username}"
/usr/sbin/squatter -v -o -z active -t temp -u "${username}"
fi
done
else
/bin/echo "No active users found in temp"
fi
echo ""
echo "Checking individual active usage"
if /bin/ls /var/spool/index/active/*/user/*/xapian* 1> /dev/null 2>&1; then
find /var/spool/index/active -type d -name "xapian*" 2>/dev/null | /bin/sed
's/\(.*\)\/xapian\(.*\)/\1/' | sort -u |
while read longpath
do
path=`/bin/echo ${longpath} | /bin/sed 's/\(.*\)\/xapian\(.*\)/\1/' |
sort -u`
echo ${path}
/usr/bin/du -bs ${path} |
while read size name
do
username=`/bin/echo ${name} | /bin/sed 's/\(.*\)\/user\/\(.*\)/\2/'`
if [ "${size}" -le "${maxactive}" ]; then
/bin/echo "username [ ${username} ] with size [ ${size} ] is
smaller than [ ${maxactive} ]"
else
/bin/echo "username [ ${username} ] with size [ ${size} ] is
greater than [ ${maxactive} ]"
/bin/echo "EXEC : squatter -v -z archive -t active,archive -u
${username}"
/usr/sbin/squatter -v -z archive -t active,archive -u "${username}"
fi
done
done
else
/bin/echo "No active users found in active"
fi
