[Akonadi] [Bug 364114] moving a folder within one maildir resource is extremely slow and inefficient

[Martin Steigerwald]

https://bugs.kde.org/show_bug.cgi?id=364114

--- Comment #9 from Martin Steigerwald  ---
Bug 389638 - When moving imported maildir folders, before they are completely
written to disk, they arrive empty, losing all the mails.

may be related.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 364114] moving a folder within one maildir resource is extremely slow and inefficient

[Martin Steigerwald]

https://bugs.kde.org/show_bug.cgi?id=364114

Martin Steigerwald  changed:

   What|Removed |Added

 Ever confirmed|0   |1
 Status|UNCONFIRMED |CONFIRMED

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 364114] moving a folder within one maildir resource is extremely slow and inefficient

[Martin Steigerwald]

https://bugs.kde.org/show_bug.cgi?id=364114

--- Comment #8 from Martin Steigerwald  ---
During akonadictl fsck:

Moved 15029 unreferenced files to lost+found.

Also it only found about 21000 files in there at all (output scrolled out of
buffer, but I bet it was 15029+6371).

On second run it now reports "Found 6371 external files." and this now seems to
be correct:

martin@merkaba:~/.local/share/akonadi> find file_db_data -type f | wc -l
6371

Yet it only moved 15029 files to lost+found:

martin@merkaba:~/.local/share/akonadi> find file_lost+found -type f | wc -l 
15029

I seriously have no idea what it did with the other:

martin@merkaba:~/.local/share/akonadi> find file_db_data -type f | wc -l
46846
(see previous comment #7)

46846 - 15029 = 31817 files

However as I think they have been related to that aborted move operation I
don´t mind them to be gone. Still that output is not really trustworthy.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 364114] moving a folder within one maildir resource is extremely slow and inefficient

[Martin Steigerwald]

https://bugs.kde.org/show_bug.cgi?id=364114

--- Comment #7 from Martin Steigerwald  ---
Okay, I now stopped Akonadi after more than one hour of it trying to move that
folder around. So far it didn´t move a single mail file. They are all still
here:

martin@merkaba:~/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory>
find btrfs-ml -type f | wc -l 
63720

martin@merkaba:~/.local/share/local-mail/.Lichtvoll.directory/.Linux.directory>
du -sh btrfs-ml
521Mbtrfs-ml

Additionally it didn´t even create the target directory.

This is *just* moving a folder!

So my guess is again that it was still occupied with moving all those mails
into the cache, i.e. into mysqld database or if threshold exceeded lost and
found.

Well sure enough it was:

martin@merkaba:~/.local/share/akonadi> find file_db_data -type f | wc -l
46846
martin@merkaba:~/.local/share/akonadi> du -sh file_db_data
462Mfile_db_data

I bet some are also in there.

martin@merkaba:~/.local/share/akonadi> du -sh db_data 
4,2Gdb_data

Luckily after the restart Akonadi forgot that it was to move that large folder.
So… I will just fsck and then vacuum the whole thing back to some sanity again.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 364114] moving a folder within one maildir resource is extremely slow and inefficient

[Martin Steigerwald]

https://bugs.kde.org/show_bug.cgi?id=364114

--- Comment #6 from Martin Steigerwald  ---
I just confirmed this again with KMail 5.2.3, KDEPIM and Akonadi 16.04.3. Its
still moving that 5+ mail btrfs mailinglist folder around and its working
on that since way more than half an hour. I hope it will eventually complete
the job during the day. And then will move it back using my workaround
mentioned in comment #1. A move is completed much faster when using that
workaround, although Akonadi still does way to much needless work there.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 364114] moving a folder within one maildir resource is extremely slow and inefficient

[Martin Steigerwald via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=364114

--- Comment #5 from Martin Steigerwald  ---
https://phabricator.kde.org/T630

is somewhat related.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 364114] moving a folder within one maildir resource is extremely slow and inefficient

[Simon Andric via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=364114

Simon Andric  changed:

   What|Removed |Added

 CC||simonandr...@gmail.com

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 364114] moving a folder within one maildir resource is extremely slow and inefficient

[Martin Steigerwald via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=364114

--- Comment #4 from Martin Steigerwald  ---
I see how it is challenging to solve this within current Akonadi design. If
using a folder ID that is just stored within the database, it won´t detect
manual moves. But well, it could write a .folderinfo file into each folder
containing a unique hash of the folder. It would then store in database folder
xyz has this internal Akonadi ID and this hash and is currently located at this
path.

I wonder whether this changes the consequence of a database loss, but I don´t
think so, I think it can even provide a path to make filter handling much more
robust. If the filter rules stores the hash of the folder, akonadi mailfilter
agent can ask Akonadi about the internal database ID for the folder and thus as
long as the user does not remove the .folderinfo or whatever it is called file
from the folder, the filter rules would still work after a complete database
loss. (Of course that doesn´t solve this issue for IMAP accounts, but it would
be a start.)

For the individual mail items in the database Akonadi can still use an internal
ID, cause on database loss, this information is lost as well, so it will have
to reindex all the folders anyway.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 364114] moving a folder within one maildir resource is extremely slow and inefficient

[Martin Steigerwald via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=364114

--- Comment #3 from Martin Steigerwald  ---
Okay, it took about 10 minutes and several GiB of disk traffic, but it seems it
is now done. Folders at their new location do not yet display amount of unread
mails and I expect Akonadi to create even more I/O and CPU usage when I click
on a folder there, so for now I just won´t and call it a day.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 364114] moving a folder within one maildir resource is extremely slow and inefficient

[Martin Steigerwald via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=364114

--- Comment #2 from Martin Steigerwald  ---
And yeah, I bet I know what its doing: Its removing all the stale mails entries
from the mysqld database and then adding them again at the new location. Also
KMail still showed the folders at their old location for minutes. Now it
doesn´t display them there anymore, but in the new location they have no unread
mail count, maybe it indexes those folders then.

Honestly I´d expect Akonadi to reference the folder a mail is in by some *id*.
And if I ask to move it elsewhere, it notes that it is elsewhere now in some
mail folder table and *is* done with it.

Or in whatever other way: As a user I expect a folder move operation to be
*instant* or *almost* instant. And cause *next* to no disk I/O at all.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 364114] moving a folder within one maildir resource is extremely slow and inefficient

[Martin Steigerwald via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=364114

--- Comment #1 from Martin Steigerwald  ---
Today I tried to be clever:

- I just stopped Akonadi and waited till it was gone (including mysqld
process).
- Then I manually moved several large folders with LKML mails, one with 26+
mails within the maildir to another directory within the very same local
folders resource, a directory/folder I use for archival purpose

But again, KMail is unresponsive when I ask it to display the contents of a
mail. It can display the list of mails that a folder contains, but displaying a
mail content does not work. While it is busy in the background with about
50-120% cpu usage for the mysqld process and about 120-140 MiB – in words more
than hundred MiB – of writes to the dual SSD BTRFS RAID 1 every 10 seconds.

So for a simple action of moving a folder it creates write I/Os in excess of
several GiB *easily*. Sorry, but this is just broke in my eyes.

-- 
You are receiving this mail because:
You are watching all bug changes.