[Akonadi] [Bug 389607] akonadictl fsck: Show more helpful messages

[Nick]

https://bugs.kde.org/show_bug.cgi?id=389607

--- Comment #8 from Nick  ---
Hello Martin, 
Thank you for your email and your comments .
I agree that some emails may be lost . I took the time to try making/finding
some correlations with what the user sees .
-
First of all I converted to IMAP as per ip provider's tech-support .
So the database contains records from the "old" [maildir? ] system and the new
"IMAP" system .  Also I had to move the database from my old machine to a new
one [ in fact entire /home  from old was transfered to the new machine ]
So the same userid had acceess to the emails immediately after opensuse leap
15.0 was installed on the new machine ].
I do not know if these changes have any impact [ the hostname was chaged ] 
-

Based on the way akonadictl presents its resulst it(akonadictl) is NOT a user
friendly tool . On the contrary the actual reporting scares the user with those
  messages "no RID" or "dirty" or "records removed from data base and moved  in
lost-and-found" without at least to ask the user if [s]he wants/agrees with
that move .

I looked closely to pimitemtable and it seems that the column remoteId is
nullable [ i.e. accepts an undefinned/unidentified/unknowm value ]. 

I believe that the hub of all these issues is not with the database server but
with akonadi because switching between MariaDB and Postgresql does not stop the
isses from happening. These problems in kmail-database are inherited from the
time when kmail was using sqlite .

In my humble opinion, the database server does what is told/comanded by the
interface between the user and the database---that is akonadi---.  

It is strange that a such important item as remoteId was translated in a
datamodel that accepts it as being nullable . 
That allows that any  event that is not controlled by the code inserts/updates
a record as having remoteId  NULL, and from this all users' complaints .

This assumption does not match the reality . I would like to know how it comes
that receiving an email---which has a sender and a receiver--- is recorded in
the database with a remoteId of NULL value ?

I provide below some info collected from my machine before c;eaning up the
records with RID NULl in pimitemtable .
You van notice that that record had a valid collectionId  but has invalid
remoteId .  I checked that many other emails with valid remoteId pointer to the
same collection Id .

I tend to agree with you that . 
``
However it also could be that Akonadi just messes up big time like in storing
the items into remote storage, but somehow failing to store the remote ID or
whatever. 
`` 

I believe that KDE development should have a look at kmail-database-datamodel
if it matches technical-requirements  and/or ifv akonadi handles the email
requests properly . 

Also it is very dificult for the user to find any item that is wrong using
kmail 
and not to dig in databse tables .  That's the role of akonadictl and/or what
ever any USER ORIENTED tools . I believe that akonadikconsole is too powerful
for the casual user .  At the same time akonadi seem to keep their cards
close to the vest . making easy for developers to point to user errors .

The users have had it for too long asking and asking and asking and nothing to
be done .  

Thank you,
Nick
= info from my machine =
--
describe pimitemtable
--

Field   TypeNullKey Default Extra
id  bigint(20)  NO  PRI NULLauto_increment
rev int(11) NO  0   
remoteIdvarbinary(255)  YES MUL NULL
remoteRevision  varbinary(255)  YES NULL
gid varbinary(255)  YES MUL NULL
collectionIdbigint(20)  YES MUL NULL
mimeTypeId  bigint(20)  YES MUL NULL
datetimetimestamp   NO  current_timestamp() 
atime   timestamp   NO  current_timestamp() 
dirty   tinyint(1)  YES NULL
sizebigint(20)  NO  0   
--
select count(*) from pimitemtable
--

count(*)
8691

Record without RID as reported by akonadictl fsck [ 38264 item ... RID ] 
38264   5   NULLNULLNULL123 4   2019-04-05 01:12:16
2019-04-05 01:12:16 1   81206

collectionId 123 is one of directories under "Local Folders 



--
describe pimitemflagrelation
--

Field   TypeNullKey Default Extra
PimItem_id  bigint(20)  NO  PRI NULL
Flag_id bigint(20)  NO  PRI NULL
--
select count(*) from pimitemflagrelation
--

count(*)
13859

Records associated with id 38264 [ from pimitemtable ]
38264   1
38264   5
38264   12
38264   13

--
describe parttable
--

Field   TypeNullKey Default Extra
id  bigint(20)  NO  PRI NULLauto_increment
pimItemId  

[Akonadi] [Bug 389607] akonadictl fsck: Show more helpful messages

[Martin Steigerwald]

https://bugs.kde.org/show_bug.cgi?id=389607

--- Comment #7 from Martin Steigerwald  ---
Just to confirm that this still happens with Akonadi 5.9.3 (KDEPIM/Akonadi
18.08).

Here are some examples of an akonadictl fsck run after switching to PostgreSQL
database backend just a few weeks ago:

Looking for resources in the DB not matching a configured resource...
Looking for collections not belonging to a valid resource...
Checking collection tree consistency...
Looking for items not belonging to a valid collection...
Looking for item parts not belonging to a valid item...
Looking for item flags not belonging to a valid item...
Looking for overlapping external parts...
Verifying external parts...
Found 8951 external files.
Found 2069 external parts.
Found unreferenced external file:
/home/martin/.local/share/akonadi/file_db_data/27/231727_r1
Found unreferenced external file:
/home/martin/.local/share/akonadi/file_db_data/37/85637_r2
Found unreferenced external file:
/home/martin/.local/share/akonadi/file_db_data/69/230869_r1
Found unreferenced external file:
/home/martin/.local/share/akonadi/file_db_data/44/216944_r1
[… >6000 more of those …]
Moved 6882 unreferenced files to lost+found.
Checking size treshold changes...
Found 0 parts to be moved to external files
Found 0 parts to be moved to database
Looking for dirty objects...
Collection "Search" (id: 1) has no RID.
Collection "OpenInvitations" (id: 360) has no RID.
Collection "DeclinedInvitations" (id: 361) has no RID.
Found 3 collections without RID.
Item "1144154" in collection "266" has no RID.
Item "1144155" in collection "256" has no RID.
Item "1144156" in collection "256" has no RID.
Item "1144157" in collection "256" has no RID.
Item "1144158" in collection "256" has no RID.
Item "1144159" in collection "256" has no RID.
Item "1144160" in collection "152" has no RID.
Item "1144161" in collection "180" has no RID.
Item "1144163" in collection "266" has no RID.
Item "1144164" in collection "256" has no RID.
Item "1144165" in collection "256" has no RID.
[… more of those …]
Found 43 items without RID.
Found 0 dirty items.
Looking for rid-duplicates not matching the content mime-type of the parent
collection
Checking Search
Checking Notizen
[…]


Related bugs:
- Bug 406958 - Unreferenced files in file_db_data
- Bug 406087 - akonadictl fsck incorrectly reports success when file_lost+found
folder is absent 
- Bug 406856 - Found 3734 items without RID.; Item "30024" has RID and is
dirty.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Akonadi] [Bug 389607] akonadictl fsck: Show more helpful messages

[Nick]

https://bugs.kde.org/show_bug.cgi?id=389607

--- Comment #6 from Nick  ---
Similar issues .
akonadi fsck / vacuum reports many issues/messages but did not give to the user
any hints on how to solve them .

What about having an option akonadictl fix [ noRID | dirtyRID ...etc ] ?
Such tool will reduce the number of bugs reported by akonadictl fsck 

Also, what about telling the users :
1) procedures of fixing issues 
2) procedures of getting rid on unnnecessary messages and/or
3) procedures of cleaning up the akonadi db of "dead" records
and above all
4) how the user should access akonadi mysql db which is handled [ start/stp] by
akonadi .

See bug 406856  recently opened 

Thanks .
Nick

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Akonadi] [Bug 389607] akonadictl fsck: Show more helpful messages

[Nick]

https://bugs.kde.org/show_bug.cgi?id=389607

Nick  changed:

   What|Removed |Added

 CC||ndor...@gmail.com

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Akonadi] [Bug 389607] akonadictl fsck: Show more helpful messages

[Nate Graham]

https://bugs.kde.org/show_bug.cgi?id=389607

Nate Graham  changed:

   What|Removed |Added

 CC||n...@kde.org

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Akonadi] [Bug 389607] akonadictl fsck: Show more helpful messages

[Martin Steigerwald]

https://bugs.kde.org/show_bug.cgi?id=389607

--- Comment #5 from Martin Steigerwald  ---
Dear Richard.

(In reply to Richard Bos from comment #4)
> I agree with this.

Thanks for confirming the report.

[…]
> Migrating parts to new cache hierarchy...
> 
> CRASH of akonadi!
> 
> This happens now everytime.  It worked before.
> 
> Now how to make akonadictl fsck work again???!

However please do not use this bug report to report a crash of akonadictl fsck.

Also please do not use this bug report for user support questions.

Why? Cause this was the reason for many bugs having so many comments that that
they are not useful for a developer anymore. Please help to keep this bug
report clean, simple and about *one* issue, please help to keep it actionable.
Thank you.

Please file a different bug report.

> BTW: creating a bug report (with stackback trace) does not work :(

Then please use kdepim-users or another appropriate support channel to get help
with that. Please refrain from further comments that are not related to this
bugs issue.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Akonadi] [Bug 389607] akonadictl fsck: Show more helpful messages

[Richard Bos]

https://bugs.kde.org/show_bug.cgi?id=389607

Richard Bos  changed:

   What|Removed |Added

 CC||richard@xs4all.nl

--- Comment #4 from Richard Bos  ---
I agree with this.

In my case I get:

# akonadictl --verbose fsck
Looking for resources in the DB not matching a configured resource...
Looking for collections not belonging to a valid resource...
Checking collection tree consistency...
Looking for items not belonging to a valid collection...
Looking for item parts not belonging to a valid item...
Looking for item flags not belonging to a valid item...
Looking for overlapping external parts...
Verifying external parts...
Found 165 external files.
Found 165 external parts.
Found no unreferenced external files.
Checking size treshold changes...
Found 0 parts to be moved to external files
Found 0 parts to be moved to database
Looking for dirty objects...
Collection "Search" (id: 1) has no RID.
Collection "OpenInvitations" (id: 1416) has no RID.
Collection "DeclinedInvitations" (id: 1417) has no RID.
Found 3 collections without RID.
Item "243448" has no RID.
Item "243531" has no RID.
Item "243813" has no RID.
Item "243868" has no RID.
Item "243869" has no RID.
Item "243986" has no RID.
Item "244023" has no RID.
Item "244026" has no RID.
Item "244111" has no RID.
Item "244176" has no RID.
Item "244379" has no RID.
Item "244393" has no RID.
Item "244399" has no RID.
Item "319051" has no RID.
Item "328336" has no RID.
Item "328379" has no RID.
Found 16 items without RID.
Item "232461" has RID and is dirty.
Item "239145" has RID and is dirty.
Item "243363" has RID and is dirty.
Item "328334" has RID and is dirty.
Found 4 dirty items.
Looking for rid-duplicates not matching the content mime-type of the parent
collection
Checking Notes
Checking Notities
Checking Search
Checking akonadi_davgroupware_resource_1
Checking akonadi_vcarddir_resource_10
Checking akonadi_vcarddir_resource_11
Checking akonadi_vcarddir_resource_12
Checking akonadi_vcarddir_resource_8
Checking akonadi_vcarddir_resource_9
Checking DeclinedInvitations
Checking OpenInvitations
Checking https:///remote.php/caldav/calendars//contact_birthdays/
Checking https:///remote.php/caldav/calendars//default calendar/
Checking https:///remote.php/carddav/addressbooks//default/

___checking of all folders___

Migrating parts to new cache hierarchy...

CRASH of akonadi!

This happens now everytime.  It worked before.

Now how to make akonadictl fsck work again???!

BTW: creating a bug report (with stackback trace) does not work :(

Would be nice / good to give this attention at Akademy

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Akonadi] [Bug 389607] akonadictl fsck: Show more helpful messages

[Richard Bos]

https://bugs.kde.org/show_bug.cgi?id=389607

Richard Bos  changed:

   What|Removed |Added

 Ever confirmed|0   |1
 Status|UNCONFIRMED |CONFIRMED

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Akonadi] [Bug 389607] akonadictl fsck: Show more helpful messages

[Martin Steigerwald]

https://bugs.kde.org/show_bug.cgi?id=389607

--- Comment #3 from Martin Steigerwald  ---
Thank you Dan, for your prompt responses. So I still misunderstood some of the
error messages despite your explainations. I think this hints and the
underlying issue: there is an akonadictl fsck, it often does not really fix the
issue the user sees, yet it reports some kind of probable inconsistencies and
leaves the user without anything sensible to do about… So I wonder about the
purpose of it? Aside from the migration work it does, is it a developer tool?
Whatever, no need to discuss it here. I reported what I think I understood from
your answer on the mailing list. If there are different actions to be taken
than the ones I suggest, feel free to close some of the reports I made. I just
wanted to help to move this forward.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Akonadi] [Bug 389607] akonadictl fsck: Show more helpful messages

[Daniel Vrátil]

https://bugs.kde.org/show_bug.cgi?id=389607

Daniel Vrátil  changed:

   What|Removed |Added

 CC||dvra...@kde.org

--- Comment #2 from Daniel Vrátil  ---
> Of course, even better would be if akonadictl fsck could fix the issues it 
> finds.

Yes, but not everything is an issue.

> Some suggestions to improve the situation:
> Item "1289323" has no RID. =>
>
> Item "1289323" only stored in database, not locally in XYZ resource.
>
> XYZ should be the name of the affected resource.

There are two problems:
1) The item can be owned by an online Resource (e.g. IMAP) and the Item may
have no RID simply because the Resource wasn't online yet. In such case the
situation above is not an error.

2) Your message is not precise. "locally in XYZ resource" is wrong, again in
case of online resource. I don't want fsck to contain misguiding messages. In
its own way, the current message (Item has no RID) is perfect, because
describes the situation perfectly. I think the best way would be to just add a
link to Wiki where the issue is described in detail.

> Item "1360766" has RID and is dirty. =>
> 
> Item "1360766" has changes that are only stored in the database, not locally 
> in XYZ resource

Again, the case can simply be because the change couldn't be replayed by an
online Resource yet because it's offline. Same solution as above would apply.

> => Exactly tell which files are identically to give the user the chance to 
> remove duplicate files. It is not obvious from the above:

The "duplicate" refers to duplicate RID, meaning there are multiple Items
stored in the database that belong to the same Collection and have the same
RID. That in no way implies there's a duplicate in the backing storage (maildir
or IMAP) and in most cases there's not even any actual file to refer to (e.g.
IMAP).

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Akonadi] [Bug 389607] akonadictl fsck: Show more helpful messages

[Martin Steigerwald]

https://bugs.kde.org/show_bug.cgi?id=389607

--- Comment #1 from Martin Steigerwald  ---
That is with Akonadi 17.08.3

-- 
You are receiving this mail because:
You are the assignee for the bug.