[kmail2] [Bug 417206] KMail wrong date in IMAP APPEND command
https://bugs.kde.org/show_bug.cgi?id=417206 --- Comment #10 from Erik Quaeghebeur --- (In reply to kernel_panic from comment #8) > For example, GMail uses the timestamp provided in the APPEND command to > store, index, and display messages in its web interface. This timestamp may > be different to the one provided in the message header. However, it appears > that this timestamp is stored separately (presumably in some indexing > database) since the original message's Date header is left intact and > showing the correct timestamp. […] I think there is a misunderstanding. The INTERNAl DATE is *not* the same as the Date header value and it should not be according to the RFC. (Usually, it will be close, but can differ significantly[*] if message delivery is delayed.) Please read https://tools.ietf.org/html/rfc3501#section-2.3.3 for a description. GMail does follow the RFC here AFAICT and uses the ‘REFS’ threading option. (As does, e.g., Fastmail.) [*] So that order of mails in a thread differs based on INTERNAL DATE or Date header. -- You are receiving this mail because: You are the assignee for the bug.
[kmail2] [Bug 417206] KMail wrong date in IMAP APPEND command
https://bugs.kde.org/show_bug.cgi?id=417206 --- Comment #9 from kernel_panic --- (In reply to Erik Quaeghebeur from comment #7) > Non-similar as setInternalDate is *not* called here (for when ‘changing an > item’): > > https://invent.kde.org/pim/kdepim-runtime/-/blob/master/resources/imap/ > changeitemtask.cpp#L53 > > In this last case, the INTERNAL DATE is probably set by the server to the > current date. It may be that this is where my issue originates. I think you're right in this case. I'm hesitant to overgeneralise, but it appears it's a selection of individual "Task" implementations that do not include the internal timestamp and the server will use the current one when storing the message. Which makes sense, as the KIMAP clearly does have support for supplying the correct timestamp. (In reply to Erik Quaeghebeur from comment #6) > In case you agree, we should modify the product/component of this bug. > Agreed. I think it's the underlying PIM runtime task implementations which KMmail ultimately uses that may need to be corrected. We should probably update this to "kdepim" or "kdepimlibs". I suspect the latter would be more appropriate, what do you think? -- You are receiving this mail because: You are the assignee for the bug.
[kmail2] [Bug 417206] KMail wrong date in IMAP APPEND command
https://bugs.kde.org/show_bug.cgi?id=417206 --- Comment #8 from kernel_panic --- (In reply to Erik Quaeghebeur from comment #5) > TBH, I cannot reproduce your issue. When I move messages between IMAP > accounts on different servers, they keep their INTERNAL DATE. I've tried > with related(plain,image) and > alternative(plain,related(html,image,image,image)) messages. I think whether the issue is de-facto observed is also dependent on the implementation of the server and its RFC compliance. For example, GMail uses the timestamp provided in the APPEND command to store, index, and display messages in its web interface. This timestamp may be different to the one provided in the message header. However, it appears that this timestamp is stored separately (presumably in some indexing database) since the original message's Date header is left intact and showing the correct timestamp. Consequently, the messages with their correct timestamps in a mail client (e.g. KMail or Thunderbird) which use the Date headers for indexing, but are completely messed up in GMail's web interface. On my private mail server, which uses Postfix and a simple Mbox format for storage and indexing, the message's internal Date header is always used. I don't recall explicitly configuring this but it appears that it is interpreting the "SHOULD" clauses in the RFC slightly differently. Going back to the RFC section quoted above, I think I may have misinterpreted what it actually means. Notably, it only concerns mail /servers/ and not mail clients. It seems that if a timestamp has been specified in the APPEND command the server SHOULD use that timestamp and it SHOULD supersede any internal message timestamp (e.g. Date header). Moreover, it SHOULD overwrite said internal timestamp. Likewise, if a timestamp is not provided, the server SHOULD use the current date/time and set the message's internal timestamp to that one. So, in that regard, it would appear GMail's implementation is at least partially conformant wrt its web interface and how it indexes the messages there. But I don't see any section of the RFC that concerns clients issuing the APPEND command. In either case, my view is that it would make sense for clients, by default, to use the message's internal timestamp - if present - in all instances so as to avoid the above issue. From my mini-analysis, Thunderbird does that. And KMail also does this in most cases. So it seems it's only the small set of cases where this isn't done that are of concern. I'll see if I can narrow down when else this might occur. So far, the easiest way to consistently reproduce the issue is with filters (e.g. PGP encrypt and/or decrypt) which pushes an encrypted copy of the message to the IMAP server before deleting the old copy from the IMAP server. -- You are receiving this mail because: You are the assignee for the bug.
[kmail2] [Bug 417206] KMail wrong date in IMAP APPEND command
https://bugs.kde.org/show_bug.cgi?id=417206 --- Comment #7 from Erik Quaeghebeur --- (In reply to Erik Quaeghebeur from comment #6) > I think the line where the INTERNAL DATE is set is […] I shouldn't be so quick to point to this instance only. Similar (for when ‘replacing’ a message): https://invent.kde.org/pim/kdepim-runtime/-/blob/master/resources/imap/replacemessagejob.cpp#L36 Non-similar as setInternalDate is *not* called here (for when ‘changing an item’): https://invent.kde.org/pim/kdepim-runtime/-/blob/master/resources/imap/changeitemtask.cpp#L53 In this last case, the INTERNAL DATE is probably set by the server to the current date. It may be that this is where my issue originates. -- You are receiving this mail because: You are the assignee for the bug.
[kmail2] [Bug 417206] KMail wrong date in IMAP APPEND command
https://bugs.kde.org/show_bug.cgi?id=417206 --- Comment #6 from Erik Quaeghebeur --- I think the line where the INTERNAL DATE is set is https://invent.kde.org/pim/kdepim-runtime/-/blob/master/resources/imap/additemtask.cpp#L56 In case you agree, we should modify the product/component of this bug. So next, to find out what determines msg->date()->dateTime(). My guess is it is Akonadi's PimItemTable.datetime, which is the last modified date, which is (in general) not the internal date of the message on the (originating) server. That would be against the RFC and therefore a clear bug. -- You are receiving this mail because: You are the assignee for the bug.
[kmail2] [Bug 417206] KMail wrong date in IMAP APPEND command
https://bugs.kde.org/show_bug.cgi?id=417206 Erik Quaeghebeur changed: What|Removed |Added Version|5.12.3 |5.14.3 -- You are receiving this mail because: You are the assignee for the bug.
[kmail2] [Bug 417206] KMail wrong date in IMAP APPEND command
https://bugs.kde.org/show_bug.cgi?id=417206 --- Comment #5 from Erik Quaeghebeur --- (In reply to kernel_panic from comment #4) > […], this doesn't explain why single part plaintext emails are not affected > - at least not in the scenarios I provided. […] TBH, I cannot reproduce your issue. When I move messages between IMAP accounts on different servers, they keep their INTERNAL DATE. I've tried with related(plain,image) and alternative(plain,related(html,image,image,image)) messages. So I'm afraid we may be hitting different bugs both related to INTERNALDATE handling. The fact that copying between different servers works (in some cases) actually makes me hopeful, as that means that it is possible without having to add a database column to Akonadi. -- You are receiving this mail because: You are the assignee for the bug.
[kmail2] [Bug 417206] KMail wrong date in IMAP APPEND command
https://bugs.kde.org/show_bug.cgi?id=417206 --- Comment #4 from kernel_panic --- I remember I was looking at the same "appendjob.cpp" file when I first reported the issue. At the time I was hoping to be able to pinpoint the issue and even make a small patch that fixes it. Unfortunately, I'm not at all familiar with KDE's codebase and there seem to be a number of different components at play. But I think you're absolutely right. From what I understand in the way that KDE works, KMail connects to Akonadi which actually does the handling of all PIM-related tasks. So even though the issue manifests itself in KMail, it may in fact be related to the way that Akonadi handles IMAP resources in which case this report ought to be filed under Akonadi. In either case, if Akonadi has no sense of what the internal date is, this doesn't explain why single part plaintext emails are not affected - at least not in the scenarios I provided. Unless, as you suggest, it's fetching the date from IMAP on-the-fly when the operation is performed. I will have another look at the source code when I get a chance. -- You are receiving this mail because: You are the assignee for the bug.
[kmail2] [Bug 417206] KMail wrong date in IMAP APPEND command
https://bugs.kde.org/show_bug.cgi?id=417206 --- Comment #3 from Erik Quaeghebeur --- (In reply to kernel_panic from comment #2) > […] I was beginning to lose hope that this > would ever be picked up […] I think that if we want this to be picked up, it would help if we can at least point out in the code where the issue lies. This is filed in the product kmail2, but it could be an akonadi or kimap issue. I think kimap can be excluded, because it clearly provides functionality for setting the INTERNAL DATE: https://invent.kde.org/pim/kimap/-/blob/release/20.04/src/appendjob.cpp (This was also apparent from your initial report.) AFAICT from the Akonadi database structure, it only has a concept of * last access time * datetime, which seems to correspond to last *modification* time (I do not know which modifications count: flags, tags, …?). For sure changing the message's MIME structure counts, for the issue I encounter. So if I'm correct it has no concept that maps to INTERNAL DATE. That means that upon append to an IMAP store from an IMAP store, the code that appends must explicitly include a call to the original IMAP store to get the INTERNAL DATE and use it upon append. Further investigation is necessary. -- You are receiving this mail because: You are the assignee for the bug.
[kmail2] [Bug 417206] KMail wrong date in IMAP APPEND command
https://bugs.kde.org/show_bug.cgi?id=417206 --- Comment #2 from kernel_panic --- @Erik, Thanks for the update to the report. I was beginning to lose hope that this would ever be picked up and am glad somebody else has also noticed this behaviour. A quick update, as of KMail v5.14.3 (plasma v5.18.5, apps v20.04.3, frameworks v5.70.0) the issue is still present. -- You are receiving this mail because: You are the assignee for the bug.
[kmail2] [Bug 417206] KMail wrong date in IMAP APPEND command
https://bugs.kde.org/show_bug.cgi?id=417206 Erik Quaeghebeur changed: What|Removed |Added CC||k...@equaeghe.nospammail.net Ever confirmed|0 |1 Severity|minor |normal Status|REPORTED|CONFIRMED --- Comment #1 from Erik Quaeghebeur --- I can confirm this. I hit this issue when filtering using the "Pipe through" action. Upon APPEND, the INTERNAL DATE of the filtered message should be set to the one of the original one. -- You are receiving this mail because: You are the assignee for the bug.
[kmail2] [Bug 417206] KMail wrong date in IMAP APPEND command
https://bugs.kde.org/show_bug.cgi?id=417206 kernel_panic changed: What|Removed |Added CC|vic.m.iva...@gmail.com | -- You are receiving this mail because: You are the assignee for the bug.
[kmail2] [Bug 417206] KMail wrong date in IMAP APPEND command
https://bugs.kde.org/show_bug.cgi?id=417206 kernel_panic changed: What|Removed |Added CC||vic.m.iva...@gmail.com -- You are receiving this mail because: You are the assignee for the bug.
