[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Albert Astals Cid via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #23 from Albert Astals Cid  ---
[LeaveOnServer]
seenUidList=00320035446ce738,00320036446ce738,00320037446ce738,00320038446ce738,00320039446ce738,0032003a446ce738
seenUidTimeList=1462884349,1462884349,1462884349,1462884349,1462884392,1462884437
I´d expect the Uids and Times to be *different* for each item, but they are the
same again. 


As i told you, the times don't matter at all, and the uids are different, so no
idea what you're complaining about.

This is your list of uids
00320035446ce738
00320036446ce738
00320037446ce738
00320038446ce738
00320039446ce738
0032003a446ce738

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Martin Steigerwald via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #22 from Martin Steigerwald  ---
Hmmm, no, the times are different. Oh, they have always been, just the same of
each retrieval attempt it seems. I completely missed that all the time cause
they looked all the same, as I usually have several hundred mails on each
download since I switched it to a 2 hour interval. Seems it gets the time once
for each "download mail" operation. But the uids are all the same.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Martin Steigerwald via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #21 from Martin Steigerwald  ---
This still happens with Applications/16.04. Since starting Akonadi + KMail
16.04 + setting to leave mail on server for one day it received six mails. This
is the result:

martin@merkaba:~/.config> cat akonadi_pop3_resource_0rc
[General]
host=[…]
intervalCheckEnabled=true
intervalCheckInterval=120
leaveOnServer=true
leaveOnServerDays=1
login=martin
pipelining=true
targetCollection=361
useTLS=true

[LeaveOnServer]
seenUidList=00320035446ce738,00320036446ce738,00320037446ce738,00320038446ce738,00320039446ce738,0032003a446ce738
seenUidTimeList=1462884349,1462884349,1462884349,1462884349,1462884392,1462884437

I´d expect the Uids and Times to be *different* for each item, but they are the
same again.

I checked it and for each mail it receives is just adds the same value to each
of the lists again, which doesn´t make any sense to me.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Martin Steigerwald via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #20 from Martin Steigerwald  ---
Thanks for the support, Michael. I never was clear on what to put into
branch-group to get the desired result. I somehow managed to have kdesrc-build
to build from an Applications branch.

Now rebuilding with globally with stable-kf-qt5 to have consistent versions for
everything. Will obviously take a while to complete even on Sandybridge i5
dualcore, 16 GiB RAM and Dual SSD BTRFS RAID 1 (or especially as the BTRFS
filesystem may not be running at optimal performance for being near full).

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Albert Astals Cid via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #18 from Albert Astals Cid  ---
Lol i'm stupid and can't write

Applications/16.04 

That's the right branch name

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Albert Astals Cid via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #17 from Albert Astals Cid  ---
Sorry the proper branch name is Applications/15.04 :D

Can you try if fixing that helps? I've no idea on how to use kdesrc-build so
can't really help there.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Michael Pyne via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

Michael Pyne  changed:

   What|Removed |Added

 CC||mp...@kde.org

--- Comment #19 from Michael Pyne  ---
Also, you can try "branch-group stable-kf5-qt5" in the global section if you
don't mind being off the bleeding edge a little bit for all those git modules.

Alternately, "branch-group stable-kf5-qt5" can be placed within the KDE
PIM-related module-sets (in place of the branch "Application/16.04" you would
otherwise add).

Just don't forget to run kdesrc-build with --refresh-build after you make that
change to avoid needless CMake confusion.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Martin Steigerwald via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #16 from Martin Steigerwald  ---
Hi Albert, from reading about branch and branch-group in kdesrc-build doc¹ I do
not get what to tell kdesrc build to build Applications/15.06 for me instead of
master.

In branch-group I can specify what modules to build, but not the branch (i.e.
version). In "branch"  I can specify what branch to build, but not the versions
to use. How do I turn

  1 global
  2 
  3 #qtdir /path/to/custom/qt  # Uncomment if you have your own qt
build
  4 source-dir  /home/kde/sources
  5 build-dir  /home/kde/build
  6 kdedir  /home/kde/install
  7 log-dir  /home/kde/logs
  8 
  9 git-repository-base kde-projects kde:   
 10 branch-groupkf5-qt5
 11 
 12 cmake-options -DCMAKE_BUILD_TYPE:STRING=debug
 13 
 14 cxxflags -pipe -DQT_STRICT_ITERATORS -DQURL_NO_CAST_FROM_STRING
-DQT_NO_HTTP -DQT_NO_FTP -Wformat -Werror=format-security -Werror=return-type
-Wno-variadic-macros -Wlogical-op -Wmissing-include-dirs
 15 
 16 # If you want to use ninja instead of make (it's faster!), add -GNinja
to cmake-options above
 17 # and uncomment the next line
 18 #custom-build-command ninja
 19 
 20 # Adjust to the number of  CPU cores
 21 make-options -j4
 22 
 23 ignore-kde-structuretrue# Downloads all modules directly
into the source folder instead of subdirectories
 24 #stop-on-failure true# Stop kdesrc-build when a build
fails.
 25 
 26 end global
 27 
 28 include /home/kde/sources/kdesrc-build/kf5-frameworks-build-include
 29 #include /home/kde/sources/kdesrc-build/kf5-workspace-build-include
 30 
 31 #Uncomment the next two lines to build application and PIM modules
 32 #include /home/kde/sources/kdesrc-build/kf5-applications-build-include
 33 include /home/kde/sources/kdesrc-build/kf5-kdepim-build-include

into building Applications/15.06 for me?

I did

diff --git a/kf5-kdepim-build-include b/kf5-kdepim-build-include
index 114d746..7242b02 100644
--- a/kf5-kdepim-build-include
+++ b/kf5-kdepim-build-include
@@ -27,6 +27,7 @@ module-set kde-pimlibs

 # The PIM Baloo code is now in akonadi-search
 use-modules kde/pim prison kdepimlibs akonadi-search libkgapi
+branch Applications/15.06
 end module-set

 module libkolab
@@ -37,5 +38,6 @@ end module
 module-set kde-pim
 repository kde-projects
 use-modules kde/kdepim kdepim-runtime
+branch Applications/15.06
 end module-set


but I get:

martin@merkaba:/home/kde/sources/kdesrc-build#1> ./kdesrc-build 
Updating kde-build-metadata (to branch master)
 * Downloading projects.kde.org project database...
 * Module kde/pim is apparently XML-based, but contains no
active modules to build!
Although no active modules are available, there were
72 inactive modules. Perhaps the git modules are not ready?
 * Module prison is apparently XML-based, but contains no
active modules to build!
Although no active modules are available, there were
1 inactive modules. Perhaps the git modules are not ready?
 * Module kdepimlibs is apparently XML-based, but contains no
active modules to build!
Although no active modules are available, there were
1 inactive modules. Perhaps the git modules are not ready?
 * Module akonadi-search is apparently XML-based, but contains no
active modules to build!
Although no active modules are available, there were
1 inactive modules. Perhaps the git modules are not ready?
 * Module libkgapi is apparently XML-based, but contains no
active modules to build!
Although no active modules are available, there were
1 inactive modules. Perhaps the git modules are not ready?
No modules were defined for the module-set kde-pimlibs
You should use the use-modules option to make the module-set useful.
 * Module kde/kdepim is apparently XML-based, but contains no
active modules to build!
Although no active modules are available, there were
1 inactive modules. Perhaps the git modules are not ready?
 * Module kdepim-runtime is apparently XML-based, but contains no
active modules to build!
Although no active modules are available, there were
1 inactive modules. Perhaps the git modules are not ready?

So I have no idea on how to accomplish the necessary build for now.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Albert Astals Cid via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #15 from Albert Astals Cid  ---
Could you try to use Applications/15.06 instead of master to see if it's a bug
introduced in master?

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Martin Steigerwald via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #14 from Martin Steigerwald  ---
Albert, seenUidTimeList also contains duplicated values, but if its the value
in which it saw the uid in question and it saw them all at the same time, it
may be correct. I don´t know how the code works. I would have expected to see
different time stamps there as well.

As I first saw this happening, I removed the config file, and had it redownload
everything – working manually to remove all duplicated mails. It again filled
the config file with duplicate values. Then I worked around this by disabling
leaving mails on server. As I had some duplicates then still for whatever
reason, I enabled leaving mails on server again and it again filled the config
file with duplicate values. Then I retried to disable leaving mails on server
and this works okayish since then.

So I am pretty confident with the git master branch I self-compiled KF5 +
KDEPIM on top of rest from Debian Sid/Experimental packages from, this is
repeatable. I currently use:

- bf3b97e3e00706f6f743f186766f6e54152783c1 of kdepim
- 3badd8e0c8c4acd80c8d15068e9bb68670366459 of kdepim-runtime
- 77fc3fff200f2647b01a91e996e080beb1d9125c of akonadi

I intent to update everything once I have a working copy of Qt 5.6 + dev files
on my system. I intend to wait for distro packages of 5.6.1, which the Qt/KDE
team of Debian hopefully prepares beginning of May as I do not want to take the
extra effort and time to work out how to self-compile Qt and configure to use
self-compiled Qt as well in addition to self-compiling KF5 + KDEPIM at the
moment.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Albert Astals Cid via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #13 from Albert Astals Cid  ---
Ok, i got confused by Laurent pointing to seenUidTimeList, so the problem is
duplicated seenUidList values, yes that's not good.

Is this something that keeps happening? Or is it something that broke at some
point and now works?

Because the code is pretty explicit in doing 

if (!seenUIDs.contains(uid))
seenUIDs.append(uid);

So it seems quite strange to me that this would break.

Also are you compiling things for yourself? Which branch?

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Albert Astals Cid via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

Albert Astals Cid  changed:

   What|Removed |Added

  Latest Commit||http://commits.kde.org/kdep
   ||im-runtime/a961e34924647fb1
   ||aa06a6abd0d8fb5147b6f624
 Status|CONFIRMED   |RESOLVED
 Resolution|--- |FIXED

--- Comment #11 from Albert Astals Cid  ---
Git commit a961e34924647fb1aa06a6abd0d8fb5147b6f624 by Albert Astals Cid.
Committed on 28/04/2016 at 21:01.
Pushed by aacid into branch 'Applications/16.04'.

Cache idsToSave

Seems doing all the work each time shouldDeleteId is called is not as fast as i
thught, so cache it until the next list job finishes (i.e. we're checking email
again).

Acked by Laurent

REVIEW: 127775

M  +52   -46   resources/pop3/pop3resource.cpp
M  +4-0resources/pop3/pop3resource.h

http://commits.kde.org/kdepim-runtime/a961e34924647fb1aa06a6abd0d8fb5147b6f624

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Albert Astals Cid via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #10 from Albert Astals Cid  ---
> Albert, the duplicates easily get excessive. Here an example of it with a 
> retention time of just 2 days.

It is not duplication, it's the time each of the ids of the messages was
seen/saved, it just happens that they were seen/saved, that is not a bug and is
always how it has worked, unless you know how the code works and you can prove
this is a bug i don't see why it would be a bug.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Martin Steigerwald via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #8 from Martin Steigerwald  ---
Created attachment 98669
  --> https://bugs.kde.org/attachment.cgi?id=98669=edit
excessive duplicates in akonadi_pop3_resource_0rc

Albert, the duplicates easily get excessive. Here an example of it with a
retention time of just 2 days.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Laurent Montel via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #7 from Laurent Montel  ---
It helps for me.
Perhaps that Martin Steigerwald can test it too.

I test all the day.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Albert Astals Cid via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #6 from Albert Astals Cid  ---
Laurent, can you check if https://git.reviewboard.kde.org/r/127775/ helps?

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Laurent Montel via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #5 from Laurent Montel  ---
It's faster when I revert it.
I store 30 days on server for kde.org mail... so it's not just 55 items for me
and it's very slow with master pop3 resource.
And pop3 doesn't take now 100% of cpu.

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Albert Astals Cid via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

--- Comment #4 from Albert Astals Cid  ---
A duplicate seenUidTimeList is not a problem is it?

My rework to the pop3resource made it a bit slower when leaving messages on
server, but i was not expecting "for minutes"

I mean you have only around 55 items in seenUidTimeList it should not be that
horribly slow.

Can you check if reverting
https://github.com/KDE/kdepim-runtime/commit/e7ba619bef10fed4c1e623c61822da505b270d2a
makes it faster again?

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Laurent Montel via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

Laurent Montel  changed:

   What|Removed |Added

 Ever confirmed|0   |1
 Resolution|WAITINGFORINFO  |---
 Status|NEEDSINFO   |CONFIRMED

--- Comment #3 from Laurent Montel  ---
Same for me it's duplicate:
seenUidTimeList=1456260920,1456260920,1456260920,1456260920,1456260920,1456260920,1456260920,1456260920,1456260920,1456260920,1456260920,1456260920,1456260920,1456260920,1456260920,1456260920,1456260920,1456260920,1456260920,1456263334,1456263334,1456263334,1456263334,1456263334,1456263334,1456263334,1456263334,1456263334,1456265040,1456265040,1456265040,1456265040,1456265040,1456265040,1456265040,1456265040,1456265040,1456265040,1456265040,1456265040,1456265040,1456265040,1456265040,1456265040,1456265040,1456265040,1456292814,1456292814,1456292814,1456292814,1456292814,1456292814,1456292814,1456292814,1456292814

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Laurent Montel via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

Laurent Montel  changed:

   What|Removed |Added

 CC||mon...@kde.org

--- Comment #2 from Laurent Montel  ---
I confirm that pop3 is very slow this last week.
And it took 100% of cpu each time.

When I look at paste I see that it's duplicate:
see:
seenUidTimeList=1460582993,1460582993,1460582993,1460582993,1460582993,1460582993,14605829
:)

-- 
You are receiving this mail because:
You are watching all bug changes.

[Akonadi] [Bug 361737] duplicates values in SeenUidList and SeenUidTimeList until it crawls to an halt on saving messages

[Albert Astals Cid via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=361737

Albert Astals Cid  changed:

   What|Removed |Added

 Resolution|--- |WAITINGFORINFO
 Status|UNCONFIRMED |NEEDSINFO

--- Comment #1 from Albert Astals Cid  ---
Where is the duplicate? I can't see any in the example you pasted

-- 
You are receiving this mail because:
You are watching all bug changes.