Re: KDirModelV2, KDirListerV2 and UDSEntryV2 suggestions
Hi, 2013/1/9 Mark: On Wed, Jan 9, 2013 at 2:51 PM, David Faure wrote: On Wednesday 09 January 2013 11:15:20 Mark wrote: A little more in depth questions for KDirLister and KFileItem. In my profiling KFileItem ends up high due to various reasons, but KDirLister is also a bit of a heavy resource hog due to it's default behavior of fetching all file information (thus at least 1 stat call per file) which severely slows down the dir listing process for large folders. This stat call happens in kio_file though, not in the GUI process where KDirLister lives, right? So I'm surprised that you see that when profiling... or is there a nasty stat() in KDirLister somewhere? ehm, well i'm not really monitoring stat calls. I'm monitoring the time it takes for a directory listing on 1 million files to be available in my application. I agree that we should try to behave well even if there are many files. However, if it is ensured if all algorithms involved in loading item information and view setup are O(N) or at most O(N*log(N)) (like sorting), then the remaining slowness for many files is something that the user might just have to live with IMHO. I'm not saying that everything is good as it is. If there are some operations with quadratic or worse complexity involved, then this should be looked into, of course. And I also do realise that Dolphin could benefit from making use of your listJob batching improvements (https://git.reviewboard.kde.org/r/107520/, very nice work BTW!). But making extremely intrusive changes to well-tested code which are only motivated by the (IMHO rather exotic) 1 million files use case and which would require large changes everywhere in KDE might not be such a good idea. Just my opnion, of course. Best regards, Frank P.S.: I agree that the lazy folder icon loading issue mentioned elsewhere in this thread can be disturbing sometimes and could be improved. ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
Patch: Remove kDebug includes from kio/kio
Hi David, I have removed all the kdebug.h includes from kio/kio. Please review it. Thanks, Jignesh remove_kdebug_includes.patch Description: Binary data ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
Re: KDirModelV2, KDirListerV2 and UDSEntryV2 suggestions
On Thu, Jan 10, 2013 at 9:23 AM, Frank Reininghaus frank7...@googlemail.com wrote: Hi, 2013/1/9 Mark: On Wed, Jan 9, 2013 at 2:51 PM, David Faure wrote: On Wednesday 09 January 2013 11:15:20 Mark wrote: A little more in depth questions for KDirLister and KFileItem. In my profiling KFileItem ends up high due to various reasons, but KDirLister is also a bit of a heavy resource hog due to it's default behavior of fetching all file information (thus at least 1 stat call per file) which severely slows down the dir listing process for large folders. This stat call happens in kio_file though, not in the GUI process where KDirLister lives, right? So I'm surprised that you see that when profiling... or is there a nasty stat() in KDirLister somewhere? ehm, well i'm not really monitoring stat calls. I'm monitoring the time it takes for a directory listing on 1 million files to be available in my application. I agree that we should try to behave well even if there are many files. However, if it is ensured if all algorithms involved in loading item information and view setup are O(N) or at most O(N*log(N)) (like sorting), then the remaining slowness for many files is something that the user might just have to live with IMHO. I'm not saying that everything is good as it is. If there are some operations with quadratic or worse complexity involved, then this should be looked into, of course. And I also do realise that Dolphin could benefit from making use of your listJob batching improvements (https://git.reviewboard.kde.org/r/107520/, very nice work BTW!). Thank you for the compliment :) That could be a nice improvement in dolphin indeed. However, while testing that in my application (with 1 million files) i found an issue that you are very likely to have as well. While the files do show up very rapidly, the actual view is still barely usable as long as files are dripping in. This is the same in QML (ListView) as in QWidgets (QTreeView). The issue here is that the view - even though it's not showing the items - is doing all the calculations for every item added. I have no clue how to fix that issue and if it's fixed it likely has to be done upstream in Qt itself. For Dolphin that might be different since it has it's own fully implemented view so you probably have to look at it there. But making extremely intrusive changes to well-tested code which are only motivated by the (IMHO rather exotic) 1 million files use case and which would require large changes everywhere in KDE might not be such a good idea. Just my opnion, of course. Well, in my test cases my initial goal of 100.000 - 200.000 files is reached :) So then i upped my demands to 1 million. But you already notice slowdowns in dolphin with 10.000+ files, it kinda depends on your hardware as well. I know that i'm testing extremes right now :) Best regards, Frank P.S.: I agree that the lazy folder icon loading issue mentioned elsewhere in this thread can be disturbing sometimes and could be improved. ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
Re: Patch: Remove kDebug includes from kio/kio
On Thursday 10 January 2013 15:35:29 Jignesh Kakadiya wrote: Hi David, I have removed all the kdebug.h includes from kio/kio. Please review it. Go for it :) -- David Faure, fa...@kde.org, http://www.davidfaure.fr Working on KDE, in particular KDE Frameworks 5 ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
Re: KDirModelV2, KDirListerV2 and UDSEntryV2 suggestions
On Thursday, January 10, 2013 11:30:35 Mark wrote: very rapidly, the actual view is still barely usable as long as files are dripping in. This is the same in QML (ListView) as in QWidgets The issue here is that the view - even though it's not showing the items - is doing all the calculations for every item added if so, then the QML is being done wrong in some way. the default views in QML (along with some we've written ourselves) only creates delegates for items that are shown along with +/-N pages for caching (so when you scroll you don't see things updating; the updates are happening N pages further down the scrolling) under no circumstances should the entire set of icons be laid out at any time. memory usage and CPU usage simply prohibits this, as you noted :) QML makes it really simple to just lay out what is needed (shown +/- some scroll-ahead pages); this in turn makes it easier to write the model so it is lazy loading: only load the essentials (even progressively; QML views handle this just fine) and then when requested load details. because QML binds data to properties, the async nature of this does not cause further complexity in your code: it all happens in the QML runtime. a new delegate is made for a given index (automatically based on scrol position and view size), it binds to the data for that index, the model then knows something has requested that data and goes to fetch it (whenever/however it feels :) and when it updates that data then the delegate also automatically is updated .. all without dealing with paint events and other annoyances of views with delegates written in the imperative style. this allows huge (even infinite!) size models to be written that feed views that are performant even on mobile devices ... with very little effort. in the case of dolphin, i can imagine a goal that could lead to over- calculating: wanting even spacing between all icons, which in turn means knowing how much space each and every icon will require (thumbnail? how much text? etc.) if that is the case currently, i would suggest dropping this goal and only change the spacing dimension when new icons are shown that require it. (again, easy enough with QML.) also, the model itself can process some statistics (e.g. longest name) as it loads. this can be put into a Q_PROPERTY that the QML can then access. with C++/QWidget, the annoyance with this approach is then you need to reference the C++ object with the exact type and you end up welding models and views together and .. with QML all the properties are dynamic. you can test for their existence and use them if they are there ... and this is also rather efficiently implemented. all of this is a very long-winded way of saying: if you do QML correctly and write the model accordingly, you should be able to have a stupidly fast view. (at least from the user's perspective :) i know because i've seen QML views on models with millions of entries scrolling insanely smoothly with complex and beautiful delegates. i've also helped write a couple. including doing it wrong a few times while learning :) -- Aaron J. Seigo signature.asc Description: This is a digitally signed message part. ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
Re: KDirModelV2, KDirListerV2 and UDSEntryV2 suggestions
On Thu, Jan 10, 2013 at 12:20 PM, Aaron J. Seigo ase...@kde.org wrote: On Thursday, January 10, 2013 11:30:35 Mark wrote: very rapidly, the actual view is still barely usable as long as files are dripping in. This is the same in QML (ListView) as in QWidgets The issue here is that the view - even though it's not showing the items - is doing all the calculations for every item added if so, then the QML is being done wrong in some way. the default views in QML (along with some we've written ourselves) only creates delegates for items that are shown along with +/-N pages for caching (so when you scroll you don't see things updating; the updates are happening N pages further down the scrolling) under no circumstances should the entire set of icons be laid out at any time. memory usage and CPU usage simply prohibits this, as you noted :) QML makes it really simple to just lay out what is needed (shown +/- some scroll-ahead pages); this in turn makes it easier to write the model so it is lazy loading: only load the essentials (even progressively; QML views handle this just fine) and then when requested load details. because QML binds data to properties, the async nature of this does not cause further complexity in your code: it all happens in the QML runtime. a new delegate is made for a given index (automatically based on scrol position and view size), it binds to the data for that index, the model then knows something has requested that data and goes to fetch it (whenever/however it feels :) and when it updates that data then the delegate also automatically is updated .. all without dealing with paint events and other annoyances of views with delegates written in the imperative style. this allows huge (even infinite!) size models to be written that feed views that are performant even on mobile devices ... with very little effort. in the case of dolphin, i can imagine a goal that could lead to over- calculating: wanting even spacing between all icons, which in turn means knowing how much space each and every icon will require (thumbnail? how much text? etc.) if that is the case currently, i would suggest dropping this goal and only change the spacing dimension when new icons are shown that require it. (again, easy enough with QML.) also, the model itself can process some statistics (e.g. longest name) as it loads. this can be put into a Q_PROPERTY that the QML can then access. with C++/QWidget, the annoyance with this approach is then you need to reference the C++ object with the exact type and you end up welding models and views together and .. with QML all the properties are dynamic. you can test for their existence and use them if they are there ... and this is also rather efficiently implemented. all of this is a very long-winded way of saying: if you do QML correctly and write the model accordingly, you should be able to have a stupidly fast view. (at least from the user's perspective :) i know because i've seen QML views on models with millions of entries scrolling insanely smoothly with complex and beautiful delegates. i've also helped write a couple. including doing it wrong a few times while learning :) -- Aaron J. Seigo ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel Hi Aaron, I will explain my model - qml listview approach a little. This is 100% off topic though. I'm loading my massive file list (1 million) into a KDirModel subclass [1]. That class is then assigned to QML [2]. Where it's being used [3]. Now when i'm opening a massive folder [4] i do see my data in the qml part very rapidly. No issue thus far. But when i start scrolling _while_ the data is dripping in it's ... not very responsive. It works, no doubt about it but isn't fluid. As soon as the data finished loading (aka, no more data is added to the model) the list is very responsive and can easily scroll through millions of items. Am i doing something wrong? [1] http://gitorious.org/porpoise/master/blobs/master/dirmodel.h [2] http://gitorious.org/porpoise/master/blobs/master/main.cpp#line26 [3] http://gitorious.org/porpoise/master/blobs/master/qml/Porpoise/ViewContainer.qml#line27 [4] http://gitorious.org/porpoise/master/blobs/master/qml/Porpoise/main.qml#line24 ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
Re: KDirModelV2, KDirListerV2 and UDSEntryV2 suggestions
On Thursday, January 10, 2013 14:12:30 you wrote: part very rapidly. No issue thus far. But when i start scrolling _while_ the data is dripping in it's ... not very responsive. after a quick look: the DirModel is doing all the data retrieval in the main thread as well as instantly on demand (no batching, e.g.). this means: * every allocation of a FileItem and all the work to populate it takes cycles away from the UI thread at the same time as the list is trying to scroll * KFileItems are created *and* populated regardless of whether they will be used or not, which means during scrolling things will get slower the more you scroll. they are also fully populated on creation. neither of these things is good. (and some of these issues have already been touched on in this thread) this probably requires changes to KDirModel such as: * put data processing into a thread; whether that means putting the whole of KDirModel into a worker thread or just having a thread pool for processing new KDirModelNodes / KFileItems .. others more familiar with the code would be able to answer that far better than i can right now :) but there should be near-zero time spent in the GUI thread for actually getting the file information * KFileItem needs to support lazy loading to avoid unecessary work when the information is not actually requested. there is even a TODO about this in KFileItemPrivate::init() in the 4.10 branch :) * methods like KDirModelPrivate::_k_slotNewItems need be faster and/or have their work moved into a thread. the number of times beginInsertRows/endInsertRows is called should also be minimized as that will innevitably create pauses in the GUI thread while it reacts to this perhaps some of this is already done or being done in the frameworks branch (haven't looked yet; apologies in advance for that) as to the DirModel class: * the image cache is done in the main thread. that means updating a database and writing to disk in the UI thread .. when the goal is 60fps or better, this is likely going to be too slow. * KIO::PreviewJobs are also in the main GUI thread. could be another cause for GUI latency. * DirModel::rebuildUrlToIndex is O(n) and called whenever there are new items * DirModel::updatePreview calls setData for each preview that it gets back. that causes dataChanged to be emitted. again, for each index. this needs to be batched up and/or DirModel should override data() itself to return Qt::DecorationRole when it has a preview. btw, DirModel::itemForIndex looks like it is leaking memory. so it seems DirModel is need of a scalibility refactor/rewrite, and KDirModel needs to be made read for use with threads in some fashion as well have its base performance improved with more lazy loading and batching of requests. (this feels earily similar to the model rewrites Marco ended up doing for Plasma Active ... threads and batching, threads and batching. the UI thread must not be disturbed! :) -- Aaron J. Seigo signature.asc Description: This is a digitally signed message part. ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
Re: KDirModelV2, KDirListerV2 and UDSEntryV2 suggestions
On Thu, Jan 10, 2013 at 3:53 PM, Aaron J. Seigo ase...@kde.org wrote: On Thursday, January 10, 2013 14:12:30 you wrote: part very rapidly. No issue thus far. But when i start scrolling _while_ the data is dripping in it's ... not very responsive. after a quick look: the DirModel is doing all the data retrieval in the main thread as well as instantly on demand (no batching, e.g.). this means: * every allocation of a FileItem and all the work to populate it takes cycles away from the UI thread at the same time as the list is trying to scroll * KFileItems are created *and* populated regardless of whether they will be used or not, which means during scrolling things will get slower the more you scroll. they are also fully populated on creation. neither of these things is good. (and some of these issues have already been touched on in this thread) this probably requires changes to KDirModel such as: * put data processing into a thread; whether that means putting the whole of KDirModel into a worker thread or just having a thread pool for processing new KDirModelNodes / KFileItems .. others more familiar with the code would be able to answer that far better than i can right now :) but there should be near-zero time spent in the GUI thread for actually getting the file information * KFileItem needs to support lazy loading to avoid unecessary work when the information is not actually requested. there is even a TODO about this in KFileItemPrivate::init() in the 4.10 branch :) * methods like KDirModelPrivate::_k_slotNewItems need be faster and/or have their work moved into a thread. the number of times beginInsertRows/endInsertRows is called should also be minimized as that will innevitably create pauses in the GUI thread while it reacts to this perhaps some of this is already done or being done in the frameworks branch (haven't looked yet; apologies in advance for that) as to the DirModel class: * the image cache is done in the main thread. that means updating a database and writing to disk in the UI thread .. when the goal is 60fps or better, this is likely going to be too slow. * KIO::PreviewJobs are also in the main GUI thread. could be another cause for GUI latency. * DirModel::rebuildUrlToIndex is O(n) and called whenever there are new items * DirModel::updatePreview calls setData for each preview that it gets back. that causes dataChanged to be emitted. again, for each index. this needs to be batched up and/or DirModel should override data() itself to return Qt::DecorationRole when it has a preview. Would a 300ms timer be acceptable in user experience? So if there are new thumbnails available, batch them and update the model only once every 300ms. Or is 300ms too much? btw, DirModel::itemForIndex looks like it is leaking memory. so it seems DirModel is need of a scalibility refactor/rewrite, and KDirModel needs to be made read for use with threads in some fashion as well have its base performance improved with more lazy loading and batching of requests. (this feels earily similar to the model rewrites Marco ended up doing for Plasma Active ... threads and batching, threads and batching. the UI thread must not be disturbed! :) Interesting! My DirModel is coming directly from plasma-mobile [1] which is not doing the threading you describe. Is there another dirmodel fork somewhere that does that? Link please :) -- Aaron J. Seigo ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel You just verified the need for KDirModel, KDirLister and KFileItem to be optmized for scalibility :-) My DirModel is basically a KDirModel but with some convenient parts. How much work would it be to let KDirModel run in it's separate thread? I'm guessing the public KDirModel would internally be nothing more then a signal/slot mechanism to a KDirModelPrivate class which is sent to it's own thread? Or would you say that sending KDirModel to it's own thread it too risky and should be done by the developer using KDirModel (like dolphin and my app) rather then internally in the private API? [1] http://quickgit.kde.org/?p=plasma-mobile.gita=blobh=1539ae3d6cf54f96af216593e5c7517df820a6b8hb=50664f4fae8243280f1e7362447a864b37a2039bf=components%2Fdirmodel%2Fdirmodel.cpp ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
Re: KDirModelV2, KDirListerV2 and UDSEntryV2 suggestions
Hi, 2013/1/10 Aaron J. Seigo: in the case of dolphin, i can imagine a goal that could lead to over- calculating: wanting even spacing between all icons, which in turn means knowing how much space each and every icon will require (thumbnail? how much text? etc.) The number of text lines required for each file name is indeed calculated in advance in order to get the 'maximum scroll offset' for the view, i.e., the total height of the view containing all items. This is used to set up the 'scroll bar value' - 'view offset' mapping. One could possibly avoid that by assuming that all non-visible items only use one line for the name, and only update that value for items which are inside (or close to) the visible area. But before considering such a thing, one would need to check if that is really a significant bottleneck. Quite a bit of code would need to be changed, and I can imagine that it's not completely trivial to get this right. if that is the case currently, i would suggest dropping this goal and only change the spacing dimension when new icons are shown that require it. (again, easy enough with QML.) Well, the hard part is probably to port Dolphin's views to QML in the first place ;-) I must admit that my QML knowledge is rather limited, but AFAIK, it does not provide any kind of tree view out of the box. Moreover, it seems to me that QML views work best with QAbstractItemModel subclasses. However, one of the main goals Peter had when rewriting the view engine was to get rid of QModelIndex, SortFilterProxyModels and things like these. And I really have the feeling that using plain ints to index items and not needing things like mapToSource() in every other function really does make things easier. Best regards, Frank ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
Re: KDirModelV2, KDirListerV2 and UDSEntryV2 suggestions
On Thu, Jan 10, 2013 at 7:24 PM, Aaron J. Seigo ase...@kde.org wrote: On Thursday, January 10, 2013 16:49:15 you wrote: On Thu, Jan 10, 2013 at 3:53 PM, Aaron J. Seigo ase...@kde.org wrote: * DirModel::updatePreview calls setData for each preview that it gets back. that causes dataChanged to be emitted. again, for each index. this needs to be batched up and/or DirModel should override data() itself to return Qt::DecorationRole when it has a preview. Would a 300ms timer be acceptable in user experience? So if there are new thumbnails available, batch them and update the model only once every 300ms. Or is 300ms too much? probably too much (and probably more than is needed for the rate of file listing). but once you have that delay in there it is easy to test it, right? 100, 200, 300 .. it will really come down to average listing performance. if items come in clumps, then the update might only need to be really small (50ms or even less) to drop the # of updates by orders of magnitude. Guess i just have to experiment there. Interesting! My DirModel is coming directly from plasma-mobile [1] which is not doing the threading you describe. yes, i noticed :) unfortunately for you, that class was thrown together very quickly for the limited needs of listing directories on a mobile device. file counts tend to be rather low there, and file system browsing is not as core a function as it is on the desktop. so for Plasma Active, this was acceptable performance. for the desktop .. not even close :) Well, the class didn't have _that_ much extra stuff on top of KDirModel and i added even more so mine is quite different at the moment. Though not threaded. Is there another dirmodel fork somewhere that does that? Link please :) well, what we ought to do is do a threaded version and then put it somewhere we can all share it properly :) i'll discuss with Marco what he thinks and come back with a proposal. you make it sound like it's closed source..? no link to some exotic highly experimental git branch? You just verified the need for KDirModel, KDirLister and KFileItem to be optmized for scalibility :-) yep. How much work would it be to let KDirModel run in it's separate thread? I'm guessing the public KDirModel would internally be nothing more then a signal/slot mechanism to a KDirModelPrivate class which is sent to it's own thread? Or would you say that sending KDirModel to it's own thread it too risky and should be done by the developer using KDirModel (like dolphin and my app) rather then internally in the private API? for scalable models in Plasma Active, we've been putting the threading in the model itself exactly as you described (threaded signal/slot communication between the public API model and members in its private class) since doing this in each application is error prone. once it is done properly in the model, it then works everywhere. so +1 from me for making KDirModel do its business in an encapsulated thread. I guess that is the way forward then. But i do wonder, if the KDirModel API stays the same but the internals change completely, is that KF5 material or can it go in 4.xx as well? -- Aaron J. Seigo (remember to press reply to all ^_-) ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
Re: KDirModelV2, KDirListerV2 and UDSEntryV2 suggestions
On Thursday, January 10, 2013 19:37:39 you wrote: Is there another dirmodel fork somewhere that does that? Link please :) well, what we ought to do is do a threaded version and then put it somewhere we can all share it properly :) i'll discuss with Marco what he thinks and come back with a proposal. you make it sound like it's closed source..? no link to some exotic highly experimental git branch? hehe.. no, none yet ... reason i want to talk about with Marco is this will impact our devel efforts elsewhere in the plasma-mobile code, so we would need to factor any changes in there. and this really seems like the sort of thing that ought to be available in the Plasma Components in kde-runtime so that everyone can use it (and hack on it) without making copies :) so +1 from me for making KDirModel do its business in an encapsulated thread. I guess that is the way forward then. But i do wonder, if the KDirModel API stays the same but the internals change completely, is that KF5 material or can it go in 4.xx as well? imho, probably KF5 because: a) we want KF5 sooner rather than later, so efforts should focus there.[1] b) KDirModel is such a key component with a high complexity and few people who know it very well; so it needs lots of testing, and 5.0 would be a better target for that kind of development imho c) it will give us something to market about 5.0 - SO MUCH FASTER / MORE FLUID DIRECTORY LISTINGS (that's mostly, but only mostly, tongue-in-cheek) [1] as an aside: we've re-focused effort on libplasma2 in KF5. the new QML based containment stuff being worked on has been moved to libplasma2-only and i expect other such development in Plasma to continue that way. we're also expecting / planning Plasma Active 5 to be focused on moving to Qt5 + KF5. -- Aaron J. Seigo signature.asc Description: This is a digitally signed message part. ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
