Re: Opinions on KIO Slave side sorting? Possible GSoC project?
On Sun, May 3, 2015 at 12:44 AM, David Faure fa...@kde.org wrote: On Thursday 08 January 2015 10:45:13 Mark Gaiser wrote: (wow, time flies) The issue i see here is different processes. KIO::listDir is a process, KDirListerCache lives in the client process. I guess you didn't mean this litterally, because in that case it's wrong, both happen in the client process. You want to do sorting in both processes. 1. When the client initially asks a listing you want to do the sorting as quick as possible to send back as little data as possible. So sorting on the SlaveBase side. As long as not all data is known in the KDirListerCache sorting would have to be done on the SlaveBase side. That would be the case to change sort order _while_ data is dripping in. 2. When all data is fetched you don't want to go over a socket to request a different sorting method so then you would need to have the same sorting operations on the KIO client side with data fed from KDirListerCache. Obviously more complex than the current solution, but OK, let's see where that leads us. I do think that KIO would need a new class for this that can handle a set of predefined sorting and filtering operations if this entire slave side sorting is to be considered for KIO. Just to be clear, when i said slave side sorting i meant sorting in - for instance - SlaveBase. Not in the actual slave plugins. Where exactly, i don't know, but certainly before data is being send back to the application that requested a KIO::listDir. Sorry if i confused you here. Well, this requires the actual kioslave to first give all items to SlaveBase, doesn't it? Otherwise it can't sort them. So instead of incremental listing, this would wait until everything is available and then send everything; not sure it would appear faster to the user. (this depends on whether the time is spent doing the actually listing and creating of UDSEntries, or if more time is spent sending the stuff over the socket --- but you optimized that :-) The advantage i see with doing this is allowing data to be visible for the user as soon as data is available which will give the user a smoother and faster experience. Interesting you should say that, I think the suggested approach actually goes into the opposite direction from this goal. You need for all udsentries to exist before sorting, so it might take longer for the first items to appear, compared to the current solution, where a first set of items get sent over as soon as they are available, and then the other batch comes in and the two get sorted together. Which can make items move down, so I'm not saying it's ideal, but it technically does reach the above goal better :) It allows for a mechanism of show me the first 100 items of this massive folder in this particular sort order. Something that isn't fully possible right now since the client would have to wait till all data has arrived over the socket, then sort it, then present it. All data would still have to be fetched on the slave side, but only X number of entries would have to be send to the client allowing it to immediately present the data. The rest of the data can then be fetched on a need to know basis. Eg. when scrolling down for the next batch of 100 items. By that time the slave would probably (depending on the folder size and sort order) be done fetching all items and is ready to immediately send those pre sorted batches to the client. So the very first step is to find out how long it takes to create udsentries in the slave, and how long it takes to transfer them over the socket. If the first one is much bigger than the second one, then this idea would delay quite a lot the time until when the first items appear... An added advantage is that the client could then just use KIO as streaming API. Just like a youtube api or whatever streaming api. The client would just have to implement canFetchMode and fetchMode if the Qt classes are used. It should prevent the client from needing to implement complicated threading and UI tricks to keep the UI smooth. fetchMore is pull (get me more now), while entries get listed asynchronously and pushed via a signal, so I don't see how that would work, what would fetchMore really do? -- David Faure, fa...@kde.org, http://www.davidfaure.fr Working on KDE Frameworks 5 Time flies indeed! I completely forgot about this thread. Just going to respond here instead of inline. I fully agree with your comments and actually start to doubt the usefulness (again) of having a SlaveBase side sorting approach. I theory it sounds so neat to have a streaming api approach, but in practice the SlaveBase side would just have to wait till all items are fetched before it can sort. There would be a measurable time difference between: 1. SlaveBase fetching all items and sending a sorted batch to the client 2. and SlaveBase
Re: Opinions on KIO Slave side sorting? Possible GSoC project?
On Thursday 08 January 2015 10:45:13 Mark Gaiser wrote: (wow, time flies) The issue i see here is different processes. KIO::listDir is a process, KDirListerCache lives in the client process. I guess you didn't mean this litterally, because in that case it's wrong, both happen in the client process. You want to do sorting in both processes. 1. When the client initially asks a listing you want to do the sorting as quick as possible to send back as little data as possible. So sorting on the SlaveBase side. As long as not all data is known in the KDirListerCache sorting would have to be done on the SlaveBase side. That would be the case to change sort order _while_ data is dripping in. 2. When all data is fetched you don't want to go over a socket to request a different sorting method so then you would need to have the same sorting operations on the KIO client side with data fed from KDirListerCache. Obviously more complex than the current solution, but OK, let's see where that leads us. I do think that KIO would need a new class for this that can handle a set of predefined sorting and filtering operations if this entire slave side sorting is to be considered for KIO. Just to be clear, when i said slave side sorting i meant sorting in - for instance - SlaveBase. Not in the actual slave plugins. Where exactly, i don't know, but certainly before data is being send back to the application that requested a KIO::listDir. Sorry if i confused you here. Well, this requires the actual kioslave to first give all items to SlaveBase, doesn't it? Otherwise it can't sort them. So instead of incremental listing, this would wait until everything is available and then send everything; not sure it would appear faster to the user. (this depends on whether the time is spent doing the actually listing and creating of UDSEntries, or if more time is spent sending the stuff over the socket --- but you optimized that :-) The advantage i see with doing this is allowing data to be visible for the user as soon as data is available which will give the user a smoother and faster experience. Interesting you should say that, I think the suggested approach actually goes into the opposite direction from this goal. You need for all udsentries to exist before sorting, so it might take longer for the first items to appear, compared to the current solution, where a first set of items get sent over as soon as they are available, and then the other batch comes in and the two get sorted together. Which can make items move down, so I'm not saying it's ideal, but it technically does reach the above goal better :) It allows for a mechanism of show me the first 100 items of this massive folder in this particular sort order. Something that isn't fully possible right now since the client would have to wait till all data has arrived over the socket, then sort it, then present it. All data would still have to be fetched on the slave side, but only X number of entries would have to be send to the client allowing it to immediately present the data. The rest of the data can then be fetched on a need to know basis. Eg. when scrolling down for the next batch of 100 items. By that time the slave would probably (depending on the folder size and sort order) be done fetching all items and is ready to immediately send those pre sorted batches to the client. So the very first step is to find out how long it takes to create udsentries in the slave, and how long it takes to transfer them over the socket. If the first one is much bigger than the second one, then this idea would delay quite a lot the time until when the first items appear... An added advantage is that the client could then just use KIO as streaming API. Just like a youtube api or whatever streaming api. The client would just have to implement canFetchMode and fetchMode if the Qt classes are used. It should prevent the client from needing to implement complicated threading and UI tricks to keep the UI smooth. fetchMore is pull (get me more now), while entries get listed asynchronously and pushed via a signal, so I don't see how that would work, what would fetchMore really do? -- David Faure, fa...@kde.org, http://www.davidfaure.fr Working on KDE Frameworks 5 ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
Re: Opinions on KIO Slave side sorting? Possible GSoC project?
On Sunday 04 January 2015 21:37:21 Mark Gaiser wrote: Hi, I was reading back an mailing list thread i started about half a year ago [1]. Later on in that thread Aaron suggested a cool idea of slave side sorting [2]. At that time i found it cool, but didn't really see a need for it. Dolphin is doing a fine job at sorting anyway, so why would we bother moving it to the KIO side. Well, a good reason for moving it to the KIO side just became apparent to me when looking at this RR [3]. Sorting is already quite complex in Dolphin. Adding in QCollator while also using threads makes it a bit more nasty since a QCollator object per thread is required (hence the RR). Sure, the end result (and Dolphin is probably going to move there) is to use QCollatorSortKey which will make sorting quite a bit faster [4] and remove a QCollator requirement, but it won't be any cleaner. Quite the opposite, It will be more complicated due to the bookkeeping required to use QCollatorSortKey. Moving complexity elsewhere doesn't make things less complex overall, though :-) All this sorting stuff is quite complicated and only available in Dolphin. Other people would need to duplicate it's efforts for the same results. That and considering that KIO is a framework that can be used by external (non KDE) parties made me think that having slave side sorting might be a very good thing to have. It would allow for performance optimizations that everyone can use and allows every KIO user to easily add sorting. It would completely remove a need for Dolphin to have sorting and would greatly simplify code that uses KIO. This is an argument for doing sorting in KIO. Sure, why not. But I don't see how it could be done slave-side. When using KDirModel/KDirLister the result of the directory listing is put into KDirListerCache and then reused from any view that wants to list that dir. These views might have different sorting... What then? Sorting in the slaves is mixing core and gui issues, IMHO. It's not the slave's job to know what kind of sorting the view wants. In fact I see no advantage in implementing sorting on the slave side compared to sorting on the app side. If it's about not doing that in the app's main thread so it remains responsive, then threads are the alternative as you already noted. I haven't looked into QCollator in details, but isn't there a way to make it threadsafe? -- David Faure, fa...@kde.org, http://www.davidfaure.fr Working on KDE Frameworks 5 ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
Re: Opinions on KIO Slave side sorting? Possible GSoC project?
On Thu, Jan 8, 2015 at 9:14 AM, David Faure fa...@kde.org wrote: On Sunday 04 January 2015 21:37:21 Mark Gaiser wrote: Hi, I was reading back an mailing list thread i started about half a year ago [1]. Later on in that thread Aaron suggested a cool idea of slave side sorting [2]. At that time i found it cool, but didn't really see a need for it. Dolphin is doing a fine job at sorting anyway, so why would we bother moving it to the KIO side. Well, a good reason for moving it to the KIO side just became apparent to me when looking at this RR [3]. Sorting is already quite complex in Dolphin. Adding in QCollator while also using threads makes it a bit more nasty since a QCollator object per thread is required (hence the RR). Sure, the end result (and Dolphin is probably going to move there) is to use QCollatorSortKey which will make sorting quite a bit faster [4] and remove a QCollator requirement, but it won't be any cleaner. Quite the opposite, It will be more complicated due to the bookkeeping required to use QCollatorSortKey. Moving complexity elsewhere doesn't make things less complex overall, though :-) True :( All this sorting stuff is quite complicated and only available in Dolphin. Other people would need to duplicate it's efforts for the same results. That and considering that KIO is a framework that can be used by external (non KDE) parties made me think that having slave side sorting might be a very good thing to have. It would allow for performance optimizations that everyone can use and allows every KIO user to easily add sorting. It would completely remove a need for Dolphin to have sorting and would greatly simplify code that uses KIO. This is an argument for doing sorting in KIO. Sure, why not. But I don't see how it could be done slave-side. When using KDirModel/KDirLister the result of the directory listing is put into KDirListerCache and then reused from any view that wants to list that dir. These views might have different sorting... What then? Regarding KDirListerCache, i think that should stay as it is. The sorting should not influence the cache in anyway, the cache should just be a raw container of the actual data like it is now. When you want to have a certain sorted view you simply get the cache back for that folder with a proxy on top of it that re-arranges the indexes in the sorted order. You can compare this with any Q*Model class with a QSortFilterProxyModel on top of that. The principle is the same, the implementation would just be without models and directly on QVector/QList objects. The extra memory overhead for sorting would just be one vector of ints. Fairly cheap :) The issue i see here is different processes. KIO::listDir is a process, KDirListerCache lives in the client process. You want to do sorting in both processes. 1. When the client initially asks a listing you want to do the sorting as quick as possible to send back as little data as possible. So sorting on the SlaveBase side. As long as not all data is known in the KDirListerCache sorting would have to be done on the SlaveBase side. That would be the case to change sort order _while_ data is dripping in. 2. When all data is fetched you don't want to go over a socket to request a different sorting method so then you would need to have the same sorting operations on the KIO client side with data fed from KDirListerCache. I do think that KIO would need a new class for this that can handle a set of predefined sorting and filtering operations if this entire slave side sorting is to be considered for KIO. Sorting in the slaves is mixing core and gui issues, IMHO. It's not the slave's job to know what kind of sorting the view wants. In fact I see no advantage in implementing sorting on the slave side compared to sorting on the app side. If it's about not doing that in the app's main thread so it remains responsive, then threads are the alternative as you already noted. Just to be clear, when i said slave side sorting i meant sorting in - for instance - SlaveBase. Not in the actual slave plugins. Where exactly, i don't know, but certainly before data is being send back to the application that requested a KIO::listDir. Sorry if i confused you here. The advantage i see with doing this is allowing data to be visible for the user as soon as data is available which will give the user a smoother and faster experience. It allows for a mechanism of show me the first 100 items of this massive folder in this particular sort order. Something that isn't fully possible right now since the client would have to wait till all data has arrived over the socket, then sort it, then present it. All data would still have to be fetched on the slave side, but only X number of entries would have to be send to the client allowing it to immediately present the data. The rest of the data can then be fetched on a need to know basis. Eg. when scrolling
Opinions on KIO Slave side sorting? Possible GSoC project?
Hi, I was reading back an mailing list thread i started about half a year ago [1]. Later on in that thread Aaron suggested a cool idea of slave side sorting [2]. At that time i found it cool, but didn't really see a need for it. Dolphin is doing a fine job at sorting anyway, so why would we bother moving it to the KIO side. Well, a good reason for moving it to the KIO side just became apparent to me when looking at this RR [3]. Sorting is already quite complex in Dolphin. Adding in QCollator while also using threads makes it a bit more nasty since a QCollator object per thread is required (hence the RR). Sure, the end result (and Dolphin is probably going to move there) is to use QCollatorSortKey which will make sorting quite a bit faster [4] and remove a QCollator requirement, but it won't be any cleaner. Quite the opposite, It will be more complicated due to the bookkeeping required to use QCollatorSortKey. All this sorting stuff is quite complicated and only available in Dolphin. Other people would need to duplicate it's efforts for the same results. That and considering that KIO is a framework that can be used by external (non KDE) parties made me think that having slave side sorting might be a very good thing to have. It would allow for performance optimizations that everyone can use and allows every KIO user to easily add sorting. It would completely remove a need for Dolphin to have sorting and would greatly simplify code that uses KIO. When sorting moves to KIO, then grouping should also move to KIO imho. The goal with this mail is to see if there is a need for slave side sorting and how this should be used by applications wanting to use it. If there is a demand for this then i'd like to propose to make this a GSoC project where i volunteer to be a co-mentor. I look forward to your opinions. Best regards, Mark [1] http://mail.kde.org/pipermail/kde-frameworks-devel/2014-May/015782.html [2] http://mail.kde.org/pipermail/kde-frameworks-devel/2014-June/016341.html [3] https://git.reviewboard.kde.org/r/121817/ [4] http://kdeblog.mageprojects.com/2014/07/06/natural-string-sorting-an-in-depth-analysis/ ___ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel
