Hi, -- Be sure to read all of this (long!) mail or parts won't make sense! -- -- cross posting to frameworks and KCD since it concerns both --
Recently i've been playing a lot with KDirModel in combination with QML and i certainly found quite a few points for improvement. Even so much that i think i version 2 would be best. The same is true for KDirLister because i'd like to change data that it's signals return when new files are emitted. Before i explain any API details it is important to know what i want to do with KFileItem. KFileItem itself is somewhat OK though i'd like to change that as well. That's another subject though. What currently happens when you see a directory listing using KDirModel and KDirLister is that KDirLister is listening to a directory and emits all the results as one giant list with KFileItem in it. In "most" cases that would be oke, but if you have really massive folders then having a list of - let say - 100.000 files/folders in KFileItem objects begins to be quite taxing for the CPU. That alone isn't an issue, but the fact that you won't even see those 100.000 items in any view makes it quite wasteful to even store it. So what i want to do is store a big (much smaller in memory) UDSEntry list where KFileItem objects are "lazy loaded" when actually needed. That would be a lot less taxing for the CPU and memory. Also the caching in KDirListerCache would very likely be a lot smaller when using a UDSEntryList. My following KDirModelV2 api suggestion doesn't reflect any of that on the API level below. KDirListerV2 will reflect it along with UDSEntryV2. Here is my proposed API for both of them and why i'd like to remove/change some methods. --- KDirModelV2 API ---- (on top of the functions that have to be re-implemented) - void setDirLister( KDirLister* dirLister ); - KDirLister* dirLister() const; - KFileItem itemForIndex( const QModelIndex& index ) const; - void setAdditionalRoles(const QHash<int, QByteArray> & roleNames) - The column names should get some defaults that can also be used in QML. - All the required functions for drag/drop support should be implemented though that doesn't matter much anymore when used in QML.. Remove the following ones from KDirModel (thus won't be added to KDirModelV2) - QModelIndex indexForItem( const KFileItem* ) const; (already deprecated) - QModelIndex indexForItem( const KFileItem& ) const; - QModelIndex indexForUrl(const QUrl& url) const; * reasoning for the above: these are convenient functions, they don't belong in this class. If a user wants them then just subclass it and add those convenient functions. - void expandToUrl(const QUrl& url); * reasoning: expanding to some folder can be done bet using setRootIndex (in the view) - void itemChanged( const QModelIndex& index ); * perhaps as an internal API, certainly not visible for the outside world. - static QList<QUrl> simplifiedUrlList( const QList<QUrl> & urls ); * Not the right place. Either re-implement the model or use KIO::ListDir (or even KDirLister) - void requestSequenceIcon(const QModelIndex& index, int sequenceIndex); - void needSequenceIcon(const QModelIndex& index, int sequenceIndex); * Yes, but should be done internally and automatically. The user should only have an option to enable/disable this. --- KDirListerV2 API --- For the KDirListerV2 i want to get rid of all KFileItem mentions. It's a nice container but imho should not be used there. It should be constructed, used and returned in KDirModel. Another major point is getting rid of QWidget as a dependency. * signals * - void started(); - void completed(); - void canceled(); - void redirection( const QUrl& _url ); - void redirection( const QUrl& oldUrl, const QUrl& newUrl ); - void clear(); - void newItems( const UDSEntryVector& items ); - void itemsAdded(const QUrl& directoryUrl, const UDSEntryVector& items); - void itemsDeleted( const UDSEntryVector& items ); - void refreshItems( const UDSEntryVector& items ); // i doubt if this should even stay - void infoMessage( const QString& msg ); - void percent( int percent ); - void totalSize( KIO::filesize_t size ); - void processedSize( KIO::filesize_t size ); - void speed( int bytes_per_second ); * removed signals * - void started( const QUrl& _url ); // You know the object and url your're listening to. No need to emit the url. - void completed( const QUrl& _url ); // see above - void canceled( const QUrl& _url ); // see above - void clear( const QUrl& _url ); // see above - void itemsFilteredByMime( const KFileItemList& items ); // Wrong place to filter * functions * - virtual bool openUrl( const QUrl& _url, OpenUrlFlags _flags = NoFlags, Filter filters ); // The filters are the same as Qt with QDir::Filter http://qt-project.org/doc/qt-4.8/qdir.html + a filter for delayed mime detection and auto update. - <new> Filter filter() - <new> void setFilter(Filter filters) - <new> refresh() - virtual void stop(); - QUrl url() const; - QList<QUrl> directories() const; - bool isFinished() const; - KFileItem rootItem() const; - virtual void setMimeFilter( const QStringList &mimeList ); - QStringList mimeFilters() const; * removed function * - virtual void stop( const QUrl& _url ); // Again, you know which url your indexing, no need for functions/signals like this - virtual void updateDirectory( const QUrl& _dir ); - KFileItemList itemsForDir( const QUrl& dir, - KFileItemList items( WhichItems which = FilteredItems ) const; - static KFileItem cachedItemForUrl(const QUrl& url); - bool autoErrorHandlingEnabled() const; // Adios QWidget - void setAutoErrorHandlingEnabled( bool enable, QWidget *parent ); // Adios QWidget - void setMainWindow( QWidget *window ); // Adios QWidget - QWidget *mainWindow(); // Adios QWidget - bool dirOnlyMode() const; - virtual void setDirOnlyMode( bool dirsOnly ); - bool showingDotFiles() const; - virtual void setShowingDotFiles( bool _showDotFiles ); - virtual void setNameFilter( const QString &filter ); // For this and the 3 below, searching is easier using KCompleter or whatever it's name is. - QString nameFilter() const; - bool matchesFilter( const QString& name ) const; - virtual KFileItem findByUrl( const QUrl& _url ) const; - virtual KFileItem findByName( const QString& name ) const; - virtual void clearMimeFilter(); // Setting an empty list also clears it. - virtual void emitChanges(); Next up (in this already way too long mail) is UDSEntryV2. Right now if you have a massive folder the UDSEntry is quite high in the call costs (far from the top, but you will notice it due to the hash table). And it doesn't even need a hash for it to function. Thus i would like to implement a significantly lighter UDSEntry. One that uses a Vector so the memory is also aligned + very fast lookup. Another thing is the http://api.kde.org/4.9-api/kdelibs-apidocs/kio/html/classKIO_1_1UDSEntry.html#a8c3c5c6ee998a9f0e413b8aafcc98597 that currently also supports "UDS_EXTRA" and "UDS_EXTRA_END". I seriously doubt that the functionality is even used anywhere. So my proposed structure for UDSEntryV2 would look somewhat like this: QVector<QString> I would even go as far as making a fixed length QVector. One with all the UDS_ lines (about 30?) and one with details = 0 thus thiny QVectors with only 2 elements. However, the current UDS stuff also has some knowledge about how a value should be stored (string, int, ...) so if that information is "really" required (which i doubt) then the following structure would also work: QVector<QVariant> The lookup for an element would be very easy: <vector>.value(UDS_SIZE) As you have probably seen already, a list of UDS entries would become "UDSEntryVector" which would be defined as: typedef QVector<QString> UDSEntryV2; typedef QVector<UDSEntry> UDSEntryVector; Why vectors? Well, since Qt5 the vectors are really better in nearly every case compared to QList. Also, if data doesn't change then it's pointless to use the power of a QList when it's not even needed. An added bonus is memory alignment with a QVector. So there it is, my proposal to completely restructure KDirModel, KDirLister, UDSEntry and UDSEntryList. There is no "real" need for it at the moment since other operations are still a _lot_ slower then any of the above (like sorting), but i'd like to change make these changes to have a good base in Qt5 for file listings - something i'm working on a lot lately. The need _is_ there if you browse through massive folders. KDE (KIO specifically) really isn't up to the task to browse through folders with 10.000+ files, you get slowdowns everywhere in the code. (hash table calculations + resizing, cleanUrl checks in the model, a lot of KFileItem instances ...). It will be ready to handle those cases as well with the changes i mentioned above and it greatly simplifies the API's, likely greatly reduced memory cost, greatly reduced CPU cost and just a lot more scalable for bigger file systems. Perhaps we can deprecate the current KDirModel, KDirLister, UDSEntry and UDSEntryList if this change ends up being nice :) I'd like to know if these changes would be to start working on and make a review request for. All we need to do now is agree on the API changes i guess. Yes, this is for frameworks only, not for KDE 4.x. Kind regards, Mark _______________________________________________ Kde-frameworks-devel mailing list Kde-frameworks-devel@kde.org https://mail.kde.org/mailman/listinfo/kde-frameworks-devel