Re: [Interest] how do I load massive table views instantly?
>> 1) quickly calculate/get the total number of items and return that value
>> from rowCount() to let views properly configure their scroll bars;
> is that a method one overrides? where is "rowCount()" defined?
> do i override it by subclassing QSortFilterProxyModel?
Yes, it's one of the pure virtual methods of QAbstractItemModel. And I was
talking about rowCount() in the model which is behind QSortFilterProxyModel
and/or other proxies.
>> 2) load data in the background;
> i think this is already happening
>
>> 3) return some placeholder values from data() for items that have not been
>> loaded yet;
>> 4) when a new batch of items is available, add it to the model's internal
>> storage and emit the dataChanged() signal with appropriate index range.
> wouldn't the view ask the model for whatever data is revealed as visible, all
> by itself?
> so no need to show mock data? the point is i want to show the real data, even
> if the user jumps to the last rows immediately
>
> do i *need* to manually do this?
Yes, views ask only about visible items, but not all visible items may have
real data, especially if whatever provides that data is slow. So, if the
now-loaded items are visible in some views, dataChanged() will update these
views, otherwise nothing will happen. Without the dataChanged() signal views
may never replace mock data with real one.
If the user scrolls to items without real data, you should show at least
something — empty rows are ambiguous/confusing ("not-yet-loaded items?", "items
with empty data?", "bug?").
> thanks for the suggestions, i'm trying to understand
>
> -dave
signature.asc
Description: OpenPGP digital signature
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest
Re: [Interest] how do I load massive table views instantly?
BTW.. I missed that you are using QSRTM, since you are use QSRTM, you shouldn’t need to use any proxy model. Simply use the setFilter and setOrder methods. The setOrder should be handled automatically by your view when you set the model, and enable sorting on the view. Scott -Original Message- From: Interest On Behalf Of David M. Cotter Sent: Monday, June 8, 2020 4:43 PM To: interest@qt-project.org Subject: Re: [Interest] how do I load massive table views instantly? right okay so my source model class is QSqlRelationalTableModel, no subclass. does this answer your question? sorry i'm still learning about this stuff. now that you know that, is it easier to answer my original question? > On Jun 8, 2020, at 1:37 PM, Francis Herne wrote: > > On Friday, 5 June 2020 17:23:37 BST David M. Cotter wrote: >>>>> What table view are you talking about, specifically? >>>> >>>> QTableView with QSortFilterProxyModel >>> >>> What is the model _behind_ QSFPM? >> >> i'm trying to understand your question. i'm not sure what you mean, >> cuz i'm not the SQLite dev what i know is that each song has a couple >> dozen bits of data (name, artist, album, several file paths, rating, >> duration etc etc) besides that, what are you asking?= > > QSortFilterProxyModel is a *proxy* that makes other models sortable. > it only works when its `sourceModel` property is set to an instance of > a different model class that actually contains the data. ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] how do I load massive table views instantly?
One thing I have found in years past, sorting via a proxy when dealing with sql models, is a bad idea. All models, have the "fetch" functionality. Allowing the application, to load the data incrementally.. This is critical for large tables. Yes, your scroll bars are "screwed up", but in the end, you can load very large databases, very quickly. Your limited by the SQL engine/server time to return the first 250 records, rather than the millions in the full table . But, if you use a proxy model, the SQL model is going to have to fully load the table, then sort it. This is equivalent to selecting from the table, and sorting based on a key that isn’t indexed. You wouldn’t want a "sorted" table to change the order as you scrolled down. Ie, if the first 250 records sorted by last name didn’t have an A in the records, and 251 did, you wouldn’t want that record to pop to the top. A much better approach, I have used in the past, is to have the SQL model, change the select statement based on the sort column requested. Ie, select * from table order by column_name asc Is much faster, and a more effective method of sorting the table than select * from table, and then sorting the complete table. For small tables, honestly you might not see any difference, but for big tables the difference can be dramatic. Scott -Original Message- From: Interest On Behalf Of David M. Cotter Sent: Monday, June 8, 2020 4:43 PM To: interest@qt-project.org Subject: Re: [Interest] how do I load massive table views instantly? right okay so my source model class is QSqlRelationalTableModel, no subclass. does this answer your question? sorry i'm still learning about this stuff. now that you know that, is it easier to answer my original question? > On Jun 8, 2020, at 1:37 PM, Francis Herne wrote: > > On Friday, 5 June 2020 17:23:37 BST David M. Cotter wrote: >>>>> What table view are you talking about, specifically? >>>> >>>> QTableView with QSortFilterProxyModel >>> >>> What is the model _behind_ QSFPM? >> >> i'm trying to understand your question. i'm not sure what you mean, >> cuz i'm not the SQLite dev what i know is that each song has a couple >> dozen bits of data (name, artist, album, several file paths, rating, >> duration etc etc) besides that, what are you asking?= > > QSortFilterProxyModel is a *proxy* that makes other models sortable. > it only works when its `sourceModel` property is set to an instance of > a different model class that actually contains the data. ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] how do I load massive table views instantly?
right okay so my source model class is QSqlRelationalTableModel, no subclass. does this answer your question? sorry i'm still learning about this stuff. now that you know that, is it easier to answer my original question? > On Jun 8, 2020, at 1:37 PM, Francis Herne wrote: > > On Friday, 5 June 2020 17:23:37 BST David M. Cotter wrote: > What table view are you talking about, specifically? QTableView with QSortFilterProxyModel >>> >>> What is the model _behind_ QSFPM? >> >> i'm trying to understand your question. i'm not sure what you mean, cuz i'm >> not the SQLite dev what i know is that each song has a couple dozen bits of >> data (name, artist, album, several file paths, rating, duration etc etc) >> besides that, what are you asking?= > > QSortFilterProxyModel is a *proxy* that makes other models sortable. it only > works when its `sourceModel` property is set to an instance of a different > model class that actually contains the data. ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] how do I load massive table views instantly?
On Friday, 5 June 2020 17:23:37 BST David M. Cotter wrote: > >>> What table view are you talking about, specifically? > >> > >> QTableView with QSortFilterProxyModel > > > > What is the model _behind_ QSFPM? > > i'm trying to understand your question. i'm not sure what you mean, cuz i'm > not the SQLite dev what i know is that each song has a couple dozen bits of > data (name, artist, album, several file paths, rating, duration etc etc) > besides that, what are you asking?= QSortFilterProxyModel is a *proxy* that makes other models sortable. it only works when its `sourceModel` property is set to an instance of a different model class that actually contains the data. ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] how do I load massive table views instantly?
>>> What table view are you talking about, specifically? >> QTableView with QSortFilterProxyModel > > What is the model _behind_ QSFPM? i'm trying to understand your question. i'm not sure what you mean, cuz i'm not the SQLite dev what i know is that each song has a couple dozen bits of data (name, artist, album, several file paths, rating, duration etc etc) besides that, what are you asking? > Does that model support async loading? okay, how would i answer this question? i guess ask the developer of the DB > Note that QSFPM may still need to load the entire source model (didn't check > its source code) if you ask it to filter or sort. For 200k elements, I'd > expect the source model itself to support filtering and sorting, without the > need of a proxy in front of it. would this allow me to have multiple playlists, each with it's own view into the model, with different columns showing in different orders, different sets of songs, possibly with a filter? and switch between them instantly? ie: not have to take time to switch, to show the first page of data? and allow the scroll thumb to instantly be correctly sized, such that the user can take the thumb and drag it to the bottom, and instantly see the last page? no waiting? thanks, i'm a newbie here in terms of SQLite and how it all works, and i'm speaking on behalf of the dev, so i only know JUST enough to ask maybe dumb questions but not enough to answer perhaps some smart ones... sigh -dave ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] how do I load massive table views instantly?
Il 04/06/20 23:31, David M. Cotter ha scritto: What table view are you talking about, specifically? QTableView with QSortFilterProxyModel What is the model _behind_ QSFPM? Does that model support async loading? Note that QSFPM may still need to load the entire source model (didn't check its source code) if you ask it to filter or sort. For 200k elements, I'd expect the source model itself to support filtering and sorting, without the need of a proxy in front of it. My 2 c, -- Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer KDAB (France) S.A.S., a KDAB Group company Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com KDAB - The Qt, C++ and OpenGL Experts smime.p7s Description: Firma crittografica S/MIME ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] how do I load massive table views instantly?
On 6/5/20 5:00 AM, interest-requ...@qt-project.org wrote: alternately, if "loading only what you see" is just "how it's done", then can i just set the scroll bar height to be the total height? so CAN still grab the scroll thumb, and drag it to the end, and see the data at the end, without waiting for all the unseen data to load? i know other apps do this... how is it done? I believe what you are trying to do is covered in "The Minimum You Need to Know About Qt and Databases." https://www.theminimumyouneedtoknow.com/qt_book.html You can see if WorldCat shows a university with a copy you get check out. https://www.worldcat.org/ They've done something to their search engine recently though because it doesn't show you all of the libraries anymore and it doesn't find all of the titles easily. There is a short (57 page) preview of the book on ScribD. https://www.scribd.com/document/156381069/The-Minimum-You-Need-to-Know-About-Qt-and-Databases This was written with an early version of Qt 4.x and there were bugs in QSqlDatabase it had to work around. On the page for the book is a download link to the source code for the book if you just want to muddle through. You should find what you are looking for there though. I distinctly remember doing it. I've had to do it at many sites. -- Roland Hughes, President Logikal Solutions (630)-205-1593 http://www.theminimumyouneedtoknow.com http://www.infiniteexposure.net http://www.johnsmith-book.com http://www.logikalblog.com http://www.interestingauthors.com/blog ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] how do I load massive table views instantly?
> 1) quickly calculate/get the total number of items and return that value from > rowCount() to let views properly configure their scroll bars; is that a method one overrides? where is "rowCount()" defined? do i override it by subclassing QSortFilterProxyModel? > 2) load data in the background; i think this is already happening > 3) return some placeholder values from data() for items that have not been > loaded yet; > 4) when a new batch of items is available, add it to the model's internal > storage and emit the dataChanged() signal with appropriate index range. wouldn't the view ask the model for whatever data is revealed as visible, all by itself? so no need to show mock data? the point is i want to show the real data, even if the user jumps to the last rows immediately do i *need* to manually do this? thanks for the suggestions, i'm trying to understand -dave___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] how do I load massive table views instantly?
Here is my idea: 1) quickly calculate/get the total number of items and return that value from rowCount() to let views properly configure their scroll bars; 2) load data in the background; 3) return some placeholder values from data() for items that have not been loaded yet; 4) when a new batch of items is available, add it to the model's internal storage and emit the dataChanged() signal with appropriate index range. On 06/05/2020 12:39 AM, David M. Cotter wrote: >> What did you try so far? > the other thing tried was showing the table without pre-loading it, but then > loading the rest in the background > > but this has the problem that the user can't then pick up the scroll thumb > and drag it to the bottom, because not all the data is loaded. as soon as > the user lets go, more data keeps getting loaded and the scroll thumb slides > out from under their mouse as more data comes in. > > uck > > :) signature.asc Description: OpenPGP digital signature ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] how do I load massive table views instantly?
> What did you try so far? the other thing tried was showing the table without pre-loading it, but then loading the rest in the background but this has the problem that the user can't then pick up the scroll thumb and drag it to the bottom, because not all the data is loaded. as soon as the user lets go, more data keeps getting loaded and the scroll thumb slides out from under their mouse as more data comes in. uck :) ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
Re: [Interest] how do I load massive table views instantly?
>> alternately, if "loading only what you see" is just "how it's done", then
>> can i just set the scroll bar height to be the total height? so CAN still
>> grab the scroll thumb, and drag it to the end, and see the data at the end,
>> without waiting for all the unseen data to load?
>> i know other apps do this... how is it done?
>
> What table view are you talking about, specifically?
QTableView with QSortFilterProxyModel
> What did you try so far?
when showing the view for the proxy model:
while (modelP->canFetchMore(modelIndex)) {
modelP->fetchMore(modelIndex);
if (dlgP) {
dlgP->Inc(false);
}
}
but that is way slow
i'd rather just quick-calc the number of items and set the scroll bar to max at
the start
then let the viewer fetch what it needs to when the rows get revealed?
is this possible?
or can i pre-load the proxyModel at startup and just "swap it in" when the user
switches playlists?
-dave___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest
Re: [Interest] how do I load massive table views instantly?
Il 04/06/20 21:12, David M. Cotter ha scritto: alternately, if "loading only what you see" is just "how it's done", then can i just set the scroll bar height to be the total height? so CAN still grab the scroll thumb, and drag it to the end, and see the data at the end, without waiting for all the unseen data to load? i know other apps do this... how is it done? What table view are you talking about, specifically? What did you try so far? -- Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer KDAB (France) S.A.S., a KDAB Group company Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com KDAB - The Qt, C++ and OpenGL Experts smime.p7s Description: Firma crittografica S/MIME ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
[Interest] how do I load massive table views instantly?
say i have a few "playlists" that are really big, like 200k items, with many columns i'd like to be able to switch to them and have them be instantly loaded. by that i mean the entire list, such that i can take the scroll thumb and drag it to the bottom and see the last item, ie: no "loading by pages" alternately, if "loading only what you see" is just "how it's done", then can i just set the scroll bar height to be the total height? so CAN still grab the scroll thumb, and drag it to the end, and see the data at the end, without waiting for all the unseen data to load? i know other apps do this... how is it done? -dave ___ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
