Re: [Interest] Small survey on necessary Qt Container size

[Roland Hughes via Interest]

On 9/28/22 05:00, Ulf Hermann wrote:

I think Andre's point is that you should not use Qt containers for such
large amounts of data, but rather some other data structure better
suited for your case (most trivially, std::vector instead of QList).


My point was you don't get a choice given the single-thread-i-ness of Qt 
and trying to stuff everything in the main event loop.


https://doc.qt.io/qt-5/qtablewidget.html

int *currentRow *() 
const


https://doc.qt.io/qt-5/qsqltablemodel.html

virtual int *rowCount 
*(const QModelIndex 
&/parent/ = QModelIndex()) const override


When your database has segments and each segment can have a ROWID 0 - 
ULLONG_MAX, honestly, even ULONG_MAX gets you in trouble when you have 
multiple segments in your return query.


Somewhere in all of that code is a "container." Not every database 
engine will allow one to move bidirectionally through a cursor. If you 
want to support a user being able to drag the scrollbar all the way to 
the bottom (or just clicking there) then going back to the first row, 
the data is either loaded into a container or the cursor must have 
bidirectional capabilities.


I could be wrong, but I believe part of Quentin's point of stuffing 
everything into RAM is the fact sliding cursor windows aren't 
implemented and many database engines don't support bidirectional 
movement. Leaving data in a cursor allows the OS and database engine to 
determine what should be "in RAM" without having to do all of the I/O.


If one is stuck with low end databases and tools, then one has little 
choice, they have to do the I/O and stuff all that data into RAM. This 
is especially true if you are making a decision spread across multiple 
databases from multiple vendors. The capabilities of Qt don't quite 
measure up to the 1980s product Cognos PowerHouse


https://www.teamblue.unicomsi.com/products/powerhouse-4gl/

which would let you pull a cursor from say, Oracle, and drive it against 
an indexed file or one of the other supported database engines. I think 
you could have up to seven different database engines in a single 
transaction, but that may be an outdated number. It never loaded that 
stuff into RAM more than a few records at a time. I know this because 
the databases I was using were many times the size of physical + virtual 
memory. It all "just worked."


People don't want data dictionaries anymore, but they all want the data 
capacity and throughput.


Yes, developers can customize their own QSQLTableModel to periscope over 
the cursor, and force bidirectional in, but they shouldn't have to in 
2022. The underlying code needs to also realize that ROWID is no longer 
a table unique number. It is __segment__ unique. I forget the official 
name for the groups of segments, but there is one.


--
Roland Hughes, President
Logikal Solutions
(630)-205-1593

http://www.theminimumyouneedtoknow.com
http://www.infiniteexposure.net
http://www.johnsmith-book.com
http://www.logikalblog.com
http://www.interestingauthors.com/blog

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Small survey on necessary Qt Container size

[Ulf Hermann]

I think Andre's point is that you should not use Qt containers for such 
large amounts of data, but rather some other data structure better 
suited for your case (most trivially, std::vector instead of QList).


The implicit sharing of Qt containers is a nice trait that enables you 
to write more concise code as long as it's feasible to copy the data 
when needed. If that is not feasible (i.e. billions of elements), the 
equally implicit detach on write will come back to bite you.


However, I guess there are a lot of people who consciously or 
unconsciously ignore this particular bit of advice and "just" never copy 
a large container unless all of its copies remain const forever. Is that so?


Ulf
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Small survey on necessary Qt Container size

[qt . dantec]

Le Mon, 5 Sep 2022 19:25:59 +0200, A. Pönitz 
écrivait:

>  How often do people /need/ /Qt/ containers with /more than
>  1 billion elements/ ? 

Anybody amongst the fast growing community doing big data, a field no
longer reserved to big corporations :
- Small teams doing stock market research, where the number of
transactions in a single day far outnumbers your figures: Doing proper
backtesting in a reasonable amount of time makes it imperative to
store data in Ram ;
- Research teams trying to discover unknown correlations between
already approved medicine and pathologies for which no effect is yet
documented.

The list could go on forever. The point is that thank's to Internet
and high bandwidth optical lines, more and more people have access to
large datasets, and most of them cannot afford access to mainframes.
Hence the necessity to store as much data as possible in central
memory before analysis.

Quentin.
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Small survey on necessary Qt Container size

[Roland Hughes via Interest]

On 9/7/22 05:00, Thiago Macieira wrote:

On Tuesday, 6 September 2022 06:24:19 PDT Michael Jackson wrote:

My guess is that if you are NOT in the data processing realm the Qt
containers are probably just fine for your use cases.

I don't know why Andr? decided to ask about Qt containers, since they already
do support 64-bit sizes.

The discussion on the development mailing list is not about the low-level
containers, but about the GUI types.


Probably because the widget must contain the data in something, but that 
is just a guess.


What isn't a guess is your row numbers need to be BASE-26 for table widgets.

60,000,000,000

is going to chew up quite a bit of screen for a line number. (That's 
sixty Billion for those counting commas.)


Honestly, it is too late for Qt to find its way into corporate desktop 
applications. There are legacy applications like Wireshark


https://github.com/wireshark/wireshark

that will be maintained until someone ports them to CopperSpice, LVGL, 
Elementary, etc. Same with Geany and a few others. Given the licensing 
shenanigans, elimination of OpenSource LTS, and severely limited data 
capabilities, it will never again be the tool of choice for new 
corporate or embedded development. Even basic order entry systems must 
have the capability to reference a data warehouse so the customer can 
look up their past orders.


Despite offering boot2qt, all of the major SOM module makers have been 
steering customers away from that.


So, I guess the real question, before one gets to "capacity" is

what market hasn't Qt gotten itself thrown out of yet?

That will determine how much capacity your widgets and containers must have.

If you (being Qt) could ___directly__ access the full capabilities of 
Mumps and PICK BASIC databases you "could" become the tool of choice for 
medical records systems development. You would have to do it very 
quickly though. Almost every major hospital system is paying to keep 
Windows 7 alive so they can avoid a very expensive Epic/etc. software 
upgrade combined with a Windows 10/11 upgrade.


https://www.epic.com/software

Most hospitals would love to be able to leave their existing Dell 
all-in-one systems in place, load Manjaro (or some other somewhat secure 
Linux) on them and load high quality medical records software that could 
use the existing database sans conversion. Despite what MS says publicly 
about 2023 being the end of the end, hospitals will be able to get 
another 2-5 years as will federal offices and they will all ride it out 
to the bitter end.


Licensing shenanigans got Qt tossed out of the embedded world and the 
phone market.


Toyota threw them out of the automotive world.

https://www.reuters.com/article/us-toyota-tech-idUSKBN18R1CW

Even if Qt was absolutely free, draconian size limitations mean it 
cannot be used to access corporate data so cannot be used for corporate 
desktops.


What market is Qt actually targeting now? That determines what the 
capabilities need to be. Given the licensing, it will have to be an 
incredibly obscure niche with deep pockets. Nothing else will pay those 
fees.


--
Roland Hughes, President
Logikal Solutions
(630)-205-1593

http://www.theminimumyouneedtoknow.com
http://www.infiniteexposure.net
http://www.johnsmith-book.com
http://www.logikalblog.com
http://www.interestingauthors.com/blog

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Small survey on necessary Qt Container size

[Thiago Macieira]

On Tuesday, 6 September 2022 06:24:19 PDT Michael Jackson wrote:
> My guess is that if you are NOT in the data processing realm the Qt
> containers are probably just fine for your use cases.

I don't know why André decided to ask about Qt containers, since they already 
do support 64-bit sizes.

The discussion on the development mailing list is not about the low-level 
containers, but about the GUI types.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Cloud Software Architect - Intel DCAI Cloud Engineering



___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Small survey on necessary Qt Container size

[Michael Jackson]

We write science data processing software 
(github.com/bluequartzsoftware/dream3d) and there have been more than a few 
times where we load up past 32 bit signed int number of elements in an array. 
We are adding out-of-core to our software now which means we are definitely 
going past signed 32 bit indexes. We rewrote our core software library to 
remove any Qt codes from them so that we can have this kind of access using 
std::* containers.

My guess is that if you are NOT in the data processing realm the Qt containers 
are probably just fine for your use cases. If you need something more you have 
already made the changes necessary to move your code forward. 
Std::shared_ptr> also works well to pass around data as does the 
usual "const ref" passing which helps you keep ownership figured out.

Just my 2 cents.

--
Mike Jackson


On 9/5/22, 13:56, "Interest on behalf of A. Pönitz" 
 wrote:


Hi all.

In order to base my potential arguments in a certain discussion on a
neighboring list not purely just on my own experience with both Qt and
(independently) large data (a.k.a. "gut feeling") I would appreciate
if /you/ could help me to find a - very rough, readily admitted in
advance to be completely unscientific - estimation for an answer to
the following question:

  How often do people /need/ /Qt/ containers with /more than
  1 billion elements/ ? 

The question is really meant as conjunction, i.e. I'd like to count
only setups meeting both criteria at the same time:

   1. /Some/ relevant data set is really > 1e9 entries,

   2. It really needs to be a /Qt/ container because of some Qt container
  feature (e.g. reference counting or e.g. because of Q(5)List's
  "indirect" storage etc) that a std:: container does not offer
  out-of-the-box and it /needs/ to be like that, i.e. there is no
  simple / straightforward replacement like using std::*, and
  benefits do not exceed drawbacks like more expensive write
  access.

Problems meeting only one of the two criteria won't count, but I'd
accept "Would have needed in the past, but since Qt < 6 didn't offer
it I had to go through big trouble and create a hand-crafted solution"
;-)

In case of proprietary/personal/otherwise non-disclosable examples
I'd be happy enough to get a private mail. In any case: Any feedback
is appreciated.

Andre'
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest


___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Small survey on necessary Qt Container size

[Roland Hughes via Interest]

On 9/6/22 05:00, A. P?nitz wrote:

The question is really meant as conjunction, i.e. I'd like to count
only setups meeting both criteria at the same time:

1./Some/  relevant data set is really > 1e9 entries,

2. It really needs to be a/Qt/  container because of some Qt container
   feature (e.g. reference counting or e.g. because of Q(5)List's
   "indirect" storage etc) that a std:: container does not offer
   out-of-the-box and it/needs/  to be like that, i.e. there is no
   simple / straightforward replacement like using std::*, and
   benefits do not exceed drawbacks like more expensive write
   access.


On 9/6/22 05:00, Thiago Macieira wrote:


Or foresee needing them in the next 10 years.


For applications contained entirely on the 
x86-wanna-be-a-real-computer-one-day-when-I-grow-up platform, only once.


For applications that would run on above as front ends for systems 
hosted on real computers, I've lost count. Qt simply isn't even 
considered anymore. Given that Ford is laying off 8,000 and has dumped 
manufacturing cars, Qt might want to consider the real computer market.


https://www.foxbusiness.com/economy/ford-lay-off-8000-workers-shift-ev-production

They also might want to consider making all of the QSqlTable based stuff 
operate outside of the main event loop while only nibbling data from 
cursors. Dumping the failed methodology of MVVM in favor of recreating 
the proper cursor.


Of the database engines listed for 6.x, only PostgreSQL is actually used 
for data warehousing, at least by companies that are smart enough to not 
put their critical data in the cloud. DB2 and the rest are used for 
PRODUCTION databases. They get trimmed. Despite MariaDB touting itself 
as a Data Warehousing product, I've never seen anyone serious use it for 
that.


https://mariadb.com/kb/en/data-warehousing-techniques/

Oracle gets used in many and was the first to have a database where 
tables could exceed 2TB in size. It was built for Boeing. Informix has 
long been used for Data warehousing and IBM now has Netezza appliances.


The x86-wanna-be world is oohing and aahing over Mongo and NoSQL. The 
Fortune 1000 world is dusting off PICK BASIC and Mumps


https://www-50.ibm.com/partnerworld/gsd/solutiondetails.do?solution=49945=true=en

Look under the hood of CoudAnt some time.

At any rate,

A really big software development market right now is in creating front 
ends and reporting tools for locally hosted data warehouses. Because of 
the SQL classes needing to exist in the main event loop and because 
response time from a data warehouse having hot and cold storage could be 
minutes, Qt cannot be chosen. Good thing, because with Qt's size limits, 
it also cannot be chosen.


Real requirement:

Keller MBA graduate wants a part evaluation spreadsheet (they always 
want a spreadsheet) where they can choose a part number, and have the 
spreadsheet filled in with every order ** for the entire part 
supercession chain **


The 380AMP alternator they make today will fit any truck they made from 
1970 through to today. The part number will have changed hundreds of 
times (and is stored in a circular referencing table because you don't 
get rid of trapped inventory by deleting a supercession, you superceed 
back to the original part number)


In their spreadsheet they want to see every dealer order for a 380AMP 
alternator from the beginning of time. Then they want little check boxes 
and radio buttons so they can weed out (or select) based on warranty or 
"just an order" or, I forget which else.


Last I heard, the data warehouse table for parts order detail lines had 
over 30 billion rows. That was close to ten years ago.


Twenty years ago Boeing had to solve the same problem for airplane parts.

As of right now, Informix can store a maximum of 4,278,189,825 rows in 
each table fragment and a maximum of 2047 fragments in a single table. 
ROWID is only unique within a fragment.


https://www.ibm.com/support/pages/what-maximum-number-pages-table-or-fragment-what-maximum-number-rows-page

The XPS version of Informix allows 32766 fragments and ROWID is only 
unique within a fragment.


How many of you currently have Qt applications that rely on ROWID being 
unique???



If Qt wants to find a market, now that the automotive market has 
disappeared and phones have turned their back on Qt it has to become the 
front end that can communicate with what the real computers can throw at it.


**You can't fit everything into RAM and you have to do I/O outside of 
the main event loop **



Once you achieve that you should develop classes for the M database, 
both OpenSource and the one from InterSystems.


https://en.wikipedia.org/wiki/MUMPS

https://www.intersystems.com/

The largest medical records systems in America, and possibly the world, 
use the Mumps database. Right now they have one choice for GUI 
development tools. Medical records systems are not going away.


Then you can 

Re: [Interest] Small survey on necessary Qt Container size

[Philippe]

On Mon, 5 Sep 2022 19:25:59 +0200
A. Pönitz  wrote:

>   How often do people /need/ /Qt/ containers with /more than
>   1 billion elements/ ? 

On one hand, I could say anyone that needs it once in a while could
simply use a std container.

On the other hand, for the "container of bytes" QByteArray, one could
imagine more potential cases.

I would like to reiterate something I mentioned a few years ago:
for coherence, values such as "QAbstractSlider::value"
should also be qsizetype and not int.

To scroll through a view of data (not necessarily in memory), it makes perfect 
sense to have a 64-bit range.
Actually, it makes more sense than 64-bit containers... Hopefully for Qt7.
(I have had this particular need for 10 years actually...)

Philippe

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Small survey on necessary Qt Container size

[Elvis Stansvik]

Den mån 5 sep. 2022 kl 22:30 skrev Thiago Macieira :
>
> On Monday, 5 September 2022 10:25:59 PDT A. Pönitz wrote:
> >   How often do people /need/ /Qt/ containers with /more than
> >   1 billion elements/ ?
>
> Or foresee needing them in the next 10 years.

0 times and 0 times.

I've used Qt (non)professionally on and off for 15 years.

That said, don't want my disclosure of these numbers to construed as
me taking a side in a Qt + size_t discussion - I just don't feel like
I'm read up enough on the matter. But there you go :)

Elvis

>
> --
> Thiago Macieira - thiago.macieira (AT) intel.com
>   Cloud Software Architect - Intel DCAI Cloud Engineering
>
>
>
> ___
> Interest mailing list
> Interest@qt-project.org
> https://lists.qt-project.org/listinfo/interest
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Small survey on necessary Qt Container size

[Thiago Macieira]

On Monday, 5 September 2022 10:25:59 PDT A. Pönitz wrote:
>   How often do people /need/ /Qt/ containers with /more than
>   1 billion elements/ ?

Or foresee needing them in the next 10 years.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Cloud Software Architect - Intel DCAI Cloud Engineering



___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest