Well, I would not place too much value on my interpretation of the profile,
but here is the sorted list of calls that totalled over 1 second. The total
within sqlite.py is less than 20 seconds. It spent longer than that just
converting to and fro between C and F
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
5717318 127.420 0.000 477.916 0.000 xtypes.py:76(get_scalar)
34303908 81.864 0.000 81.864 0.000 xtypes.py:27(get_scalar)
5717318 40.281 0.000 217.476 0.000 wxxtypes.py:80(get_scalar)
5718892 35.100 0.000 66.585 0.000
manager.py:503(genBatchRecords)
5718892 31.380 0.000 31.485 0.000 manager.py:463(genBatchRows)
1582 30.922 0.020 575.233 0.364 xtypes.py:913(get_aggregate)
5727031 26.915 0.000 40.451 0.000 units.py:534(__new__)
2575348 24.718 0.000 42.936 0.000
wxformulas.py:122(windchillMetricWX)
2575348 24.592 0.000 44.146 0.000 wxformulas.py:221(heatindexC)
2575348 19.911 0.000 80.997 0.000
wxxtypes.py:157(calc_windchill)
2575348 19.714 0.000 81.975 0.000
wxxtypes.py:174(calc_heatindex)
5729946/5727456 14.264 0.000 14.315 0.000 {built-in method
builtins.getattr}
5736977/5736604 13.563 0.000 13.597 0.000 {built-in method
__new__ of type object at 0x9584a0}
1161605 13.158 0.000 35.678 0.000 sqlite.py:36(guarded_fn)
5150696 12.203 0.000 12.203 0.000 units.py:47(CtoF)
5151122 12.065 0.000 12.065 0.000 units.py:50(FtoC)
577121 8.167 0.000 8.167 0.000 {function
guard.<locals>.guarded_fn at 0x7fe282fe5120}
2575348 7.448 0.000 7.448 0.000 wxformulas.py:144(heatindexF)
2575348 6.056 0.000 6.056 0.000 wxformulas.py:73(windchillF)
575069 5.854 0.000 42.773 0.000 manager.py:565(getSql)
566622 5.294 0.000 51.152 0.000 wxxtypes.py:287(get_scalar)
1782028 4.221 0.000 4.221 0.000 {method 'startswith' of 'str'
objects}
577121 3.118 0.000 11.285 0.000 sqlite.py:231(execute)
577121 2.794 0.000 4.404 0.000 sqlite.py:146(cursor)
577121 1.610 0.000 1.610 0.000 {method 'cursor' of
'sqlite3.Connection' objects}
585132 1.403 0.000 1.403 0.000 {method 'lower' of 'str'
objects}
577121 1.367 0.000 1.367 0.000 {method 'close' of
'sqlite3.Cursor' objects}
571755 1.339 0.000 1.339 0.000 {method 'endswith' of 'str'
objects}
7691 0.526 0.000 1.099 0.000 xtypes.py:1070(get_aggregate)
On Friday 29 December 2023 at 1:00:00 am UTC+10 Tom Keffer wrote:
> I wonder if this could be organized into
>>
>> a database accessor to get multiple columns, that results in a stream
>> of N values
>>
>> functions that accept N values to compute something else
>> (e.g. windchill from temp/wind).
>>
>> a function that takes a computing function and a stream
>>
>
> This is pretty much what we have now.
>
> dbmanager is the first;
> xtypes.get_scalar() is the second;
> XTypeTable.get_series() is the third.
>
> What's missing is the ability to specify the N observation types. However,
> one could just assume them all. That is, XTypeTable.get_series() would just
> ask for all the data for a year, then call get_scaler(). Of course, that
> might require a lot of memory. You could relax that requirement a bit by
> doing the calculation in tranches --- perhaps a month's worth of data at a
> time --- then stitch together the pieces.
>
> Cameron: your comment that a savings of 100x would be needed for this to
> be useful is about right. But, database accesses are *very* expensive. I
> think you'd be surprised by the savings of reducing the number of db calls
> from 100k to one.
>
> -tk
>
--
You received this message because you are subscribed to the Google Groups
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-development/6ec35cae-142c-45f0-9b9d-b7531cf9bf37n%40googlegroups.com.
