On Mon, Feb 3, 2014 at 2:40 PM, Gustaf Neumann <neum...@wu.ac.at> wrote:
>
> converting ns-sets naively into local variables will clobber other locals,
> using some strange array names etc. would still require to modify
> all functions accessing the ns-sets, or passing the ns-sets around etc.
>
db_foreach already unpacks it's row sets into local variables.
No, this:
>
> set rows [dbi_rows -columns cols -- $sql]
> foreach d [dbi_dicts $cols $rows] {
> puts [dict get $d k]
> }
>
> ...is faster and uses less memory than this:
>
> foreach row [dbi_rows -result dicts -- $sql] {
> puts [dict get $row k]
> }
>
> No, its not. The data duplication and double iteration are still needed
> in your suggested approach. Due to the lazy dict conversion there
> might be many cases, where not all dicts are created.
> But even, when all lists are converted to dicts, you suggestion is slower.
> Look at the following tests:
>
> p1: dbi_rows returns a flat list, convert the flat list to a list of
> lists, convert lazy to dicts
> p2: dbi_rows returns a flat list, convert the flat list to a list of
> dicts, use the dicts
> p3: dbi_returns a list of lists, convert lazy to dicts
>
> The times for retrieving 1000 values with 14 columns via SQL and to
> process as indicated are
> (micro seconds average of 100 runs).
>
> p1 35692
> p2 45767
> p3 29407
>
> The experiment indicates, that p2 = p3 * 1.56.
> The built-in solution (p3) is much faster, appending to the list and
> convert lazy to the dict (p1) is faster than straight dict creation (p2)
>
All you're measuring here is 14,000 extra calls to the dict command in p2
because it's written in Tcl not C.
> If we look at ns_sets, we see that ns-sets better than some people
> might suggest:
>
> p4: dbi_rows returns a flat list, convert the flat list to a list of
> ns-sets, access all columns in the loop
> p5: dbi_returns a list of lists, convert rows to ns-sets, access all
> columns in the loop
> p6: dbi_returns a list of ns-sets, access all columns in the loop
>
> p4 41607
> p5 25508
> p6 16095
>
> Interestingly, p6 is the fastest of all tests, and p4 is the slowest of
> the ns-set tests
> (p4 = p6 * 2.59). also unexpected to me, p5 is faster than p3, since the
> list-of-lists is converted to an ns-set, which is faster than the dict
> creation.
>
Similar problem. p4 is slower than p5 because because there's an
intermediate step coded in Tcl.
p5 is faster than p3 because an ns_set is faster to construct: no hash
table, no Tcl objects.
However, although the comments say "access all columns in the loop" the
test code shows only one column is accessed (object_id, which might well be
the first column). The ns_set uses a linear scan to find keys so I would
expect that the more lookups, the more columns, the slower sets will be
compared to dicts.
>
> These results can be made more similar by coding the flat-to-struct
> statements
> in C, but i see no indication that the eager dict generation will be
> better than
> the list operations used now.
>
Without a C implementation only Tcl command dispatch overhead is being
tested.
> in our current production environment (with ns_db) requests to the
> application server require on average about 9 SQL queries
> (but up to *700* SQL queries uncached for the personalized start
> page).
>
This is what happens when you make it easy to nest db_foreach :-)
If you could get rid of that, you could reduce the number of database
backends, reduce contention, and use some of the saved memory for more
buffer cache, or extra indexes.
------------------------------------------------------------------------------
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
_______________________________________________
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel
