David L. Thompson: |Randall, | |I'm about to submit a big chuck of cleanup work against dpexec and |was thinking about merging in this patch very soon afterwords. But I |wanted to know a few things before I did.
Ok. I've attached the latest patches for the locking and for the modpath
memory optimization pieces (for dx-4.1.0). (In a fresh rebuild this
morning, I discovered I'd left a file out of the latter in my original
dx-dev post.) These two buy me a 45% speedup on 1-proc cached executions.
If you have trouble applying them, let me know and I can update them for a
recent CVS (I'd do it now but it's a bit of a pain with the EPA firewall).
|In the What's left area. Will this patch create any problems that
|don't already exist with the older code?
I don't believe so. The only possibility that comes to mind is that if
someone has misused the DXGetModuleId interface and presumed that it
returns strings of a particular form (/mod:inst#/...), then that trick
wouldn't work anymore. However as mentioned in the DX code, this practice
would be wrong. Also, the old string form isn't documented in the official
DX docs for DXGetModuleId(), so there shouldn't be any problem.
An integration issue: As I mentioned in the "What's Left" section, the
symbol table used for caching module/macro names is currently global (see
the static "mod_name_str_table" in graph.c). It contains the DX built-in
module names plus any new ones it finds that are used as module and macro
names in executed networks.
In practice this list will be fairly small in terms of memory. However, it
is still a memory leak that'll show up in purify. It should be attached to
some data structure associated with the referring network, but I didn't
know where was best to put it.
|Do you see any other problems with the big networks you've been working
|on?
None so far.
|How about further ideas on problems that might arise (would this cause
|any problems during debug execution in the log window of the VPE {to turn
|on go to the Message Window and select Debug Tracing from the Commands
|menu})?
This works fine (just tried it). The conversion to the human-readable form
is handled in evalgraph.c in the execGnode(), ModBeginMsg(), and
ModEndMsg() by the _dxf_ExGFuncPathToString() calls.
Let me know if you need any help or have questions.
Thanks,
Randall
--
Randall Hopper (mailto:[EMAIL PROTECTED])
EPA Scientific Visualization Center
US EPA MD/24 ERC-1A; RTP, NC 27711
|David
|
|> This patch optimizes DX memory handling, primarily with regards to
|>module paths. With it applied to DX, I get a 25% speed boost on cached net
|>executions. I'd appreciate it if the CVS heads would review this patch and
|>see if there's commit potential here.
|>
|> Further dxexec profiling revealed that dxexec spends much of its time
|>on cached executions regrowing module path strings, one string path
|>component at a time. For the network I'm working with, dxexec calls memory
|>allocation routines over 1,000,000 times each cached execution for module
|>path construction alone.
|>
|> This patch changes the representation of module paths to remove most
|>of this overhead. Module paths are now represented as a list of
|>Module/Instance number pairs in binary form. Building and augmenting
|>module paths in this form is fast and requires no dynamic memory, once the
|>module and macro names are cached. Converting these paths to string form
|>for debug prints, module ID strings, etc. is also fast.
|>
|>OVERVIEW:
|>
|> Sample module path: /main:0/BoxAndWhiskerPlot:1/BoxAndWisker:2/Compute:5
|> Old representation:
"/main:0/BoxAndWhiskerPlot:3/BoxAndWisker:4/Compute:5"
|> New representation: (0,0),(1,3),(2,4),(3,5)
|>
|> What DX uses module paths for:
|> - Cache tags
|> - Module paths on run-time gfuncs
|> - DX debug/warning/error prints
|> - Module "begin" messages to dxui
|> - Comparison of module names and macro execution paths
|> - Names of ASYNC_LOCAL variables
|> - Module ID strings
|>
|> Patch overview
|> - New ModPath struct defined in graph.h
|> - Replaces cpath and path in both graph.h's gfunc struct and cache.h's
|> pathtag struct with associated code updates
|> - Also, pre-grows some frequently built lists in the graph execution
code
|>
|> What's left
|> - The Macro/Module Name symbol table is currently global.
|> That's probably not ideal; it should be associated with the network.
|>
|> (What's a good spot in the data structures though? on Program?)
|> - The error handling calls (ExDie, etc.) might need changed
|>
|>This patch is against the 4.1.0 source. It applies cleanly to CVS except
|>for one small piece which is easy to hand-merge. However, I didn't send a
|>patch against CVS since it is a bit broken for IRIX now, and this patch
|>isn't completely ready for commit.
|>
|>Thanks,
|>
|>Randall
|>
|>--
|>Randall Hopper (mailto:[EMAIL PROTECTED])
|>EPA Scientific Visualization Center
|>US EPA MD/24 ERC-1A; RTP, NC 27711
|--
|.............................................................................
|David L. Thompson The University of Montana
|mailto:[EMAIL PROTECTED] Computer Science Department
|http://www.cs.umt.edu/u/dthompsn Missoula, MT 59812
dx-4.1.0-locks-optimize.patch.gz
Description: Binary data
dx-4.1.0-modpath-optimize.patch.gz
Description: Binary data
