The patch attached speeds up handling of datasets with "invalid" components. With it, the DXcallm terrain app I'm working with using a sample dataset (1.7M pts; 5% invalid) takes 43% less time to run (down from 114 sec).
The problem: Determining a position's/connection's "invalid" state is
hideously slow when the invalids are maintained in an IC_SORTED_LIST
InvalidComponentHandle and the number of invalids is large. Determination
equates to a binary search, so e.g. 94k invalids yields up to ~17 search
levels. About ~40% of total runtime here was spent in the trivial
IsElementMarked function doing these searches. Note that building the
connections component (via DXInvalidateConnections) touches almost all
positions 4 times for a 2D field, so 1.7M pts -> 6.8 million binary
searches. If searches aren't cheap you can see how it kills performance.
DX's prior heuristic for retrieving invalids for processing
was to keep ref invalid components (sorted lists) as IC_SORTED_LISTs. It
original wrote a sorted list only if num invalids > 20% total num values.
For large datasets, 20% can be quite large and accessing these SORTED lists
is expensive.
The patch: The patch changes this so that two thresholds are used: one
for storage in an Array (20% of total values, as before) and another for
building the InvalidComponentHandle for transient processing (currently
1024); small size for storage and faster access for performance. SORTED
lists are always converted to HASHs or DEP_ARRAYs now for processing (as
they were when originally created), not left as SORTED_LISTs. The patch
also removes a few misc unused variables the compiler noticed.
What this patch does not do: address the repeated recalculation of
"invalid ..." components computed by Include, subsequently thrown away by
Include, and then recalculated by other modules downstream (e.g. Map,
Render, etc.).
Randy
--
Randall Hopper (mailto:[EMAIL PROTECTED])
Lockheed Martin Operation Support
EPA Scientific Visualization Center
US EPA MD/24 ERC-1A; RTP, NC 27711
invalid-perf.final.patch.gz
Description: Binary data
