In the process of investigating the TODO idents problem, I created a
TGM with 50,000 regions. I used a 64-bit, 4-cpu Intel PC, with 8 Gb
of RAM, running 64-bit Debian 6, and a disk partition with 184 Gb
free space.
I created the TGM by first generating an "old style" script to feed to
mged. The first few lines look like this:
killtree all
units mm
in s0.s sph 0 0 0 5
r s0.r u s0.s
g all s0.r
in s1.s sph 0 0 0 5
r s1.r u s1.s
g all s1.r
Then, in the trunk build directory, I created the TGM with:
$ time ./bin/mged -c idents.g < idents.script >& idents.log
real 367m13.385s
user 365m29.535s
sys 0m44.363s
The script ran to completion successfully in about 365 minutes and the
size of the TGM created was 15.7 Gb.
Running 'wc' on the script and the log results in:
$ wc --lines idents.script idents.log
150002 idents.script
150002 idents.log
300004 total
Some process time snapshots of virtual memory (VM) used, TGM size, and
the number of regions completed:
time VM TGM no.
(min) (Mb) (Gb) regions
==== === === =====
90 479 5.4 30,500
96 484 5.8 31,500
105 488 6.2 32,500
117 493 6.7 33,933
140 502 7.7 36,317
159 509 8.5 38,081
168 512 8.8 38,767
182 516 9.3 39,974
199 522 10.0 41,259
207 524 11.0 41,808
225 528 11.6 43,013
238 531 12.0 43,831
251 535 12.5 44,584
264 538 12.9 45,304
293 543 13.8 46,795
311 546 14.3 47,643
321 548 14.5 48,101
334 551 14.9 48,633
346 552 15.2 49,178
352 553 15.4 49,422
365 553 15.7 50,000
It appears the process is not exactly linear. Putting the numbers in
a spreadsheet shows the average time per region and the average file
size per region monotonically increasing as the total time increases
(the average VM per region is slightly variable but approximately
level).
(Note: it would be handy to have some mged logging functions like 'log
date', 'log sys df', 'log sys top', and 'log num_regs' to do such
studies.)
I converted the TGM to ascii form and back to see any file size differences:
$ time ./bin/g2asc idents.g idents.asc
real 6m20.486s
user 0m9.021s
sys 0m9.821s
$ ls -l idents.asc idents.script
10535519 Jun 1 12:51 idents.asc
3005582 Jun 1 06:08 idents.script
$ time ./bin/asc2g idents.asc idents2.g
real 0m9.652s
user 0m7.192s
sys 0m1.016s
$ ls -l idents.g idents2.g
15740460752 Jun 1 12:16 idents.g
13429440 Jun 1 13:09 idents2.g
Note the significant packing of the database.
Using ./bin/mged interactively on both TGMs resulted in the following:
$ ./bin/mged -c idents.g
> tops
> q
real 4m57.550s
user 0m7.244s
sys 0m8.385s
$ ./bin/mged -c idents2.g
> tops
> q
real 0m9.146s
user 0m2.068s
sys 0m0.120s
Generating the idents files at the command line:
$ time ./bin/mged -c idents.g "idents idents.idents all" >& idents.idents.log
real 2m57.379s
user 0m7.420s
sys 0m7.512s
$ time mged -c idents2.g "idents idents2.idents all" >& idents2.idents.log
real 0m3.684s
user 0m2.112s
sys 0m0.164s
Comparing output:
$ diff idents.idents idents2.idents
1,2c1,2
< 1 -8 Summary Table {idents} (written: Fri Jun 1 13:27:35 2012)
< 2 -7 file name : /disk3/extsrc/brlcad-build/idents.g
---
> 1 -8 Summary Table {idents} (written: Fri Jun 1 13:32:06 2012)
> 2 -7 file name : /disk3/extsrc/brlcad-build/idents2.g
50015,50016c50015,50016
< 1 -8 Summary Table {idents} (written: Fri Jun 1 13:27:35 2012)
< 2 -7 file name : /disk3/extsrc/brlcad-build/idents.g
---
> 1 -8 Summary Table {idents} (written: Fri Jun 1 13:32:06 2012)
> 2 -7 file name : /disk3/extsrc/brlcad-build/idents2.g
Looking at the contents of the idents files I could see no problems.
I'm not convinced there is a problem; however, I do recommend
contacting the person reporting the problem to confirm the problem
still exists with a recent trunk build. If it does, he or she needs
to provide more details such as O/S, architecture (32- or 64-bit), TGM
file size, etc. Ideally, he or she needs to provide a TGM that
exhibits the problem.
I do not think the next release should be held up for the idents "problem."
Additional experiments:
I created another 50K region TGM (idents3.g) using mostly tcl commands
except for the final grouping to 'all'. Here are the first few lines
and the last grouping line:
title {Untitled BRL-CAD Database}
units mm
put {s0.s} ell V {0 0 0} A {5 0 0} B {0 5 0} C {0 0 5}
put {s0.r} comb region yes tree {l s0.s}
attr set {s0.r} {region} {R} {region_id} {0} {material_id} {1} {los} {100}
put {s1.s} ell V {0 0 0} A {5 0 0} B {0 5 0} C {0 0 5}
put {s1.r} comb region yes tree {l s1.s}
attr set {s1.r} {region} {R} {region_id} {1} {material_id} {1} {los} {100}
...
g all s0.r s1.r ... s49999.r
$ time ./bin/mged -c idents3.g < idents3.script >& idents3.log
real 32m30.000s
user 31m43.447s
sys 0m36.206s
Compare the time to 365 min for the first TGM generation above! Max
VM observed was approx. 424 Mb.
$ time ./bin/mged -c idents3.g "idents idents3.idents all" >& idents3.idents.log
real 2m47.849s
user 0m4.988s
sys 0m7.304s
Note the time is about the same as for idents.g (reasonable given the
TGM sizes are about the same).
Some lessons learned:
1. Using tcl code for generating a TGM via scripting is significantly
faster than the "old style" code (especially if it's a complete TGM so
that one can use 'asc2g' on the script).
2. Using g2asc <=> asc2g can help save space and time by packing the
TGM database. (Perhaps a lazy mged run-time process to do the same
would be useful.)
Best regards,
-Tom
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BRL-CAD Developer mailing list
brlcad-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/brlcad-devel
