Chris,
I'm trying to gain an understanding of where memory is going when I
construct an NLMSA... and failing miserably. In particular, I'm trying
to understand why references to sequence objects are sticking around; I
think this may be why NLMSAs for > 5m sequences are blowing up my
machine when I construct them in a "normal" way.
Here's some code, annotated with refcounts for 'seq' using
'sys.getrefcount' after a 'gc.collect()':
--
al = cnestedlist.NLMSA('foobar', mode='w', pairwiseMode=True)
seq = db['000505_2504_0990,id=631459']
db.clear_cache()
# seq: refcount=3
al += seq
# seq: refcount 5
al[seq] += seq
# seq: refcount 6
al.build()
# seq: refcount 6
del al
# seq: refcount 6
db.clear_cache()
# seq: refcount 6
---
I've done things like force-disabling caching in the code (hardcoding
cacheMax=0, etc.) and I can't get lower the dang refcounts. I feel
like at the very least, deleting 'al' and doing a gc.collect() in the
asence of any caching should lower the refcount to what it was before I
added the sequence to 'al'!
Any thoughts? I've attached the script & data file.
thanks,
--titus
--
C. Titus Brown, [email protected]
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"pygr-dev" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/pygr-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
import sys, os.path
sys.path.insert(0, '/Users/t/dev/pygr')
from pygr import cnestedlist, seqdb
import gc
n = 0
def count():
global n
n += 1
gc.collect()
print 'AT', n, 'REFCOUNT:', sys.getrefcount(seq)
##
db = seqdb.SequenceFileDB(os.path.abspath('mda3.fa'))
al = cnestedlist.NLMSA('foobar', mode='w', pairwiseMode=True)
seq = db['000505_2504_0990,id=631459']
count()
db.clear_cache()
count()
al += seq
count()
al[seq] += seq
count()
al.build()
count()
del al
count()
db.clear_cache()
count()
>000505_2504_0990,id=631459
CGCGGCCTGGGCCTTGTTCTGCGACGGGATCATCTGGATCCACAGGCCCAGGCGCATGTCGTTGACGAAGATAAAACCATGCACCGGGTCGGTGGAAAGACTGCCCCAGTTCATGCCGCCCAGGGAACCGGGAAAGCTCAGGGATTTATCCGTGCCCGGCGCGGTGTACAGGCCGTCGTAGCGCATG
>000506_2901_1606,id=631460
AGAAAGATCGGCAAGAACAGAAACAGGAACACATTGGACGAGAAAACCATCCCGATCTCTCCATGTTTAACCAACAGTCAAGGGCCAGAGCCCCCCCAAACCCCACCGAAAAACGGGAAGGCGGTACTACATTCCTACTGACGAAACCGGCTCAAAAATAGTGGGAGGGGGCGTTGCCCCGACTGGCGGTGTAACAGTCGCCATCTGTGTTGAATGACACCCTGCTA
>000508_2356_1049,id=631461
GCTGGAAACCTGGCTGCAGGAGTACACCTTTCCCCTGGAAGCGCGCTACGGCGATGCCGGCTTTGCCACTTCGGTGTACGACAGCGT
>000510_2314_0053,id=631462
CGGGGCGAAAGGTGTGCCCCAGCGCGATGAACTGGTAGTTGCCGCGCCGGTCGGTGAAAACAGCCAGGCCGCCAGGCGTCAAGGCCACGTTGAACCAGTCGTTGTCGGTGCGCGGTTGGTCCAGGTCCAGCTTGCCGGACTGGCTCAGTGCAAGCACGCTGCCTGCCAGGCTTTCCAGGTTCAGAACGCTGAGCTGGCGGCCGTCAGCGCCATACAGCCGTAGCTCTCCGGTGGCTTCGACCTTG
>000511_3029_1889,id=631463
TCGGACTCGGCGAACAGACGCTCGGCCACGGCGTCAGCGGCGAGCATGCCGGACTTCA
>000514_3469_0938,id=631464
CCTTATGCCATCCATATGGATGGTTTGGGGATGGTTTCAAGGCCTGGTGTTGCAATTTGACGCGGCATGTCGTGGGGTATCCGGCGGACGGCGCTGCAGCGTGTGAGTTGCAATCATTGCCCTGTCACGTCTGAGTCGTAAGGTGGCGGAGCCGGGCACGGCGTCCGGCGTTCCTTGGTCATGCGACAGGAGGCTTCATGCCAGACTTGCAGGCAGGCTTGCCGAGCGGCACGAGCGTTTGG
>000516_2314_3283,id=631465
ACGTGGATTTTCCCGAGCCGTTGGCGCCGATAACGGCGGTAAGCGAGCCGCGGGCAATACTACCATTGAGATGGTGCACCGCCGCCCGGCCGGCATATCCAAGGGTCAGATTTGAGAAGATAAGGCACGGATCGTTCATCGATCCCTCCGGAGCAGGATGTCATTCGATGACCCTTGATATGTAATGTTATTACATTCGTCAAACGAATATGTAATGAAATAACATTTACTGGTGGAAACGGACTGGCACGCTGTCCCCTCCGG
>000519_3299_0436,id=631466
CTTGCACAATTGCGCCGTAACTACACCCTGTATGGTCTGCGGGACGAAGCCGCGCAGGCCGATCCCATCGAGCTGTTCGGCCAATGGCTGAGCCAGGCGCGCAAGACCGAAG
>000520_2808_1649,id=631467
GCCTGATCATGATGGTCAAGGAAAAGTGGCCCGAGACCGAGGTCCATCTCTCGGTGCAGGCCAACACCACCAACTGGGCCACGGTGAAGTTCTGGCAGAAGGCCGGCGTCTCGCGCATCATCCTGTCGCGCGAGCTGTCGCTGGACGAGATCGAGAAGATCCGCCAGGAATGCCCGGACATGGAGATCGAGGTCTTCGTGCACGGCGCGCTGTGCATCGCCTACTCGGGCCG
>000522_2925_0934,id=631468
ATAACGCGTTTGAATACGTTTCAGCGATTGAGCAAATTCCGGAAGTCAATGCTGTAAGTCCCA
>000528_2232_1120,id=631469
GGCCCCGCGCTCAACGAAGGCGGACTGCTACCCTACATGGCCCGCGTCTTCGTGCTCGGCACGGTCTGGCCGTCGGCGCGCACGCGTGCGCAAATGCCTCCCGGAACATACGACTGATACGACCCTGAACCCATGCCCGTCGCCTATTCCAACGTCTATCTGCATGCCGCAGGCATGTTCCTGCCCGGTGAACCCGTGGACAACGCAGGCATGGATGCCTACGTGGCGCCGCTCAACCGCCTGTCCGAGCCG
>000530_3007_1084,id=631470
GCCACCGGCTTCTTGTCCTCATGGATGAGCTGGCGGGCGTGGGCGATCTGCTCGGCCGACAG
>000531_2240_0368,id=631471
AGGGCTACACGGTTCACGGCGCGGCCCTGTCTGGCAAGGCGGCCGAGGGCCTGGAGGAAAGTTCCGGCATCCAGAGCCGCACGCTGGCGTCATGGTCCCGCGGTTGGGAGAACGACCGCGGCCAGCTCAACCGCGGCGACGTGTTTGTCATCGACGAGGCGGGCATGGTCGGGAGCCGGCAGCTCGCCCGGTTTGTCACCGAGGCCGAGGCGCGCGGGGGCGAAGATCGTCCTT
>000537_2800_1812,id=631472
GCCCGACGCTCTGCTGGTCCAGTTCGACGGGCGCGCGGTATTCTTTGCTCTCTTCGATCAACCGAACCGTTTCGGCCAGCTCTTGCCGTAAGCGGTGCTCGAATGCCCGAATGACTTCGGCGTCCTCTATGGTTTTCATGAACGTCCTCCCCGGAACGAAGGGCGGGA
>000540_2318_1266,id=631473
TTATCAGAACGCGATCGTTGTTGCCCCGGAAAGCTCGAACTTTCAGGTCAGCGTGAACAATGTCGGGCTTGGTGCAGCGCGTATCGAGGCGCCGGACAGTCAGGGGAATATTTCCCTTAAAATTCCGCCCGGTGTCCTTCAGGTCGGCGCGAATGTCGTGTCGCTCGGGTGCAGCAGCGCCACAGAACAGATTGCACGATCCAGTCCACTTACGATCTATGGACCGATGTCGAT
>000544_2225_1047,id=631474
TCATCGAGCTTCTTGCGGTTCTCCGGGGTATCCTCCCCGCCATCACCAGCTTGCAGCCAAGGCACACAGGGGATAGG
>000545_2526_2263,id=631475
GCGCGAGGTAGCCGGTGACAAAGATGCCGAAGCTTTGCAGCAGGCGCAGCCATTCGGGCATTTCCGGCGGGAAGAACAGTTGGCTCAGGGTGAGCGCGAAAAACACAAAGATGATGAAGTCAT
>000546_3029_0645,id=631476
GGTGATCGAGGTCGGCCACCGCCACGCCGATATGCCGCGCGTATTCTTCAGCGGCAGGTTCGGCATTGAGTTCGAATACCCGGCGGGACGGGCTGTCGGCCGCGGTCACCACCAGTTTTTCCGCACGCGGCAGG
>000548_2463_1312,id=631477
GGGGACAACGAGCGCAGGGGCGAACGCGACGAGGCCCAGGCCCTGCAGGCGCGCGTGATGGAGCTGGAGATCAAGGCCAGCTACACCGACGACCTGCTGGAGCAACTGAACCTGACCATCTACCGCCAGCAGCAGCAGATCGACCGGCTCCTGGACGAACTGCGCGACCTGCG
>000553_2349_0881,id=631478
CTGACCAACACCCCGGCGTGCTGGACAAGGCCGGCAGCCTGCTGACCGACCTCACGGCGCGCGAGATCGACGCCCTGTTCGCTGA
>000554_2568_3049,id=631479
GATCACCCGGCTAGACCGCCTCGCGCGATCGACGGCCGACCTTCTGGCGATCGCGGAGCGGATCAAAGAGGCCGGCGCGGGCCTGCGCTCGCTGGCCGAGCCGTGGGCCGACACCACGA
>000557_2625_3733,id=631480
CGCAGTGCAGCCAGCAGCCGCCACAAAGGCAAACCGCCAGGCCACCTGCCAAAAATCTGCACGATGCACTTGCCATGTGCGGCCAGGAGCGCGAACCGGGAAGGTTTCGCCCAAGGCAACCAGCAGCAGCATGCAGACCCGATGCAGACGGGCGGGACATATCACCCTGAACACATTCAGGGAAACCCGCGATTATCACAGCGTTGCCCACTTGATGCAATGCCCTTCGAACCCGAAGG
>000558_2916_2605,id=631481
CACGGGCGATCAGGGCGTCCAGGTCCGAGGCGGCGGCCGCGGCGGCCTCGCCTGCGGCCAGGGCGGCTCCCTCGTCGCTCAGCGTGAAGCGGTAGAAG
>000559_2641_0306,id=631482
GGCGAATTCTTTGATCAGGCGCAGCGCGGCCGCGTCCTGGCGTAGGGTGTCGGCGTTGACGGTGCCACCGGGAATCACCAGGGCATCGAAGTCGCTGCCGACCACGTCCGACAGTTGCACATCGGATTCCACGGTCTTGTCCTTTTCGGTGTCGCCAACGAACGTCTGGGTGGTGCCGCCCTTGCTGGACGCATGGGTCACATTGGCGCCGAAACCGCGCAACGCTTCCAGCGGCTTGACCAGTTCGTCGCGTTCGATACCGGTA
>000560_2582_2119,id=631483
TGGGAAAGAAACGCCAGGCATCGCCGTCGGCACTGTAGTCCTCTCGTACCGTGCCCACTGGCGCTCACTTAAATACGGGCCCC
