Actually, sorry I wasn't clear. When I set maxlen and maxint to sys.maxint I don't get that error anymore.
Instead here is the error, when ulimit for open files = 1024 INFO:root:MAF FILES:['UCSC/genomes/MOUSE/mm10/maf/test/chr1.maf'] INFO buildNLMSA.main: Processing MAF file: UCSC/genomes/MOUSE/mm10/maf/test/chr1.maf INFO:pygr-log:Processing MAF file: UCSC/genomes/MOUSE/mm10/maf/test/chr1.maf Traceback (most recent call last): File "/home/vishalrp/codebase/genomics/trunk/code/python/scripts/buildNLMSA.py", line 79, in <module> File "/home/vishalrp/codebase/genomics/trunk/code/python/scripts/buildNLMSA.py", line 65, in main File "cnestedlist.pyx", line 1508, in pygr.cnestedlist.NLMSA.__init__ File "cnestedlist.pyx", line 1785, in pygr.cnestedlist.NLMSA.readMAFfiles File "cnestedlist.pyx", line 1622, in pygr.cnestedlist.NLMSA.newSequence File "cnestedlist.pyx", line 1326, in pygr.cnestedlist.NLMSASequence.__init__ IOError: unable to open in write mode: UCSC/genomes/MOUSE/mm10/maf/pygrdata2/895.build When the ulimit was increased to 4096, It gave a similar error being unable to open file "3967.build". When the ulimit was increased to 10224, It fails with Segmentation fault (core dump). Is this linked to "im" array. Thanks! Vishal On Thursday, April 25, 2013 10:23:33 AM UTC-7, Vishal Patel wrote: > > Hi, > > We usually download the pygr NLMSA from the UCLA server. However, the > latest mm10 - multiz60way alignment does not exist there so we decided to > build it. > > First I ran into issues with max size, > > * 1 msa = cnestedlist.NLMSA(pathstem=pathstem,* > * 2 seqDict=genomeUnion,* > *----> 3 mafFiles=maflist, mode="w")* > * > * > *site-packages/pygr-0.8.2-py2.7-linux-x86_64.egg/pygr/cnestedlist.so in > pygr.cnestedlist.NLMSA.__init__()* > *site-packages/pygr-0.8.2-py2.7-linux-x86_64.egg/pygr/cnestedlist.so in > pygr.cnestedlist.NLMSA.readMAFfiles() > * > * > * > *ValueError: MAF block too long! Increase max size* > > Setting the maxlen and maxint to a billion also did not help so I set it > to sys.maxint. > > However this code still fails because it open 1000's of ".build" files and > fails because it does not have "write permission" on the next file handle > it tries to open. Basically hits the max open files limit. I increased the > ulimit to 10,000 files at which point it fails with "Segmentation fault > (core dumped)" > > Here is lsof out every 0.1 s when it failed, > 170 # these are file handles to genome files > 170 > 170 > 170 > 170 > 170 > 170 > 726 # starts creating .build files > 1682 > 2443 > 3706 > 6300 > 7209 > 9160 # hits the upper limit again and fails with a core dump. > > I have tried this with py2.6/pygr0.8.1 and py2.7/pygr0.8.2 and get the > same error. > > Has anyone experienced this? > What are the .build files? > > Vishal > > > > > > > > > -- You received this message because you are subscribed to the Google Groups "pygr-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to pygr-dev+unsubscr...@googlegroups.com. To post to this group, send email to pygr-dev@googlegroups.com. Visit this group at http://groups.google.com/group/pygr-dev?hl=en. For more options, visit https://groups.google.com/groups/opt_out.