Hi Chris and community, So after applying the modifications according to pyrex, we're now encountering the following IO error message on a machine with ulimit (soft/hard) already set to 4096.
The error message follows: INFO buildNLMSA.main: Processing MAF file: /home/baldig/projects/genomics/nonsvn/data/UCSC/genomes/MOUSE/mm10/maf/chr19.maf INFO:pygr-log:Processing MAF file: /home/baldig/projects/genomics/nonsvn/data/UCSC/genomes/MOUSE/mm10/maf/chr19.maf Traceback (most recent call last): File "/home/prigor/codebase/genomics/trunk/code/python/scripts/buildNLMSA.py", line 91, in <module> File "/home/prigor/codebase/genomics/trunk/code/python/scripts/buildNLMSA.py", line 77, in main File "cnestedlist.pyx", line 1508, in pygr.cnestedlist.NLMSA.__init__ File "cnestedlist.pyx", line 1798, in pygr.cnestedlist.NLMSA.readMAFfiles File "cnestedlist.pyx", line 1622, in pygr.cnestedlist.NLMSA.newSequence File "cnestedlist.pyx", line 1326, in pygr.cnestedlist.NLMSASequence.__init__ IOError: unable to open in write mode: /extra/baldig1/genomics/pygrdata/alignments/MOUSE/mm10/multiz60way/3967.build On Thu, Apr 25, 2013 at 12:14 PM, Paul Rigor <paul.ri...@uci.edu> wrote: > Hi Chris, > > Nevermind, I guess the warnings were actual errors. So I've changed the > __new__ to __cinit__ according to the messages. All seems well and the > module compiles. We'll keep you posted on the actual NLMSA building for the > MAFs, in case we encounter anything else. > > Thanks again, > Paul > > > -- > Paul Rigor > http://www.ics.uci.edu/~prigor > > > On Thu, Apr 25, 2013 at 12:07 PM, Paul Rigor <paul.ri...@uci.edu> wrote: > >> Hi Chris, >> >> So which version of Pyrex and GCC is the most compatible? Currently, when >> attempting to recompile using the modifications you recommended, we get the >> following compilation warnings and error below. We have used pyrex versions >> 0.8.2.x - 0.9.9 and gcc 4.1.2, 4.3.0, & 4.7.1. >> >> THanks! >> Paul >> >> pyrexc pygr/cnestedlist.pyx --> pygr/cnestedlist.c >> /home/prigor/scratch/pygr/pygr-0.8.2/pygr/cnestedlist.pyx:8:2: __new__ >> method of extension type will change semantics in a future version of >> Pyrex. Use __cinit__ instead. >> /home/prigor/scratch/pygr/pygr-0.8.2/pygr/cnestedlist.pyx:51:2: __new__ >> method of extension type will change semantics in a future version of >> Pyrex. Use __cinit__ instead. >> /home/prigor/scratch/pygr/pygr-0.8.2/pygr/cnestedlist.pyx:167:2: __new__ >> method of extension type will change semantics in a future version of >> Pyrex. Use __cinit__ instead. >> /home/prigor/scratch/pygr/pygr-0.8.2/pygr/cnestedlist.pyx:371:2: __new__ >> method of extension type will change semantics in a future version of >> Pyrex. Use __cinit__ instead. >> /home/prigor/scratch/pygr/pygr-0.8.2/pygr/cnestedlist.pyx:424:2: __new__ >> method of extension type will change semantics in a future version of >> Pyrex. Use __cinit__ instead. >> /home/prigor/scratch/pygr/pygr-0.8.2/pygr/cnestedlist.pyx:446:2: __new__ >> method of extension type will change semantics in a future version of >> Pyrex. Use __cinit__ instead. >> /home/prigor/scratch/pygr/pygr-0.8.2/pygr/cnestedlist.pyx:1112:2: __new__ >> method of extension type will change semantics in a future version of >> Pyrex. Use __cinit__ instead. >> /home/prigor/scratch/pygr/pygr-0.8.2/pygr/cnestedlist.pyx:1138:2: __new__ >> method of extension type will change semantics in a future version of >> Pyrex. Use __cinit__ instead. >> /home/prigor/scratch/pygr/pygr-0.8.2/pygr/cnestedlist.pyx:167:40: >> Warning: 'not None' will become the default in a future version of Pyrex. >> Use 'or None' to allow passing None. >> building 'pygr.cnestedlist' extension >> creating build/temp.linux-x86_64-2.6/pygr/apps >> gcc -pthread -pg -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall >> -Wstrict-prototypes -fPIC >> -I/home/baldig/shared_libraries/centos64/pkgs/python/2.6.5/include/python2.6 >> -c pygr/intervaldb.c -o build/temp.linux-x86_64-2.6/pygr/intervaldb.o >> gcc -pthread -pg -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall >> -Wstrict-prototypes -fPIC >> -I/home/baldig/shared_libraries/centos64/pkgs/python/2.6.5/include/python2.6 >> -c pygr/cnestedlist.c -o build/temp.linux-x86_64-2.6/pygr/cnestedlist.o >> pygr/cnestedlist.c:1:2: error: #error Do not use this file, it is the >> result of a failed Pyrex compilation. >> error: command 'gcc' failed with exit status 1 >> >> >> -- >> Paul Rigor >> http://www.ics.uci.edu/~prigor >> >> >> On Thu, Apr 25, 2013 at 11:58 AM, Christopher Lee <cjlee...@gmail.com>wrote: >> >>> >>> >>> On Thursday, April 25, 2013 10:23:33 AM UTC-7, Vishal Patel wrote: >>>> >>>> * >>>> * >>>> * >>>> * >>>> *ValueError: MAF block too long! Increase max size* >>>> >>>> Setting the maxlen and maxint to a billion also did not help so I set >>>> it to sys.maxint. >>>> >>> >>> These settings have nothing to do with the MAF block size limitation you >>> ran into. All you need to do is boost the buffer size for reading a single >>> MAF block (currently set to 4096). It's trivial but it requires editing >>> the source code to change that number in two places: >>> >>> cnestedlist.pyx: 1731, change size of im array from 4096 to something >>> big like 16384 >>> cnestedlist.pyx: 1774, change 4096 to the same increased value you used >>> for im >>> >>> Then recompile pygr: >>> >>> python setup.py build >>> python setup.py install >>> >>> You can also enter a bug report suggesting that this be made user >>> configurable or automatically resized. >>> >>> >>>> >>>> However this code still fails because it open 1000's of ".build" files >>>> and fails because it does not have "write permission" on the next file >>>> handle it tries to open. Basically hits the max open files limit. I >>>> increased the ulimit to 10,000 files at which point it fails with >>>> "Segmentation fault (core dumped)" >>>> >>>> Here is lsof out every 0.1 s when it failed, >>>> 170 # these are file handles to genome files >>>> 170 >>>> 170 >>>> 170 >>>> 170 >>>> 170 >>>> 170 >>>> 726 # starts creating .build files >>>> 1682 >>>> 2443 >>>> 3706 >>>> 6300 >>>> 7209 >>>> 9160 # hits the upper limit again and fails with a core dump. >>>> >>>> I have tried this with py2.6/pygr0.8.1 and py2.7/pygr0.8.2 and get the >>>> same error. >>>> >>>> Has anyone experienced this? >>>> What are the .build files? >>>> >>>> Vishal >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "pygr-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to pygr-dev+unsubscr...@googlegroups.com. >>> To post to this group, send email to pygr-dev@googlegroups.com. >>> Visit this group at http://groups.google.com/group/pygr-dev?hl=en. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >> >> > -- You received this message because you are subscribed to the Google Groups "pygr-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to pygr-dev+unsubscr...@googlegroups.com. To post to this group, send email to pygr-dev@googlegroups.com. Visit this group at http://groups.google.com/group/pygr-dev?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
