Hi Paul, I don't know what parameters you are using, but let me clarify a few things:
- the number of .build files is NOT the number of open files. It is writing to the .build files in parallel but automatically limits the number that are actually *open* at any one time based on the value of maxOpenFiles2. - your error message indicates to me that the linked-list queue method of limiting the number of open files is operating, because your OS is not going to let you open 2 million files (or even more than a few thousand) at once! - the fact that it's generating such a crazy number of .build files suggests either that you're using an odd set of parameters that's forcing it to split up the union coordinate system in an extremely inefficient way, or that the total coordinate space of all the sequences in the alignment is insanely huge (approximately 4 x 10^15). The latter doesn't make sense to me, so what exactly are your build parameters? Based on your comments about setting maxOpenFiles=4096 it sounds like you're not following what I suggested, that is, just calling NLMSA construction with the default parameter settings (no keyword argument values). At this point what we need to do is to ensure that the union coordinate system is getting split in the usual, efficient way. Note that the new "rebuf" code did not alter that aspect of things in any way. Why not just run NLMSA construction with default values (i.e. don't pass in any kwarg parameter values)... We may end up needing to debug this "live" e.g. with skype screen sharing, as that will be a lot more efficient than going back and forth with email messages. -- Chris On Tuesday, May 7, 2013 7:35:03 PM UTC-7, Paul Rigor wrote: > > Hi Chris, > > Thanks so much for the update. I did have a chance grab, compile, and run > the code. However, on my tests, I came across the same problem. However, > now we've reach an unexpected number of open files, namely, 2,097,148. I've > set both of the maxFileOpen/2 parameters to 4096 each. Are there supposed > to be that many number of open files? > > -- You received this message because you are subscribed to the Google Groups "pygr-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to pygr-dev+unsubscr...@googlegroups.com. To post to this group, send email to pygr-dev@googlegroups.com. Visit this group at http://groups.google.com/group/pygr-dev?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
