Also, using no -mem-limit option will first try to cache the entire input file in memory, so that it doesn't need to read rows from disk more than once. In your case, this would need 1.2TB of memory, so I strongly suggest that you use -mem-limit if you still want to compute it this way.
Tim On Fri, Feb 24, 2017 at 3:55 PM, Timothy Coalson <[email protected]> wrote: > As noted by others, we used a different approach to avoid making such a > large concatenated file, and the differences from excluding 5% of subjects > should be negligible. > > As for your error: "Exceeded job memory limit" - this is not an error > message from wb_command, this is an error from the queueing system. Some > queueing systems enforce limits on memory or runtime of the jobs, and I > suspect this may be what is going on. If you didn't tell the queueing > system that you expect the job to use 5GB when using -mem-limit 5, it may > kill the job when some default limit is reached. > > Also, the -mem-limit option isn't intended to be exact, so be sure to give > the job some extra room above the -mem-limit amount. Since each row in > your file is 13.5MB, the single-row sized memory allocations for processing > (as opposed to the caching logic) can start to add up. > > Finally, note that wb_command's multithreading isn't written with > multi-socket systems in mind, and can complete things like > -cifti-correlation faster when restricted to one socket than if run across > all sockets, contrary to the increase in computation resources. One reason > for this is that communication between cores on a single socket is at least > an order of magnitude faster than communication across sockets. The > default behavior of what we use for multithreading is to run on all cores, > ignoring whether they are on different sockets, so manual intervention is > recommended for this case. > > Tim > > > On Fri, Feb 24, 2017 at 8:30 AM, Xavier Guell Paradis <[email protected]> > wrote: > >> Dear HCP experts, >> After demeaning and merging resting-state files of 700 subjects >> (resulting in a 1200GB file), I would like to do -cifti-correlation to get >> a .dconn file. I am using a computational cluster, and even by using a node >> with 300GB of memory the command does not seem to work (I get the message: >> "Exceeded job memory limit, Job step aborted: waiting up to 32 seconds for >> job step to finish"). I have tried to use no -mem-limit as well as a >> -mem-limit as low as 5, and I still get the same message. >> >> Do you know if it is possible to use -cifti-correlation for a huge file >> (700 subjects merged); and if so what level of memory would be required to >> do this? >> >> Thank you very much, >> Xavier. >> >> _______________________________________________ >> HCP-Users mailing list >> [email protected] >> http://lists.humanconnectome.org/mailman/listinfo/hcp-users >> > > _______________________________________________ HCP-Users mailing list [email protected] http://lists.humanconnectome.org/mailman/listinfo/hcp-users
