Hi Sébastien,
In my case it was due to the shear file I/O, i.e. number of nodes writing files
all the same time. This is only an issue on our batch cluster this isn't
designed or optimized for parallel workloads. On our larger clusters where we
have Panasas parallel scratch space, we have no issues. I just wanted to give
the warning so users would be aware of the increased file I/O these options add.
Thanks,
Ben
On Jun 12, 2012, at 1:05 PM, Sébastien Boisvert wrote:
Benjamin: Is it fairly straining in terms of input/ouput operations per
second or
because of file sizes ?
If it is about IOPS, the code can be enhanced to group I/O operations.
Allen, Benjamin S a écrit :
Louis,
That is indeed what these options do.
Ray seems to checkpoint the completion of each step, not at a regular interval.
So if you have a step in Ray thats taking longer than your max walltime, you'll
still have an issue. Also just an FYI, depending on how big of jobs you're
running (number of cores), and what your scratch file space performance is,
checkpointing can be fairly straining on the cluster.
Ben
On Jun 12, 2012, at 8:49 AM, Louis Letourneau wrote:
I saw these options on Ray
Checkpointing
-write-checkpoints
Write checkpoint files
-read-checkpoints
Read checkpoint files
-read-write-checkpoints
Read and write checkpoint files
I'm hitting walltimes on the cluster I'm using and I'm wondering if by
setting:
-read-write-checkpoints
I can resume where Ray got killed because of walltime?
If that's the purpose, what a great feature! :-)
Louis
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Denovoassembler-users mailing list
Denovoassembler-users@lists.sourceforge.net<mailto:Denovoassembler-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/denovoassembler-users
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Denovoassembler-users mailing list
Denovoassembler-users@lists.sourceforge.net<mailto:Denovoassembler-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/denovoassembler-users
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Denovoassembler-users mailing list
Denovoassembler-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/denovoassembler-users
