On Thu, Jan 10, 2008 at 10:26:46AM -0800, Doug Cutting wrote: >Joydeep Sen Sarma wrote: >>if the cluster is unused - why restrict parallelism? if someone's willing >>to wake up at 4am to beat the crowd - they would just absolutely hate this. > >[It would be better to make your comments in Jira. ] > >But if someone starts a long-running job at night that uses the entire >cluster then they could monopolize the cluster into the day. If >speculative execution is enabled, then some tasks could be killed to >make room for other jobs are started in the morning, but that's not >always possible. And, if it's not, pickling a job's state and swapping >it to HDFS would be expensive. >
I'd like to throw *job priority* into this festering pool... At least changing the job-priority (done by the cluster-admin) should result in a change in number of max_slots... thoughts? Arun PS: Yes, I do wish this was in jira - I'll add a comment there. >Note also that a task-limiting cluster cluster will still run faster at >night. If you've got 50 nodes with up to 200 tasks running at a time, >then tasks will run faster when only 50 are running. The network is >also a primary bottleneck, and it will be less congested when fewer jobs >are running, and disk contention will be lower too. So night owls would >still have significant advantages. > >It's not intended as a perfect solution, but rather a substantial >improvement for many users that's not too hard to implement. > >Doug
