Jan, On 8.05.08, Jan Ploski wrote: > [EMAIL PROTECTED] schrieb am 05/08/2008 12:36:23 PM: > > > > > 2) generic control of RAM-per-process > > > > > > We would like to move to JSDL where I would think this would be > > > covered, but after scanning, it looks like it isn't. jsdl posix has > > > MemoryLimit, but that is for the job and not for each process in the > > > job. So I don't think even JSDL provides this. > > > > 8.1.14.1 Definition > > > This element is a positive integer that describes the maximum amount > > > of physical memory that the job should use when executing. > > > The amount is given in bytes. If this is not present then the > > > consuming system MAY choose its default value10. > > > > > > > This would suffice if implemented properly. > > > > The memory per process would be > > mem_per_process = MemoryLimit / count > > > > The number of cores to assign per node on cluster with multi-core nodes > > could be calculated as > > > > available_RAM_per_node / mem_per_process > > Steve, > > I'm not sure whether it would be a correct implementation or another quick > hack. JSDL says nothing about the relationship of a POSIXApplication to a > group of processes launched by MPI. As a matter of fact, it is remarkably > silent about the relationships between jobs and processes and says nothing > about relationships among processes. Maybe noone familiar with MPI > participated in writing JSDL or maybe - more likely - the tough issue was > put off "until later". > These are just suggestions. As to correctness (as you point out) the question is "according to what"? My intent here is to help the Globus developers to find a solution, by explaining the need.
The need is pretty clear, although some of the details are fuzzy. These days, a cluster user can effectively quadruple their memory per process, say by specifying the processes per node. For certain applications, this can be crucial. > Anyway, one can reason about the exectuion of an MPI application as a > scenario involving the execution of n instances of a POSIXApplication. > This interpretation would fit quite well the actual MPI runner > implementations whose job is always to launch n processes of the > user-specified executable on m <= n machines, using whatever > system-specific means are available. Therefore, I would suggest that if > JSDL is used, the MemoryLimit in the POSIXApplication element is not some > aggregate "physical memory that the job should use when executing" to be > divided among processes using a rule of thumb. Instead, treat it as a > specification which applies to each single process of a multi-process job; > it *is*, after all, a description of an executable POSIX process. For > maximum flexibility, one should probably be able to specify a different > POSIXApplication element for each MPI process. > > Apart from these considerations, I am not sure if your "RAM per process" > requirement is covered by the intent of MemoryLimit. MemoryLimit basically > translates to "ulimit -m" in bash (JSDL authors also forgot to mention > whether the hard or soft limit was meant). Is this what you are looking > for? Or do you want to guarantee that a certain amount of memory can be > allocated by a process without incurring paging activity during its entire > execution? Perhaps both? > The user wants to set a minimum amount of RAM available to each process. That might be specified in more than one way, in principle. The issues on conventional clusters are different from those on shared-memory machines, but the request is the same. There are at least two uses of these parameters. They are unfortunately not clearly stated. One is a contract with a resource allocator/scheduler. For instance, a maximum memory requirement, like a maximum walltime requirement, can be used by the allocator to efficiently and effectively allocate resources for and schedule the job. Another is what you call a "hard limit": my user wants a certain amount of RAM per process, no less. This is like the requirement of number of processes. In our case, the job should fail immediately (with an informative message) if that requirement can't be satisfied. The Globus WS-GRAM documentation is completely silent on the intended purpose of such parameters as minMemory. Better documentation alone would help a great deal. Maybe I'll make another bug report about documentation. > By the way, here is what the documentation of PBS/TORQUE offers on the > topic of memory specifications (I bet you can find similar passages in SGE > documentation, too): > > 'mem': > "Maximum amount of physical memory used by the job. (Ignored on Darwin, > Digital Unix, Free BSD, HPUX 11, IRIX, NetBSD, and SunOS. Also ignored on > Linux if number of nodes is not 1. Not implemented on AIX and HPUX 10.)" > > 'pmem': > "Maximum amount of physical memory used by any single process of the job. > (Ignored on Fujitsu. Not implemented on Digital Unix and HPUX.)" > > 'pvmem': > "Maximum amount of virtual memory used by any single process in the job. > (Ignored on Unicos.)" > > 'vmem': > "Maximum amount of virtual memory used by all concurrent processes in the > job. (Ignored on Unicos.)" > > Despite TORQUE being a "low-level" resource manager, this is just as fuzzy > as JSDL, and remarkably non-portable. How on earth does one, in general, > measure the "amount of physical memory" used by a process? The meaning of > such magical parameters only becomes established for a particular > architecture and particular configuration of TORQUE, and can only really > be found out through experimentation or source code inspection. Some > aspects of reality are really difficult to abstract away... > The amount of memory used by a scientific process is typically quite well known by its user. It is not at all magical, and is something they regularly calculate. There may be cases where some experimentation is required, but then they still know the value. Cheers! - - - - - - - - - - - - - - - - - - - - - - - - - Steve White +49(331)7499-202 e-Science / AstroGrid-D Zi. 35 Bg. 20 - - - - - - - - - - - - - - - - - - - - - - - - - Astrophysikalisches Institut Potsdam (AIP) An der Sternwarte 16, D-14482 Potsdam Vorstand: Prof. Dr. Matthias Steinmetz, Peter A. Stolz Stiftung privaten Rechts, Stiftungsverzeichnis Brandenburg: III/7-71-026 - - - - - - - - - - - - - - - - - - - - - - - - -
