just an idea, maybe it is worse to provide brand new cmd line option
to mpirun. This option will accept filename and support combined
syntax for machinefile/hostfile (to define allocations) and rankfile
(to define placement).
YAML syntax can be used in order to describe file primitives
(http://www.yaml.org/start.html)
for example:
$ mpirun -clusterfile /path/to/clusterfile
$ cat clusterfile
hostX:
slots : int
maxslots : int
ranks : rankid[@socket:core]
example of clusterfile
===============
hostX:
slots : 4
maxslots : 4
ranks : 1,16,22
hostY:
slots : 8
maxslots : 8
ranks : 1@0:*, 3@2-3, 4@0:1, 5
By doing so, we keep backwards compatability.
after reading clusterfile, code should perform *hostfile* and
*rankfile* parts as today.
what do you think?
Mike
On Mon, Jun 22, 2009 at 1:30 PM, Terry Dontje <terry.don...@sun.com
<mailto:terry.don...@sun.com>> wrote:
Let us think about this some more. We'll try and reply later today.
--td
Ralph Castain wrote:
Had a chance to think about how this might be done, and looked
at it for awhile after getting home. I -think- I found a way
to do it, but there are a couple of caveats:
1. Len's point about oversubscribing without warning would
definitely hold true - this would positively be a "user
beware" option
2. there could be no RM-provided allocation, hostfile, or
-host options specified. Basically, I would be adding the
"read rankfile" option to the end of the current allocation
determination procedure
I would still allow more procs than shown in the rankfile
(mapping the rest bynode on the nodes specified in the
rankfile - can't do byslot because I don't know how many slots
are on each node), which means the only change in behavior
would be the forced bynode mapping of unspecified procs.
So use of this option will entail some risks and a slight
difference in behavior, but would relieve you from the burden
of having to provide a hostfile. I'm not personally convinced
it is worth the risk and probable user complaints of "it
didn't work", but since we don't use this option, I don't have
a strong opinion on the matter.
Let's just avoid going back-and-forth over wanting it, or how
it should be implemented - let's get it all ironed out, and
then implement it once, like we finally did at the end with
the whole hostfile thing.
Let me know if you want me to do this - it obviously isn't at
the top of my priority list, but still could be done in the
next few weeks.
Ralph
On Jun 21, 2009, at 9:00 AM, Lenny Verkhovsky wrote:
Sorry for the delay in response, I totally agree with
Ralph that it's not as easy as it seems, 1. rankfile
mapper uses already allocated machines ( by scheduler or
hostfile ), by using rankfile as a hostfile we can run
into problem where trying to use unallocated nodes, what
can hang the run.
2. we can't define in rankfile number of slots on each
machine, which means oversubscribing can take place
without any warning.
3. I personally dont see any problem using hostfile, even
if it has redundant info, hostfile and rankfile belong to
different layers in the system and solve different
problems. The original hostfile ( if I recall correctly )
could bind rank to the node, but the syntax wasn't very
flexible and clear.
Lenny.
On Sun, Jun 21, 2009 at 5:15 PM, Ralph Castain
<r...@open-mpi.org <mailto:r...@open-mpi.org>
<mailto:r...@open-mpi.org <mailto:r...@open-mpi.org>>> wrote:
Let me suggest a two-step process, then:
1. let's change the error message as this is easily
done and thus
can be done now
2. I can look at how to eat the rankfile as a hostfile.
This may
not even be possible - the problem is that the entire
system is
predicated on certain ordering due to our framework
architecture.
So we get an allocation, and then do a mapping against that
allocation, filtering the allocation through hostfiles,
-host,
and other options.
By the time we reach the rankfile mapper, we have already
determined that we don't have an allocation and have to
abort. It
is the rankfile mapper itself that looks for the -rankfile
option, so the system can have no knowledge that
someone has
specified that option before that point - and thus,
even if I
could parse the rankfile, I don't know it was given!
What will take time is to figure out a way to either:
(a) allow us to run the mapper even though we don't
have any
nodes we know about, and allow the mapper to insert the
nodes
itself - without causing non-rankfile uses to break
(which could
be a major feat); or
(b) have the overall system check for the rankfile
option and
pass it as a hostfile as well, assuming that a hostfile
wasn't
also given, no RM-based allocation exists, etc. - which
breaks
our abstraction rules and also opens a possible can of
worms.
Either way, I also then have to teach the hostfile
parser how to
realize it is a rankfile format and convert the info in
it into
what we expected to receive from a hostfile - another
non-trivial
problem.
I'm willing to give it a try - just trying to make
clear why my
response was negative. It isn't as simple as it
sounds...which is
why Len and I didn't pursue it when this was originally
developed.
Ralph
On Sun, Jun 21, 2009 at 5:28 AM, Terry Dontje
<terry.don...@sun.com <mailto:terry.don...@sun.com>
<mailto:terry.don...@sun.com
<mailto:terry.don...@sun.com>>> wrote:
Being a part of these discussions I can understand your
reticence to reopen this discussion. However, I
think this
is a major usability issue with this feature which
actually
is fairly important in order to get things to run
performant.
Which IMO is important.
That being said I think there are one of two things
that
could be done to mitigate the issue.
1. To eliminate the element of surprise by
changing mpirun
to eat rankfile without the hostfile.
2. To change the error message to something
understandable
by the user such that they
know they might be missing the hostfile option.
Again I understand this topic is frustrating and
there are
some boundaries with the design that make these two
option
orthogonal to each other but I really believe we
need to make
the rankfile option something that is easily usable
by our users.
--td
Ralph Castain wrote:
Having gone around in circles on
hostfile-related issues
for over five years now, I honestly have little
motivation to re-open the entire discussion
again. It
doesn't seem to be that daunting a requirement
for those
who are using it, so I'm inclined to just leave
well
enough alone.
:-)
On Fri, Jun 19, 2009 at 2:21 PM, Eugene Loh
<eugene....@sun.com <mailto:eugene....@sun.com>
<mailto:eugene....@sun.com <mailto:eugene....@sun.com>>
<mailto:eugene....@sun.com
<mailto:eugene....@sun.com> <mailto:eugene....@sun.com
<mailto:eugene....@sun.com>>>>
wrote:
Ralph Castain wrote:
The two files have a slightly different
format
Agreed.
and completely different meaning.
Somewhat agreed. They're both related to
mapping
processes onto a
cluster.
The hostfile specifies how many slots
are on a
node. The rankfile
specifies a rank and what node/slot it
is to be
mapped onto.
Agreed.
Rankfiles can use relative node indexing
and refer
to nodes
received from a resource manager - i.e.,
without
any hostfile.
This is the main part I'm concerned about.
E.g.,
% cat rankfile
rank 0=node0 slot=0
rank 1=node1 slot=0
% mpirun -np 2 -rf rankfile ./a.out
--------------------------------------------------------------------------
Rankfile claimed host node1 that was not
allocated or
oversubscribed it's slots:
--------------------------------------------------------------------------
[node0:14611] [[61560,0],0] ORTE_ERROR_LOG: Bad
parameter in file
rmaps_rank_file.c at line 107
[node0:14611] [[61560,0],0] ORTE_ERROR_LOG: Bad
parameter in file
base/rmaps_base_map_job.c at line 86
[node0:14611] [[61560,0],0] ORTE_ERROR_LOG: Bad
parameter in file
base/plm_base_launch_support.c at line 86
[node0:14611] [[61560,0],0] ORTE_ERROR_LOG: Bad
parameter in file
plm_rsh_module.c at line 1016
% mpirun -np 2 -host node0,node1 -rf
rankfile ./a.out
0 on node0
1 on node1
done
It seems to me that the rankfile has sufficient
information to
express what I want it to do. But mpirun
won't accept
this. To
fix this, I have to, e.g.,
supply/maintain/specify
redundant
information in a hostfile or host list.
So the files are intentionally quite
different.
Trying to combine
them would be rather ugly.
Right. And my issue is that I'm forced to
use both
when I only
want rankfile functionality.
On Thu, Jun 18, 2009 at 1:52 PM, Eugene Loh
<eugene....@sun.com
<mailto:eugene....@sun.com> <mailto:eugene....@sun.com
<mailto:eugene....@sun.com>>
<mailto:eugene....@sun.com
<mailto:eugene....@sun.com>
<mailto:eugene....@sun.com
<mailto:eugene....@sun.com>>>> wrote:
In order to use "mpirun --rankfile",
I also
need to specify
hosts/hostlist. But that information is
redundant with what
I provide in the rankfile. So, from
a user's
point of view,
this strikes me as broken. Yes?
Should I
file a ticket, or
am I missing something here about this
functionality?
_______________________________________________
devel mailing list
de...@open-mpi.org
<mailto:de...@open-mpi.org> <mailto:de...@open-mpi.org
<mailto:de...@open-mpi.org>>
<mailto:de...@open-mpi.org
<mailto:de...@open-mpi.org> <mailto:de...@open-mpi.org
<mailto:de...@open-mpi.org>>>
http://www.open-mpi.org/mailman/listinfo.cgi/devel
------------------------------------------------------------------------
_______________________________________________
devel mailing list
de...@open-mpi.org <mailto:de...@open-mpi.org>
<mailto:de...@open-mpi.org <mailto:de...@open-mpi.org>>
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org <mailto:de...@open-mpi.org>
<mailto:de...@open-mpi.org <mailto:de...@open-mpi.org>>
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org <mailto:de...@open-mpi.org>
<mailto:de...@open-mpi.org <mailto:de...@open-mpi.org>>
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org <mailto:de...@open-mpi.org>
<mailto:de...@open-mpi.org <mailto:de...@open-mpi.org>>
http://www.open-mpi.org/mailman/listinfo.cgi/devel
------------------------------------------------------------------------
_______________________________________________
devel mailing list
de...@open-mpi.org <mailto:de...@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org <mailto:de...@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/devel
------------------------------------------------------------------------
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
