Re: [OMPI devel] IB/OpenFabrics pow wow
Jeff Squyres wrote: Based on discussions at SC, I think it is time to have another IB/ OpenFabrics OMPI pow wow teleconference. It would be good to get status updates on what everyone is doing in terms of OpenFabrics kinds of things in OMPI so that we can coordinate the work properly. I'd like to strike a balance of getting everyone relevant involved but also keeping the group small so that we can have a meaningful discussion and come up with good work plans for moving forward. A first/tentative list of attendees include: - Cisco - Voltaire - LLNL - OGC (Chelsio) - ?ORNL? - ?Sun? Please add Mellanox to the list too ... Pasha.
Re: [OMPI devel] IB/OpenFabrics pow wow
On Nov 19, 2007, at 6:56 AM, Pavel Shamis (Pasha) wrote: - Cisco - Voltaire - LLNL - OGC (Chelsio) - ?ORNL? - ?Sun? Please add Mellanox to the list too ... Oops! That was clearly a typo/oversight, sorry... -- Jeff Squyres Cisco Systems
Re: [OMPI devel] Multiworld MCA parameter values broken
I'm not sure it is really necessary - the problem is solely within opal_cmd_line_parse and (if someone can parse that code ;-)) is truly simple to fix. The overly long cmd line issue is due to a bug that Josh was going to look at (may already have done so while I was out of touch). Ralph On 11/9/07 5:10 AM, "Jeff Squyres" wrote: > Should there be another option for passing MCA parameters between > processes, such as via stdin (or any file descriptor)? I.e., during > the command line parsing to check for command line MCA params, perhaps > a new argument could be introduced: -mcauri , where could > be a few different forms: > > - file://stdin: (note the 2 //, not 3, so "stdin" would never conflict > with a real file named /stdin) Read the parameters in off stdin. > > - rml://...rml contact info...: read in the MCA params via the RML > (although I assume that reading via the RML would be *wy* to late > during the MCA setup process -- I mentioned this option for > completeness, even though I don't think it'll work) > > - ip://ipaddress:port: open a socket back and read the MCA params in > over a socket. This could have some scalability issues...? But who > knows; it could be tied into the hierarchical startup such that we > wouldn't have to have an all-to-one connection scheme. Certainly it > would cause scalability problems when paired with today's all-to-one > RML connection scheme for the OOB. > > I'm not sure that the rml: and ip: schemes are worthwhile. Maybe a > file://stdin kind of approach could work? Or perhaps some other kind > of URI/IPC...? (I really haven't thought through the issues -- this > is off the top of my head) > > > > On Nov 8, 2007, at 2:36 PM, Ralph H Castain wrote: > >> Might I suggest: >> >> https://svn.open-mpi.org/trac/ompi/ticket/1073 >> >> It deals with some of these issues and explains the boundaries of the >> problem. As for what a string param can contain, I have no opinion. >> I only >> note that it must handle special characters such as ';', '/', etc. >> that are >> typically found in uri's. I cannot think of any reason it should >> have a >> quote in it. >> >> Ralph >> >> >> >> On 11/8/07 12:25 PM, "Tim Prins" wrote: >> >>> The alias option you presented does not work. I think we do some >>> weird >>> things to find the absolute path for ssh, instead of just issuing the >>> command. >>> >>> I would spend some time fixing this, but I don't want to do it >>> wrong. We >>> could quote all the param values, and change the parser to remove the >>> quotes, but this is assuming that a mca param does not contain >>> quotes. >>> >>> So I guess there are 2 questions that need to be answered before a >>> fix >>> is made: >>> >>> 1. What exactly can a string mca param contain? Can it have quotes or >>> spaces or? >>> >>> 2. Which mca parameters should be forwarded? Should it be just the >>> ones >>> from the command line? From the environment? From config files? >>> >>> Tim >>> >>> Ralph Castain wrote: What changed is that we never passed mca params to the orted before - they always went to the app, but it's the orted that has the issue. There is a bug ticket thread on this subject - I forget the number immediately. Basically, the problem was that we cannot generally pass the local environment to the orteds when we launch them. However, people needed various mca params to get to the orteds to control their behavior. The only way to resolve that problem was to pass the params via the command line, which is what was done. Except for a very few cases, all of our mca params are single values that do not include spaces, so this is not a problem that is causing widespread issues. As I said, I already had to deal with one special case that didn't involve spaces, but did have special characters that required quoting, which identified the larger problem of dealing with quoted strings. I have no objection to a more general fix. Like I said in my note, though, the general fix will take a larger effort. If someone is willing to do so, that is fine with me - I was only offering solutions that would fill the interim time as I haven't heard anyone step up to say they would fix it anytime soon. Please feel free to jump in and volunteer! ;-) I'm willing to put the quotes around things if you will fix the mca cmd line parser to cleanly remove them on the other end. Ralph On 11/7/07 5:50 PM, "Tim Prins" wrote: > I'm curious what changed to make this a problem. How were we > passing mca > param > from the base to the app before, and why did it change? > > I think that options 1 & 2 below are no good, since we, in > general, allow > string mca params to have spaces (as far as I understand it).
Re: [OMPI devel] Multiworld MCA parameter values broken
Sorry -- I'm just joining this conversation late: what's the problem with opal_cmd_line_parse? It should obey all quoting from shells, etc. I.e., it shouldn't care about tokens with special characters (to include spaces) because the shell divides all of that stuff up -- it just gets a char*[] that it treats as discrete tokens. Is it doing something wrong? On Nov 19, 2007, at 8:39 AM, Ralph H Castain wrote: I'm not sure it is really necessary - the problem is solely within opal_cmd_line_parse and (if someone can parse that code ;-)) is truly simple to fix. The overly long cmd line issue is due to a bug that Josh was going to look at (may already have done so while I was out of touch). Ralph On 11/9/07 5:10 AM, "Jeff Squyres" wrote: Should there be another option for passing MCA parameters between processes, such as via stdin (or any file descriptor)? I.e., during the command line parsing to check for command line MCA params, perhaps a new argument could be introduced: -mcauri , where could be a few different forms: - file://stdin: (note the 2 //, not 3, so "stdin" would never conflict with a real file named /stdin) Read the parameters in off stdin. - rml://...rml contact info...: read in the MCA params via the RML (although I assume that reading via the RML would be *wy* to late during the MCA setup process -- I mentioned this option for completeness, even though I don't think it'll work) - ip://ipaddress:port: open a socket back and read the MCA params in over a socket. This could have some scalability issues...? But who knows; it could be tied into the hierarchical startup such that we wouldn't have to have an all-to-one connection scheme. Certainly it would cause scalability problems when paired with today's all-to-one RML connection scheme for the OOB. I'm not sure that the rml: and ip: schemes are worthwhile. Maybe a file://stdin kind of approach could work? Or perhaps some other kind of URI/IPC...? (I really haven't thought through the issues -- this is off the top of my head) On Nov 8, 2007, at 2:36 PM, Ralph H Castain wrote: Might I suggest: https://svn.open-mpi.org/trac/ompi/ticket/1073 It deals with some of these issues and explains the boundaries of the problem. As for what a string param can contain, I have no opinion. I only note that it must handle special characters such as ';', '/', etc. that are typically found in uri's. I cannot think of any reason it should have a quote in it. Ralph On 11/8/07 12:25 PM, "Tim Prins" wrote: The alias option you presented does not work. I think we do some weird things to find the absolute path for ssh, instead of just issuing the command. I would spend some time fixing this, but I don't want to do it wrong. We could quote all the param values, and change the parser to remove the quotes, but this is assuming that a mca param does not contain quotes. So I guess there are 2 questions that need to be answered before a fix is made: 1. What exactly can a string mca param contain? Can it have quotes or spaces or? 2. Which mca parameters should be forwarded? Should it be just the ones from the command line? From the environment? From config files? Tim Ralph Castain wrote: What changed is that we never passed mca params to the orted before - they always went to the app, but it's the orted that has the issue. There is a bug ticket thread on this subject - I forget the number immediately. Basically, the problem was that we cannot generally pass the local environment to the orteds when we launch them. However, people needed various mca params to get to the orteds to control their behavior. The only way to resolve that problem was to pass the params via the command line, which is what was done. Except for a very few cases, all of our mca params are single values that do not include spaces, so this is not a problem that is causing widespread issues. As I said, I already had to deal with one special case that didn't involve spaces, but did have special characters that required quoting, which identified the larger problem of dealing with quoted strings. I have no objection to a more general fix. Like I said in my note, though, the general fix will take a larger effort. If someone is willing to do so, that is fine with me - I was only offering solutions that would fill the interim time as I haven't heard anyone step up to say they would fix it anytime soon. Please feel free to jump in and volunteer! ;-) I'm willing to put the quotes around things if you will fix the mca cmd line parser to cleanly remove them on the other end. Ralph On 11/7/07 5:50 PM, "Tim Prins" wrote: I'm curious what changed to make this a problem. How were we passing mca param from the base to the app before, and why did it change? I think that options 1 & 2 below are no good, since we, in general, allow string mca params to have spaces (as far as I understand it). So a more general approach is needed. T
Re: [OMPI devel] Multiworld MCA parameter values broken
My, you are joining late! ;-)
The problem is with MCA params that take string arguments. If we pass the
following:
-mca foo "my beloved string"
on the command line of an orted, we get a value for foo that includes the
quote marks. I verified this rather painfully when attempting to pass a
command line mca param for a uri. I eventually had to add specific code to
clean the paramater up.
This appears to be somehow related to the precise method used to register
the param. For example, the following deprecated method works fine:
On the setup end:
opal_argv_append(argc, argv, "--gprreplica");
if (NULL != orte_process_info.gpr_replica_uri) {
contact_info = strdup(orte_process_info.gpr_replica_uri);
} else {
contact_info = orte_rml.get_contact_info();
}
asprintf(¶m, "\"%s\"", contact_info);
opal_argv_append(argc, argv, param);
And on the receiving end:
id = mca_base_param_register_string("gpr", "replica", "uri", NULL,
orte_process_info.gpr_replica_uri);
mca_base_param_lookup_string(id, &(orte_process_info.gpr_replica_uri));
mca_base_param_set_internal(id, true);
However, the following does NOT work cleanly:
On the setup end:
rml_uri = orte_rml.get_contact_info();
asprintf(¶m, "\"%s\"", rml_uri);
opal_argv_append(argc, argv, "--hnp-uri");
opal_argv_append(argc, argv, param);
On the receiving end:
mca_base_param_reg_string_name("orte", "hnp_uri",
"HNP contact info",
true, false, NULL, &uri);
Thereby necessitating the addition of the following code to clean it up:
if (NULL != uri) {
/* the uri value passed to us will have quote marks around it to
protect
* the value if passed on the command line. We must remove those
* to have a correct uri string
*/
if ('"' == uri[0]) {
/* if the first char is a quote, then so will the last one be */
ptr = &uri[1];
len = strlen(ptr) - 1;
} else {
ptr = &uri[0];
len = strlen(uri);
}
/* we have to copy the string by hand as strndup is a GNU extension
* and may not be generally available
*/
orte_process_info.my_hnp_uri = (char*)malloc(len+1);
for (i=0; i < len; i++) {
orte_process_info.my_hnp_uri[i] = ptr[i];
}
orte_process_info.my_hnp_uri[len] = '\0'; /* NULL terminate */
free(uri);
}
It was my understanding that you wanted us to move away from the deprecated
interface hence my comment that we cannot just quote all the strings as we
would have to add this code all over the place, or (better) fix
opal_cmd_line_parse.
Hope that helps
Ralph
On 11/19/07 7:01 AM, "Jeff Squyres" wrote:
> Sorry -- I'm just joining this conversation late: what's the problem
> with opal_cmd_line_parse?
>
> It should obey all quoting from shells, etc. I.e., it shouldn't care
> about tokens with special characters (to include spaces) because the
> shell divides all of that stuff up -- it just gets a char*[] that it
> treats as discrete tokens.
>
> Is it doing something wrong?
>
>
>
> On Nov 19, 2007, at 8:39 AM, Ralph H Castain wrote:
>
>> I'm not sure it is really necessary - the problem is solely within
>> opal_cmd_line_parse and (if someone can parse that code ;-)) is
>> truly simple
>> to fix. The overly long cmd line issue is due to a bug that Josh was
>> going
>> to look at (may already have done so while I was out of touch).
>>
>> Ralph
>>
>>
>>
>> On 11/9/07 5:10 AM, "Jeff Squyres" wrote:
>>
>>> Should there be another option for passing MCA parameters between
>>> processes, such as via stdin (or any file descriptor)? I.e., during
>>> the command line parsing to check for command line MCA params,
>>> perhaps
>>> a new argument could be introduced: -mcauri , where could
>>> be a few different forms:
>>>
>>> - file://stdin: (note the 2 //, not 3, so "stdin" would never
>>> conflict
>>> with a real file named /stdin) Read the parameters in off stdin.
>>>
>>> - rml://...rml contact info...: read in the MCA params via the RML
>>> (although I assume that reading via the RML would be *wy* to late
>>> during the MCA setup process -- I mentioned this option for
>>> completeness, even though I don't think it'll work)
>>>
>>> - ip://ipaddress:port: open a socket back and read the MCA params in
>>> over a socket. This could have some scalability issues...? But who
>>> knows; it could be tied into the hierarchical startup such that we
>>> wouldn't have to have an all-to-one connection scheme. Certainly it
>>> would cause scalability problems when paired with today's all-to-one
>>> RML connection scheme for the OOB.
>>>
>>> I'm not sure that the rml: and ip: schemes are worthwhile. Maybe a
>>> file://stdin kind of approach could work? Or perhaps some other kind
>>> of URI/IPC...? (I really haven't thought through the
Re: [OMPI devel] ORTE process name,, nodeid..
Yo Galen
I'm not aware of any continuing discussion to totally remove the process
name from ORTE - I believe we coalesced to redefining how the jobid was
established to a procedure that doesn't require a name server. This hasn't
come over to the trunk yet, but will in the next couple of months.
Adding a field to the process name is an unfortunately non-trivial exercise
as it hits a lot of places, including messing with the headers in the RML
and IOF - and as you know, nobody really wants to mess with that code.
One way to resolve this would be to add your call to get the pid and nodeid
of all procs in your job to the CNOS SDS component since every process has
to call that function anyway. If we go that route, then the question becomes
how best to expose that data to the OMPI layer. Creating an orte_proc_t just
for that purpose seems slightly overkill - can anyone think of another
reason to have such an object?
An alternative solution might be to incorporate the modex in the new OMPI
framework I was about to create anyway. This framework was intended to deal
with publish/lookup of OMPI data to support a variety of methods.
Originally, we had intended only to include support there for things
specifically related to MPI_Publish etc., but there is no reason we couldn't
generalize it to support the general exchange of process "how to connect to
me" info and include a modex API in it. I was figuring we would need two
immediate components in it anyway: an ORTE one for when we have full ORTE
support in the system, and a CNOS one that would...well, I guess just bark
and say "you can't do publish/lookup on a Cray". It would be simple to add
the modex stuff there, and makes some logical sense as well.
If that makes sense, we can implement the latter approach on the branch
where we are doing the next major ORTE revision - that's where I was going
to create the new framework anyway.
Ralph
On 11/16/07 10:24 PM, "Shipman, Galen M." wrote:
>
> I am doing some work on Cray's CNL to support shared memory. To support
> shared memory I need to know if processes are local or remote. For other
> systems we simply use the modex in ompi_proc_get_info to get the proc's
> nodeid. When using CNOS I don't need the modex to get a remote processes
> nodeid. In fact, I can obtain every processes pid and nodeid (nid/pid) via a
> single CNOS call.
>
> I have explored a couple of ways to populate the proc structures on the
> CRAY. One involves using #if's to do special things in ompi_proc_get_info. I
> don't like this. The second method involves adding a CNOS nameserver and
> modifying the orte_process_name_t to include the orte_nodeid_t so that the
> nameserver can populate all the info if it can. Prior to this change, the
> orte_nodeid_t was in ompi_proc_t, which doesn't make any sense to me, it is
> an orte level concept and it is only accessible in the ompi side. I also
> don't like adding orte_nodeid_t to orte_process_name_t as it really doesn't
> have anything to do with the a name.. I think it makes more sense to have an
> orte_proc_t.. Something like the following structure:
>
>
>
> struct orte_process_name_t {
> orte_jobid_t jobid; /**< Job number */
> orte_vpid_t vpid; /**< Process number */
> /** "nodeid" on which the proc resides */
> };
>
> Struct orte_proc_t {
> opal_list_item_t super;
> orte_process_name_t proc_name;
> orte_nodeid_t nid;
> };
>
> struct ompi_proc_t {
> orte_proc_t base;
> . Etc .
>
> };
>
>
> I know there is some talk about removing the process names,,, not sure how
> that fits in here but this is what makes sense to me given the current
> architecture. Any thoughts here?
>
>
> - Galen
>
>
>
>
Re: [OMPI devel] Multiworld MCA parameter values broken
I guess I don't see why this is an opal_cmd_line_parse() problem.
If you invoke executables through system(), then a shell is used and
quoting is necessary to preserve the individual string tokens (i.e.,
"my beloved string" would be passed to the application as one argv
token, without the quotes).
But if you're building up an array of argv and calling some form of
exec(), then no shell is involved and quoting should not be
necessary. Specifically:
opal_append_argv(&argc, &argv, "my beloved string");
will be passed as one string token to the application.
opal_cmd_line_parse() is passed an array of argv, meaning that the
command line have already been split into individual string tokens. I
guess the question is whether these come in directly from the
arguments to main() or whether we are getting a single string and
breaking it up into tokens. If the latter is true, then we need to re-
evaluate our break-into-tokens algorithm. I have a dim recollection
that these come in from the arguments to main(), though.
I guess I can see where this would get complicated for rsh/ssh
invocations, because *both* models are used. (i.e., you exec locally
but it turns into a system-like invocation on the remote side). In
this case, I think you'll need to quote extended strings (e.g., those
containing spaces) for the non-local invocations not not quote it for
local invocations.
On Nov 19, 2007, at 9:19 AM, Ralph H Castain wrote:
My, you are joining late! ;-)
The problem is with MCA params that take string arguments. If we
pass the following:
-mca foo "my beloved string"
on the command line of an orted, we get a value for foo that
includes the quote marks. I verified this rather painfully when
attempting to pass a command line mca param for a uri. I eventually
had to add specific code to clean the paramater up.
This appears to be somehow related to the precise method used to
register the param. For example, the following deprecated method
works fine:
On the setup end:
opal_argv_append(argc, argv, "--gprreplica");
if (NULL != orte_process_info.gpr_replica_uri) {
contact_info = strdup(orte_process_info.gpr_replica_uri);
} else {
contact_info = orte_rml.get_contact_info();
}
asprintf(¶m, "\"%s\"", contact_info);
opal_argv_append(argc, argv, param);
And on the receiving end:
id = mca_base_param_register_string("gpr", "replica", "uri",
NULL, orte_process_info.gpr_replica_uri);
mca_base_param_lookup_string(id,
&(orte_process_info.gpr_replica_uri));
mca_base_param_set_internal(id, true);
However, the following does NOT work cleanly:
On the setup end:
rml_uri = orte_rml.get_contact_info();
asprintf(¶m, "\"%s\"", rml_uri);
opal_argv_append(argc, argv, "--hnp-uri");
opal_argv_append(argc, argv, param);
On the receiving end:
mca_base_param_reg_string_name("orte", "hnp_uri",
"HNP contact info",
true, false, NULL, &uri);
Thereby necessitating the addition of the following code to clean it
up:
if (NULL != uri) {
/* the uri value passed to us will have quote marks around
it to protect
* the value if passed on the command line. We must remove
those
* to have a correct uri string
*/
if ('"' == uri[0]) {
/* if the first char is a quote, then so will the last
one be */
ptr = &uri[1];
len = strlen(ptr) - 1;
} else {
ptr = &uri[0];
len = strlen(uri);
}
/* we have to copy the string by hand as strndup is a GNU
extension
* and may not be generally available
*/
orte_process_info.my_hnp_uri = (char*)malloc(len+1);
for (i=0; i < len; i++) {
orte_process_info.my_hnp_uri[i] = ptr[i];
}
orte_process_info.my_hnp_uri[len] = '\0'; /* NULL terminate
*/
free(uri);
}
It was my understanding that you wanted us to move away from the
deprecated interface – hence my comment that we cannot just quote
all the strings as we would have to add this code all over the
place, or (better) fix opal_cmd_line_parse.
Hope that helps
Ralph
On 11/19/07 7:01 AM, "Jeff Squyres" wrote:
> Sorry -- I'm just joining this conversation late: what's the problem
> with opal_cmd_line_parse?
>
> It should obey all quoting from shells, etc. I.e., it shouldn't
care
> about tokens with special characters (to include spaces) because the
> shell divides all of that stuff up -- it just gets a char*[] that it
> treats as discrete tokens.
>
> Is it doing something wrong?
>
>
>
> On Nov 19, 2007, at 8:39 AM, Ralph H Castain wrote:
>
>> I'm not sure it is really necessary - the problem is solely within
>> opal_cmd_line_parse and (if someone can parse that code ;-)) is
>> truly simple
>> to fix. The overly long cmd line issue is due to a bug that
Re: [OMPI devel] Multiworld MCA parameter values broken
Unfortunately, it -is- a significant problem when passing the params on to
the orteds, as Tim has eloquently pointed out in his original posting. I
guess I don't see why it would be a significant problem to just have
opal_cmd_line_parse do the "right thing" - if a string param is quoted, then
just remove the quotes.
Problem solved, and I wouldn't have to add a bunch of code to figure out
when to assemble argv arrays in different ways.
On 11/19/07 7:52 AM, "Jeff Squyres" wrote:
> I guess I don't see why this is an opal_cmd_line_parse() problem.
>
> If you invoke executables through system(), then a shell is used and
> quoting is necessary to preserve the individual string tokens (i.e.,
> "my beloved string" would be passed to the application as one argv
> token, without the quotes).
>
> But if you're building up an array of argv and calling some form of
> exec(), then no shell is involved and quoting should not be
> necessary. Specifically:
>
> opal_append_argv(&argc, &argv, "my beloved string");
>
> will be passed as one string token to the application.
>
> opal_cmd_line_parse() is passed an array of argv, meaning that the
> command line have already been split into individual string tokens. I
> guess the question is whether these come in directly from the
> arguments to main() or whether we are getting a single string and
> breaking it up into tokens. If the latter is true, then we need to re-
> evaluate our break-into-tokens algorithm. I have a dim recollection
> that these come in from the arguments to main(), though.
>
> I guess I can see where this would get complicated for rsh/ssh
> invocations, because *both* models are used. (i.e., you exec locally
> but it turns into a system-like invocation on the remote side). In
> this case, I think you'll need to quote extended strings (e.g., those
> containing spaces) for the non-local invocations not not quote it for
> local invocations.
>
>
>
>
> On Nov 19, 2007, at 9:19 AM, Ralph H Castain wrote:
>
>> My, you are joining late! ;-)
>>
>> The problem is with MCA params that take string arguments. If we
>> pass the following:
>>
>> -mca foo "my beloved string"
>>
>> on the command line of an orted, we get a value for foo that
>> includes the quote marks. I verified this rather painfully when
>> attempting to pass a command line mca param for a uri. I eventually
>> had to add specific code to clean the paramater up.
>>
>> This appears to be somehow related to the precise method used to
>> register the param. For example, the following deprecated method
>> works fine:
>>
>> On the setup end:
>>
>> opal_argv_append(argc, argv, "--gprreplica");
>> if (NULL != orte_process_info.gpr_replica_uri) {
>> contact_info = strdup(orte_process_info.gpr_replica_uri);
>> } else {
>> contact_info = orte_rml.get_contact_info();
>> }
>> asprintf(¶m, "\"%s\"", contact_info);
>> opal_argv_append(argc, argv, param);
>>
>> And on the receiving end:
>> id = mca_base_param_register_string("gpr", "replica", "uri",
>> NULL, orte_process_info.gpr_replica_uri);
>> mca_base_param_lookup_string(id,
>> &(orte_process_info.gpr_replica_uri));
>> mca_base_param_set_internal(id, true);
>>
>>
>> However, the following does NOT work cleanly:
>>
>> On the setup end:
>>rml_uri = orte_rml.get_contact_info();
>> asprintf(¶m, "\"%s\"", rml_uri);
>> opal_argv_append(argc, argv, "--hnp-uri");
>> opal_argv_append(argc, argv, param);
>>
>> On the receiving end:
>> mca_base_param_reg_string_name("orte", "hnp_uri",
>>"HNP contact info",
>>true, false, NULL, &uri);
>>
>> Thereby necessitating the addition of the following code to clean it
>> up:
>> if (NULL != uri) {
>> /* the uri value passed to us will have quote marks around
>> it to protect
>> * the value if passed on the command line. We must remove
>> those
>> * to have a correct uri string
>> */
>>if ('"' == uri[0]) {
>> /* if the first char is a quote, then so will the last
>> one be */
>>ptr = &uri[1];
>> len = strlen(ptr) - 1;
>> } else {
>> ptr = &uri[0];
>> len = strlen(uri);
>> }
>>
>> /* we have to copy the string by hand as strndup is a GNU
>> extension
>> * and may not be generally available
>> */
>>orte_process_info.my_hnp_uri = (char*)malloc(len+1);
>> for (i=0; i < len; i++) {
>> orte_process_info.my_hnp_uri[i] = ptr[i];
>> }
>> orte_process_info.my_hnp_uri[len] = '\0'; /* NULL terminate
>> */
>>free(uri);
>>
>> }
>>
>> It was my understanding that you wanted us to move away from the
>> deprecated interface hence my comment that we cannot just quote
>> all the strings as we would have to add this code all over the
>> place, or (better) fix opal_cmd_line_par
Re: [OMPI devel] IB/OpenFabrics pow wow
Sun would like to be represented at this meeting. Mon 10AM might be a problem for Sun, any of the other times are good. -DON Jeff Squyres wrote: Based on discussions at SC, I think it is time to have another IB/ OpenFabrics OMPI pow wow teleconference. It would be good to get status updates on what everyone is doing in terms of OpenFabrics kinds of things in OMPI so that we can coordinate the work properly. I'd like to strike a balance of getting everyone relevant involved but also keeping the group small so that we can have a meaningful discussion and come up with good work plans for moving forward. A first/tentative list of attendees include: - Cisco - Voltaire - LLNL - OGC (Chelsio) - ?ORNL? - ?Sun? Agenda will be short -- I'm thinking a 1 hour teleconference in the last week of December (after the US Thanksgiving holiday): 1. each organization's status update / development plans for the next 2 months 2. possible directions forward beyond 2 months (there are some pre- ideas floating around that are not yet solid enough/suitable to push out to public consumption -- I think we need to "firm them up" a bit first) I propose the following dates/times in priority order -- please reply if you cannot make these times (teleconference bridge info will be sent after a date/time is decided upon): 1. Mon, 26 Nov, 10am US East, 7am US Pacific, 5pm Israel 2. Mon, 26 Nov, 11am US East, 8am US Pacific, 6pm Israel 3. Thu, 29 Nov, 10am US East, 7am US Pacific, 5pm Israel 4. Thu, 29 Nov, 11am US East, 8am US Pacific, 6pm Israel 5. Fri, 30 Nov, 10am US East, 7am US Pacific, 5pm Israel 6. Fri, 30 Nov, 11am US East, 8am US Pacific, 6pm Israel (I cannot do Tue/Wed that week) Thanks.
Re: [OMPI devel] Multiworld MCA parameter values broken
On Nov 19, 2007, at 10:01 AM, Ralph H Castain wrote:
Unfortunately, it -is- a significant problem when passing the params
on to
the orteds, as Tim has eloquently pointed out in his original
posting. I
guess I don't see why it would be a significant problem to just have
opal_cmd_line_parse do the "right thing" - if a string param is
quoted, then
just remove the quotes.
How would opal_cmd_line_parse know when to remove quotes and when not
remove quotes? I.e., what if an MCA param (or some other value, such
as a user application argv) required quotes? This is not uncommon for
the wrapper compilers, for example (passing -D's to the C preprocessor
that take string arguments).
AFAIK, opal_cmd_line_parse() does exactly what I intended it to. :-)
It performs no shell-like quoting because it expects to be given a
list of "final" string tokens.
Problem solved, and I wouldn't have to add a bunch of code to figure
out
when to assemble argv arrays in different ways.
Er... I don't understand: I joined this thread because I thought there
was still something to fix...?
I would be against changing opal_cmd_line_parse() in the manner that
you have described because it will break other things (e.g., the
wrapper compilers).
On 11/19/07 7:52 AM, "Jeff Squyres" wrote:
I guess I don't see why this is an opal_cmd_line_parse() problem.
If you invoke executables through system(), then a shell is used and
quoting is necessary to preserve the individual string tokens (i.e.,
"my beloved string" would be passed to the application as one argv
token, without the quotes).
But if you're building up an array of argv and calling some form of
exec(), then no shell is involved and quoting should not be
necessary. Specifically:
opal_append_argv(&argc, &argv, "my beloved string");
will be passed as one string token to the application.
opal_cmd_line_parse() is passed an array of argv, meaning that the
command line have already been split into individual string
tokens. I
guess the question is whether these come in directly from the
arguments to main() or whether we are getting a single string and
breaking it up into tokens. If the latter is true, then we need to
re-
evaluate our break-into-tokens algorithm. I have a dim recollection
that these come in from the arguments to main(), though.
I guess I can see where this would get complicated for rsh/ssh
invocations, because *both* models are used. (i.e., you exec locally
but it turns into a system-like invocation on the remote side). In
this case, I think you'll need to quote extended strings (e.g., those
containing spaces) for the non-local invocations not not quote it for
local invocations.
On Nov 19, 2007, at 9:19 AM, Ralph H Castain wrote:
My, you are joining late! ;-)
The problem is with MCA params that take string arguments. If we
pass the following:
-mca foo "my beloved string"
on the command line of an orted, we get a value for foo that
includes the quote marks. I verified this rather painfully when
attempting to pass a command line mca param for a uri. I eventually
had to add specific code to clean the paramater up.
This appears to be somehow related to the precise method used to
register the param. For example, the following deprecated method
works fine:
On the setup end:
opal_argv_append(argc, argv, "--gprreplica");
if (NULL != orte_process_info.gpr_replica_uri) {
contact_info = strdup(orte_process_info.gpr_replica_uri);
} else {
contact_info = orte_rml.get_contact_info();
}
asprintf(¶m, "\"%s\"", contact_info);
opal_argv_append(argc, argv, param);
And on the receiving end:
id = mca_base_param_register_string("gpr", "replica", "uri",
NULL, orte_process_info.gpr_replica_uri);
mca_base_param_lookup_string(id,
&(orte_process_info.gpr_replica_uri));
mca_base_param_set_internal(id, true);
However, the following does NOT work cleanly:
On the setup end:
rml_uri = orte_rml.get_contact_info();
asprintf(¶m, "\"%s\"", rml_uri);
opal_argv_append(argc, argv, "--hnp-uri");
opal_argv_append(argc, argv, param);
On the receiving end:
mca_base_param_reg_string_name("orte", "hnp_uri",
"HNP contact info",
true, false, NULL, &uri);
Thereby necessitating the addition of the following code to clean it
up:
if (NULL != uri) {
/* the uri value passed to us will have quote marks around
it to protect
* the value if passed on the command line. We must remove
those
* to have a correct uri string
*/
if ('"' == uri[0]) {
/* if the first char is a quote, then so will the last
one be */
ptr = &uri[1];
len = strlen(ptr) - 1;
} else {
ptr = &uri[0];
len = strlen(uri);
}
/* we have to copy the string by hand as strndup is a GNU
extension
* and may not be generally available
*/
or
Re: [OMPI devel] Multiworld MCA parameter values broken
Hmmm...okay, I get the message: if it is going to be fixed, then I have to
fix it on my end. ;-) Which means it ain't happening anytime soon, so Tim P
is the one left in the lurch.
Sorry Tim! Perhaps you can come up with a compromise that re-enables what
you want to do.
Ralph
On 11/19/07 8:11 AM, "Jeff Squyres" wrote:
> On Nov 19, 2007, at 10:01 AM, Ralph H Castain wrote:
>
>> Unfortunately, it -is- a significant problem when passing the params
>> on to
>> the orteds, as Tim has eloquently pointed out in his original
>> posting. I
>> guess I don't see why it would be a significant problem to just have
>> opal_cmd_line_parse do the "right thing" - if a string param is
>> quoted, then
>> just remove the quotes.
>
> How would opal_cmd_line_parse know when to remove quotes and when not
> remove quotes? I.e., what if an MCA param (or some other value, such
> as a user application argv) required quotes? This is not uncommon for
> the wrapper compilers, for example (passing -D's to the C preprocessor
> that take string arguments).
>
> AFAIK, opal_cmd_line_parse() does exactly what I intended it to. :-)
> It performs no shell-like quoting because it expects to be given a
> list of "final" string tokens.
>
>> Problem solved, and I wouldn't have to add a bunch of code to figure
>> out
>> when to assemble argv arrays in different ways.
>
> Er... I don't understand: I joined this thread because I thought there
> was still something to fix...?
>
> I would be against changing opal_cmd_line_parse() in the manner that
> you have described because it will break other things (e.g., the
> wrapper compilers).
>
>
>
>>
>>
>> On 11/19/07 7:52 AM, "Jeff Squyres" wrote:
>>
>>> I guess I don't see why this is an opal_cmd_line_parse() problem.
>>>
>>> If you invoke executables through system(), then a shell is used and
>>> quoting is necessary to preserve the individual string tokens (i.e.,
>>> "my beloved string" would be passed to the application as one argv
>>> token, without the quotes).
>>>
>>> But if you're building up an array of argv and calling some form of
>>> exec(), then no shell is involved and quoting should not be
>>> necessary. Specifically:
>>>
>>> opal_append_argv(&argc, &argv, "my beloved string");
>>>
>>> will be passed as one string token to the application.
>>>
>>> opal_cmd_line_parse() is passed an array of argv, meaning that the
>>> command line have already been split into individual string
>>> tokens. I
>>> guess the question is whether these come in directly from the
>>> arguments to main() or whether we are getting a single string and
>>> breaking it up into tokens. If the latter is true, then we need to
>>> re-
>>> evaluate our break-into-tokens algorithm. I have a dim recollection
>>> that these come in from the arguments to main(), though.
>>>
>>> I guess I can see where this would get complicated for rsh/ssh
>>> invocations, because *both* models are used. (i.e., you exec locally
>>> but it turns into a system-like invocation on the remote side). In
>>> this case, I think you'll need to quote extended strings (e.g., those
>>> containing spaces) for the non-local invocations not not quote it for
>>> local invocations.
>>>
>>>
>>>
>>>
>>> On Nov 19, 2007, at 9:19 AM, Ralph H Castain wrote:
>>>
My, you are joining late! ;-)
The problem is with MCA params that take string arguments. If we
pass the following:
-mca foo "my beloved string"
on the command line of an orted, we get a value for foo that
includes the quote marks. I verified this rather painfully when
attempting to pass a command line mca param for a uri. I eventually
had to add specific code to clean the paramater up.
This appears to be somehow related to the precise method used to
register the param. For example, the following deprecated method
works fine:
On the setup end:
opal_argv_append(argc, argv, "--gprreplica");
if (NULL != orte_process_info.gpr_replica_uri) {
contact_info = strdup(orte_process_info.gpr_replica_uri);
} else {
contact_info = orte_rml.get_contact_info();
}
asprintf(¶m, "\"%s\"", contact_info);
opal_argv_append(argc, argv, param);
And on the receiving end:
id = mca_base_param_register_string("gpr", "replica", "uri",
NULL, orte_process_info.gpr_replica_uri);
mca_base_param_lookup_string(id,
&(orte_process_info.gpr_replica_uri));
mca_base_param_set_internal(id, true);
However, the following does NOT work cleanly:
On the setup end:
rml_uri = orte_rml.get_contact_info();
asprintf(¶m, "\"%s\"", rml_uri);
opal_argv_append(argc, argv, "--hnp-uri");
opal_argv_append(argc, argv, param);
On the receiving end:
mca_base_param_reg_string_name("orte", "hnp_uri",
"HNP cont
Re: [OMPI devel] Multiworld MCA parameter values broken
On Nov 19, 2007, at 10:15 AM, Ralph H Castain wrote:
Hmmm...okay, I get the message: if it is going to be fixed, then I
have to
fix it on my end. ;-) Which means it ain't happening anytime soon,
so Tim P
is the one left in the lurch.
Sorry Tim! Perhaps you can come up with a compromise that re-enables
what
you want to do.
Yep -- sorry Tim. :-(
Is this only a problem for the rsh/ssh launcher? I *think* that
should be the only launcher affected -- all others use raw argv string
arrays, right? If so, we already have a local/non-local flag in the
rsh launcher, right? I agree it kinda sucks, but it may be necessary
to construct argv differently between local/non-local (which we
already do, right?).
On 11/19/07 8:11 AM, "Jeff Squyres" wrote:
On Nov 19, 2007, at 10:01 AM, Ralph H Castain wrote:
Unfortunately, it -is- a significant problem when passing the params
on to
the orteds, as Tim has eloquently pointed out in his original
posting. I
guess I don't see why it would be a significant problem to just have
opal_cmd_line_parse do the "right thing" - if a string param is
quoted, then
just remove the quotes.
How would opal_cmd_line_parse know when to remove quotes and when not
remove quotes? I.e., what if an MCA param (or some other value, such
as a user application argv) required quotes? This is not uncommon
for
the wrapper compilers, for example (passing -D's to the C
preprocessor
that take string arguments).
AFAIK, opal_cmd_line_parse() does exactly what I intended it to. :-)
It performs no shell-like quoting because it expects to be given a
list of "final" string tokens.
Problem solved, and I wouldn't have to add a bunch of code to figure
out
when to assemble argv arrays in different ways.
Er... I don't understand: I joined this thread because I thought
there
was still something to fix...?
I would be against changing opal_cmd_line_parse() in the manner that
you have described because it will break other things (e.g., the
wrapper compilers).
On 11/19/07 7:52 AM, "Jeff Squyres" wrote:
I guess I don't see why this is an opal_cmd_line_parse() problem.
If you invoke executables through system(), then a shell is used
and
quoting is necessary to preserve the individual string tokens
(i.e.,
"my beloved string" would be passed to the application as one argv
token, without the quotes).
But if you're building up an array of argv and calling some form of
exec(), then no shell is involved and quoting should not be
necessary. Specifically:
opal_append_argv(&argc, &argv, "my beloved string");
will be passed as one string token to the application.
opal_cmd_line_parse() is passed an array of argv, meaning that the
command line have already been split into individual string
tokens. I
guess the question is whether these come in directly from the
arguments to main() or whether we are getting a single string and
breaking it up into tokens. If the latter is true, then we need to
re-
evaluate our break-into-tokens algorithm. I have a dim
recollection
that these come in from the arguments to main(), though.
I guess I can see where this would get complicated for rsh/ssh
invocations, because *both* models are used. (i.e., you exec
locally
but it turns into a system-like invocation on the remote side). In
this case, I think you'll need to quote extended strings (e.g.,
those
containing spaces) for the non-local invocations not not quote it
for
local invocations.
On Nov 19, 2007, at 9:19 AM, Ralph H Castain wrote:
My, you are joining late! ;-)
The problem is with MCA params that take string arguments. If we
pass the following:
-mca foo "my beloved string"
on the command line of an orted, we get a value for foo that
includes the quote marks. I verified this rather painfully when
attempting to pass a command line mca param for a uri. I
eventually
had to add specific code to clean the paramater up.
This appears to be somehow related to the precise method used to
register the param. For example, the following deprecated method
works fine:
On the setup end:
opal_argv_append(argc, argv, "--gprreplica");
if (NULL != orte_process_info.gpr_replica_uri) {
contact_info = strdup(orte_process_info.gpr_replica_uri);
} else {
contact_info = orte_rml.get_contact_info();
}
asprintf(¶m, "\"%s\"", contact_info);
opal_argv_append(argc, argv, param);
And on the receiving end:
id = mca_base_param_register_string("gpr", "replica", "uri",
NULL, orte_process_info.gpr_replica_uri);
mca_base_param_lookup_string(id,
&(orte_process_info.gpr_replica_uri));
mca_base_param_set_internal(id, true);
However, the following does NOT work cleanly:
On the setup end:
rml_uri = orte_rml.get_contact_info();
asprintf(¶m, "\"%s\"", rml_uri);
opal_argv_append(argc, argv, "--hnp-uri");
opal_argv_append(argc, argv, param);
On the receiving end:
mca_base_param_reg_string_name("orte", "hnp_uri",
"HNP co
Re: [OMPI devel] Multiworld MCA parameter values broken
On 11/19/07 8:19 AM, "Jeff Squyres" wrote: > On Nov 19, 2007, at 10:15 AM, Ralph H Castain wrote: > >> Hmmm...okay, I get the message: if it is going to be fixed, then I >> have to >> fix it on my end. ;-) Which means it ain't happening anytime soon, >> so Tim P >> is the one left in the lurch. >> >> Sorry Tim! Perhaps you can come up with a compromise that re-enables >> what >> you want to do. > > Yep -- sorry Tim. :-( > > Is this only a problem for the rsh/ssh launcher? I *think* that > should be the only launcher affected -- all others use raw argv string > arrays, right? If so, we already have a local/non-local flag in the > rsh launcher, right? I agree it kinda sucks, but it may be necessary > to construct argv differently between local/non-local (which we > already do, right?). > Nope, not any more - you can't do a local rsh since mpirun is acting as the local daemon. And we only have one way of constructing argv arrays for orteds as they are always non-local. I believe the problem is occurring for all environments due to the reconstruction of the argv array to include the cmd line mca params. Tim will have to clarify for his application - I know I encountered the problem on SLURM, TM, and rsh for my application, but I fixed that special case once I discovered what was going on. > > >> On 11/19/07 8:11 AM, "Jeff Squyres" wrote: >> >>> On Nov 19, 2007, at 10:01 AM, Ralph H Castain wrote: >>> Unfortunately, it -is- a significant problem when passing the params on to the orteds, as Tim has eloquently pointed out in his original posting. I guess I don't see why it would be a significant problem to just have opal_cmd_line_parse do the "right thing" - if a string param is quoted, then just remove the quotes. >>> >>> How would opal_cmd_line_parse know when to remove quotes and when not >>> remove quotes? I.e., what if an MCA param (or some other value, such >>> as a user application argv) required quotes? This is not uncommon >>> for >>> the wrapper compilers, for example (passing -D's to the C >>> preprocessor >>> that take string arguments). >>> >>> AFAIK, opal_cmd_line_parse() does exactly what I intended it to. :-) >>> It performs no shell-like quoting because it expects to be given a >>> list of "final" string tokens. >>> Problem solved, and I wouldn't have to add a bunch of code to figure out when to assemble argv arrays in different ways. >>> >>> Er... I don't understand: I joined this thread because I thought >>> there >>> was still something to fix...? >>> >>> I would be against changing opal_cmd_line_parse() in the manner that >>> you have described because it will break other things (e.g., the >>> wrapper compilers). >>> >>> >>> On 11/19/07 7:52 AM, "Jeff Squyres" wrote: > I guess I don't see why this is an opal_cmd_line_parse() problem. > > If you invoke executables through system(), then a shell is used > and > quoting is necessary to preserve the individual string tokens > (i.e., > "my beloved string" would be passed to the application as one argv > token, without the quotes). > > But if you're building up an array of argv and calling some form of > exec(), then no shell is involved and quoting should not be > necessary. Specifically: > >opal_append_argv(&argc, &argv, "my beloved string"); > > will be passed as one string token to the application. > > opal_cmd_line_parse() is passed an array of argv, meaning that the > command line have already been split into individual string > tokens. I > guess the question is whether these come in directly from the > arguments to main() or whether we are getting a single string and > breaking it up into tokens. If the latter is true, then we need to > re- > evaluate our break-into-tokens algorithm. I have a dim > recollection > that these come in from the arguments to main(), though. > > I guess I can see where this would get complicated for rsh/ssh > invocations, because *both* models are used. (i.e., you exec > locally > but it turns into a system-like invocation on the remote side). In > this case, I think you'll need to quote extended strings (e.g., > those > containing spaces) for the non-local invocations not not quote it > for > local invocations. > > > > > On Nov 19, 2007, at 9:19 AM, Ralph H Castain wrote: > >> My, you are joining late! ;-) >> >> The problem is with MCA params that take string arguments. If we >> pass the following: >> >> -mca foo "my beloved string" >> >> on the command line of an orted, we get a value for foo that >> includes the quote marks. I verified this rather painfully when >> attempting to pass a command line mca param for a uri. I >> eventually >> had to add specific code
Re: [OMPI devel] IB/OpenFabrics pow wow
I am off all next week, but I will try to call in anyway.. - Galen On 11/19/07 10:09 AM, "Don Kerr" wrote: > Sun would like to be represented at this meeting. Mon 10AM might be a > problem for Sun, any of the other times are good. > > -DON > > Jeff Squyres wrote: > >> Based on discussions at SC, I think it is time to have another IB/ >> OpenFabrics OMPI pow wow teleconference. It would be good to get >> status updates on what everyone is doing in terms of OpenFabrics kinds >> of things in OMPI so that we can coordinate the work properly. I'd >> like to strike a balance of getting everyone relevant involved but >> also keeping the group small so that we can have a meaningful >> discussion and come up with good work plans for moving forward. A >> first/tentative list of attendees include: >> >> - Cisco >> - Voltaire >> - LLNL >> - OGC (Chelsio) >> - ?ORNL? >> - ?Sun? >> >> Agenda will be short -- I'm thinking a 1 hour teleconference in the >> last week of December (after the US Thanksgiving holiday): >> >> 1. each organization's status update / development plans for the next >> 2 months >> >> 2. possible directions forward beyond 2 months (there are some pre- >> ideas floating around that are not yet solid enough/suitable to push >> out to public consumption -- I think we need to "firm them up" a bit >> first) >> >> I propose the following dates/times in priority order -- please reply >> if you cannot make these times (teleconference bridge info will be >> sent after a date/time is decided upon): >> >> 1. Mon, 26 Nov, 10am US East, 7am US Pacific, 5pm Israel >> 2. Mon, 26 Nov, 11am US East, 8am US Pacific, 6pm Israel >> 3. Thu, 29 Nov, 10am US East, 7am US Pacific, 5pm Israel >> 4. Thu, 29 Nov, 11am US East, 8am US Pacific, 6pm Israel >> 5. Fri, 30 Nov, 10am US East, 7am US Pacific, 5pm Israel >> 6. Fri, 30 Nov, 11am US East, 8am US Pacific, 6pm Israel >> >> (I cannot do Tue/Wed that week) >> >> Thanks. >> >> >> > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] IB/OpenFabrics pow wow
On Fri, Nov 16, 2007 at 11:36:39AM -0800, Jeff Squyres wrote: > 1. Mon, 26 Nov, 10am US East, 7am US Pacific, 5pm Israel > 2. Mon, 26 Nov, 11am US East, 8am US Pacific, 6pm Israel > 3. Thu, 29 Nov, 10am US East, 7am US Pacific, 5pm Israel > 4. Thu, 29 Nov, 11am US East, 8am US Pacific, 6pm Israel > 5. Fri, 30 Nov, 10am US East, 7am US Pacific, 5pm Israel > 6. Fri, 30 Nov, 11am US East, 8am US Pacific, 6pm Israel > 1,2,3 or 4 are OK with me. Friday is not a working day in Israel. -- Gleb.
Re: [OMPI devel] IB/OpenFabrics pow wow
Friday -- right, duh. My bad. So with everyone's replies so far, I think we're down to: 2. Mon, 26 Nov, 11am US East, 8am US Pacific, 6pm Israel 3. Thu, 29 Nov, 10am US East, 7am US Pacific, 5pm Israel 4. Thu, 29 Nov, 11am US East, 8am US Pacific, 6pm Israel If we get no more requirements, I'll arbitrarily pick one of these times. On Nov 19, 2007, at 11:05 AM, Gleb Natapov wrote: On Fri, Nov 16, 2007 at 11:36:39AM -0800, Jeff Squyres wrote: 1. Mon, 26 Nov, 10am US East, 7am US Pacific, 5pm Israel 2. Mon, 26 Nov, 11am US East, 8am US Pacific, 6pm Israel 3. Thu, 29 Nov, 10am US East, 7am US Pacific, 5pm Israel 4. Thu, 29 Nov, 11am US East, 8am US Pacific, 6pm Israel 5. Fri, 30 Nov, 10am US East, 7am US Pacific, 5pm Israel 6. Fri, 30 Nov, 11am US East, 8am US Pacific, 6pm Israel 1,2,3 or 4 are OK with me. Friday is not a working day in Israel. -- Gleb. ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres Cisco Systems
Re: [OMPI devel] IB/OpenFabrics pow wow
Jeff Squyres wrote: Friday -- right, duh. My bad. So with everyone's replies so far, I think we're down to: 2. Mon, 26 Nov, 11am US East, 8am US Pacific, 6pm Israel 3. Thu, 29 Nov, 10am US East, 7am US Pacific, 5pm Israel 4. Thu, 29 Nov, 11am US East, 8am US Pacific, 6pm Israel 3 or 4 will work for us. Jon and I aren't available mon 11/26.
Re: [OMPI devel] initial SCTP BTL commit comments?
Are there API functions or data structures that can be used to determine if the 1-to-many model is supported on the system? More specifically: can you have your configure.m4 script check to see if the current system a) supports SCTP, and b) if yes, if it supports 1-to-many? This kind of checking would theoretically allow running on Solaris, but automatically default to the 1-to-1 mode (if your BTL supports that). This also falls in-line with the autoconf mantra: test for the desired behavior, not the desired platform (because the list of supported platforms may change over time). :-) On Nov 14, 2007, at 1:17 PM, Brad Penoff wrote: On Nov 14, 2007 5:11 AM, Terry Dontje wrote: Brad Penoff wrote: On Nov 12, 2007 3:26 AM, Jeff Squyres wrote: I have no objections to bringing this into the trunk, but I agree that an .ompi_ignore is probably a good idea at first. I'll try to cook up a commit soon then! One question that I'd like to have answered is how OMPI decides whether to use the SCTP BTL or not. If there are SCTP stacks available by default in Linux and OS X -- but their performance may be sub-optimal and/or buggy, we may want to have the SCTP BTL only activated if the user explicitly asks for it. Open MPI is very concerned with "out of the box" behavior -- we need to ensure that "mpirun a.out" will "just work" on all of our supported platforms. Just to make a few things explicit... Things would only work out of the box on FreeBSD, and there the stack is very good. We have less experience with the Linux stack but hope the availability of and SCTP BTL will help encourage its use by us and others. Now it is a module by default (loaded with "modprobe sctp") but the actual SCTP sockets extension API needs to be downloaded and installed separately. The so-called lksctp-tools can be obtained here: http://sourceforge.net/project/showfiles.php?group_id=26529 The OS X stack does not come by default but instead is a kernel extension: http://sctp.fh-muenster.de/sctp-nke.html I haven't yet started this testing but intend to soon. As of now though, the supplied configure.m4 does not try to even build the component on Mac OS X. So in my opinion, things in the configure scripts should be fine the way the are since only FreeBSD stack (which we have confidence in) will try to work out of the box; the others require the user to install things. Greetings, I am gathering from the text above you haven't tried your BTL on Solaris at all. The short answer to that is correct, we haven't tried the Open MPI SCTP BTL yet on Solaris. In fact, the configure.m4 file checks the $host value and only tries to build if it's on Linux or a BSD variant. Mac OS X uses the same code as BSD but I have only just got my hands on a machine so even it hasn't been tested yet; Solaris remains on the TODO list. However, there's a slightly longer answer... After a series of emails with the Sun SCTP people (sctp-questi...@sun.com but mostly Kacheong Poon) a year ago, I learned SCTP support is within Solaris 10 by default. In general, SCTP supports its own socket API, in addition to the standard Berkeley sockets API; the SCTP-specific sockets API unlocks some of SCTP's newer features (e.g, multistreaming). We make use of this SCTP-specific sockets API. The Solaris stack (as of a year ago) made certain assumptions about the SCTP-specific sockets API. I'm just looking back on those emails now to refresh my memory... it looks like on the Solaris stack as of Nov 2006, it did not allow the use one-to-many sockets (the current default in our BTL) together with the sctp_sendmsg call. They mentioned an alternative just we didn't have the time to explore it. I'm not sure if this has changed on the Solaris stack within the past year... I never got the time to revisit this. In the past, we had mostly used the one-to-many socket (with our LAM and MPICH2 versions). One unique thing about this Open MPI SCTP BTL is that there is also a choice to make use of (the more TCP-like) one-to-one socket style. The socket style used by the SCTP BTL is adjustable with the MCA parameter btl_sctp_if_11 (if set to 1, it uses 1-1 sockets; by default it is 0 and uses 1-many). I've never used one-to-one sockets on the Solaris stack, but it may have a better chance of working (also one-to-many may work now; I haven't kept up-to-date). We also noticed that on Solaris we had to do some things a little different with iovec's because the struct msghdr (used by sendmsg) had no msg_control field; to get around this, we had to pack the iovec's contents into a buffer and send that buffer instead of using the iovec directly. Anyway, hope this fully answers your questions. In general, it'd be nice if we have the time/assistance to add in Solaris support eventually. brad --td ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/de
Re: [OMPI devel] ORTE process name,, nodeid..
On Monday 19 November 2007 09:42:21 am Ralph H Castain wrote:
> An alternative solution might be to incorporate the modex in the new OMPI
> framework I was about to create anyway. This framework was intended to deal
> with publish/lookup of OMPI data to support a variety of methods.
> Originally, we had intended only to include support there for things
> specifically related to MPI_Publish etc., but there is no reason we
> couldn't generalize it to support the general exchange of process "how to
> connect to me" info and include a modex API in it. I was figuring we would
> need two immediate components in it anyway: an ORTE one for when we have
> full ORTE support in the system, and a CNOS one that would...well, I guess
> just bark and say "you can't do publish/lookup on a Cray". It would be
> simple to add the modex stuff there, and makes some logical sense as well.
I think this approach is fundamentally flawed. Our frameworks are designed to
abstract out something, to allow for multiple implementations. However, doing
this would put two completely different things (the modex and the MPI
pub/sub) together in one framework. While this may be convenient for the
cray, it would be very inconvenient for someone who wanted to do the MPI
pub/sub via a ldap server (as has been discussed). The key here is that MPI
pub/sub is for very small amounts of data, accessed infrequently and in a
non-performance-critical manner, whereas the modex is for rather large
amounts of information (on big jobs) that has to be exchanged efficiently.
Before anyone misunderstands, I am *not* proposing that we add a modex
framework to ompi. Rather, I think this is a case where the RSL could make
things really easy.
The RSL defines a process attribute system. One of the original ideas (later
retracted, but now that I think about it I may re-add it) was to have some
predefined attribute keys, that the runtime would set so we could look up
information about any process.
So in the case of the cray, the rsl_init function would query to get all the
info it wants, and then populate the info into its (local) process attribute
data store.
In other systems each process would set the information in rsl_init and it
would be exchanged in the normal modex method.
Then, the information would be looked up (locally) using the 'get' function on
both platforms.
Simple, eh?
As an alternative to this, I think we could apply these same ideas into a
specialized ORTE system, but it would not be as clean, and would tie our
system closer to ORTE. I am not going to argue whether this is good or bad,
but I am just mentioning it as a consequence.
Tim
>
> If that makes sense, we can implement the latter approach on the branch
> where we are doing the next major ORTE revision - that's where I was going
> to create the new framework anyway.
>
> Ralph
>
> On 11/16/07 10:24 PM, "Shipman, Galen M." wrote:
> > I am doing some work on Cray's CNL to support shared memory. To support
> > shared memory I need to know if processes are local or remote. For other
> > systems we simply use the modex in ompi_proc_get_info to get the proc's
> > nodeid. When using CNOS I don't need the modex to get a remote processes
> > nodeid. In fact, I can obtain every processes pid and nodeid (nid/pid)
> > via a single CNOS call.
> >
> > I have explored a couple of ways to populate the proc structures on the
> > CRAY. One involves using #if's to do special things in
> > ompi_proc_get_info. I don't like this. The second method involves adding
> > a CNOS nameserver and modifying the orte_process_name_t to include the
> > orte_nodeid_t so that the nameserver can populate all the info if it can.
> > Prior to this change, the orte_nodeid_t was in ompi_proc_t, which doesn't
> > make any sense to me, it is an orte level concept and it is only
> > accessible in the ompi side. I also don't like adding orte_nodeid_t to
> > orte_process_name_t as it really doesn't have anything to do with the a
> > name.. I think it makes more sense to have an orte_proc_t.. Something
> > like the following structure:
> >
> >
> >
> > struct orte_process_name_t {
> > orte_jobid_t jobid; /**< Job number */
> > orte_vpid_t vpid; /**< Process number */
> > /** "nodeid" on which the proc resides */
> > };
> >
> > Struct orte_proc_t {
> > opal_list_item_t super;
> > orte_process_name_t proc_name;
> > orte_nodeid_t nid;
> > };
> >
> > struct ompi_proc_t {
> > orte_proc_t base;
> > . Etc .
> >
> > };
> >
> >
> > I know there is some talk about removing the process names,,, not sure
> > how that fits in here but this is what makes sense to me given the
> > current architecture. Any thoughts here?
> >
> >
> > - Galen
>
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] === CREATE FAILURE ===
This patch will fix the problem -- the OMPI_VAR_SCOPE stuff didn't make it over to the v1.2 branch: Index: config/ompi_config_asm.m4 === --- config/ompi_config_asm.m4 (revision 16740) +++ config/ompi_config_asm.m4 (working copy) @@ -801,7 +801,6 @@ # OS X Leopard ld bus errors if you have "-g" in the link line # with our assembly (!). So remove it from CCASFLAGS if it's # there (and we're on Leopard). -OMPI_VAR_SCOPE_PUSH([ompi_config_asm_flags_new ompi_config_asm_flag]) AC_MSG_CHECKING([if need to remove -g from CCASFLAGS]) case "$host" in *-apple-darwin9.0*) @@ -817,7 +816,7 @@ AC_MSG_RESULT([no]) ;; esac -OMPI_VAR_SCOPE_POP +unset ompi_config_asm_flags_new ompi_config_asm_flag AC_MSG_CHECKING([whether to enable smp locks]) AC_ARG_ENABLE([smp-locks], On Nov 19, 2007, at 9:05 PM, MPI Team wrote: ERROR: Command returned a non-zero exist status ./configure --enable-dist Start time: Mon Nov 19 21:00:11 EST 2007 End time: Mon Nov 19 21:05:13 EST 2007 = == [... previous lines snipped ...] checking size of ssize_t... 4 checking for ptrdiff_t... (cached) yes checking size of ptrdiff_t... 4 checking alignment of char... 1 checking alignment of short... 2 checking alignment of wchar_t... 4 checking alignment of int... 4 checking alignment of long... 4 checking alignment of long long... 4 checking alignment of float... 4 checking alignment of double... 4 checking alignment of long double... 4 checking alignment of void *... 4 checking for C bool type... no checking for inline... __inline__ checking for C/C++ restrict keyword... __restrict checking for weak symbol support... yes *** C++ compiler and preprocessor checking for g++... g++ checking whether we are using the GNU C++ compiler... yes checking whether g++ accepts -g... yes checking dependency style of g++... gcc3 checking how to run the C++ preprocessor... g++ -E checking for the C++ compiler vendor... gnu checking if C++ compiler works... yes configure: WARNING: -g has been added to CXXFLAGS (--enable-debug) checking if g++ supports -Wno-long-double... no configure: WARNING: -Wall -Wundef -Wno-long-long has been added to CXXFLAGS (--enable-picky) checking if g++ supports -finline-functions... yes configure: WARNING: -finline-functions has been added to CXXFLAGS checking if want C++ exception handling... no checking if C and C++ are link compatible... yes checking for C++ template_repository_directory... not used checking for C++ compiler template parameters... none needed checking if g++ supports __builtin_expect... yes checking if g++ supports __builtin_prefetch... yes checking if g++ supports const_cast<> properly... yes checking for C++ optimization flags... -g -Wall -Wundef -Wno-long- long -finline-functions checking for bool... yes checking size of bool... 1 checking alignment of bool... 1 *** Assembler checking dependency style of gcc... gcc3 checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B checking the name lister (/usr/bin/nm -B) interface... BSD nm checking for fgrep... /bin/grep -F ./configure: line 24662: syntax error near unexpected token `ompi_config_asm_flags_new' ./configure: line 24662: ` OMPI_VAR_SCOPE_PUSH(ompi_config_asm_flags_new ompi_config_asm_flag)' = == Your friendly daemon, Cyrador ___ testing mailing list test...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/testing -- Jeff Squyres Cisco Systems
Re: [OMPI devel] ORTE process name,, nodeid..
On 11/19/07 6:20 PM, "Tim Prins" wrote: > On Monday 19 November 2007 09:42:21 am Ralph H Castain wrote: > >> An alternative solution might be to incorporate the modex in the new OMPI >> framework I was about to create anyway. This framework was intended to deal >> with publish/lookup of OMPI data to support a variety of methods. >> Originally, we had intended only to include support there for things >> specifically related to MPI_Publish etc., but there is no reason we >> couldn't generalize it to support the general exchange of process "how to >> connect to me" info and include a modex API in it. I was figuring we would >> need two immediate components in it anyway: an ORTE one for when we have >> full ORTE support in the system, and a CNOS one that would...well, I guess >> just bark and say "you can't do publish/lookup on a Cray". It would be >> simple to add the modex stuff there, and makes some logical sense as well. > I think this approach is fundamentally flawed. Our frameworks are designed to > abstract out something, to allow for multiple implementations. However, doing > this would put two completely different things (the modex and the MPI > pub/sub) together in one framework. While this may be convenient for the > cray, it would be very inconvenient for someone who wanted to do the MPI > pub/sub via a ldap server (as has been discussed). The key here is that MPI > pub/sub is for very small amounts of data, accessed infrequently and in a > non-performance-critical manner, whereas the modex is for rather large > amounts of information (on big jobs) that has to be exchanged efficiently. Actually, several people talked about this before we proposed it and came to a different conclusion. The modex is in essence a "here's how to talk to me" communication, which is the same intent of publish/lookup. I agree that the volume of data involved is different. However, we are -not- proposing to use the same mechanism for the two (modex vs. pub/lookup). The proposal was based on the fact that the publish/lookup and modex effectively use similar mechanisms - i.e., the orte component would use the RML as the underlying communication mechanism. In contrast, the cray component has alternative non-RML based mechanisms for both systems. Things like the LDAP server pose an interesting challenge. In that case, the publish/lookup cannot use the RML as LDAP has no understanding of that comm mode. The modex, however, might - and might not - use that mechanism. Accordingly, the plan was to provide base functions that use RML for any component that can and wants to do so. This is identical to the approach we use throughout the code base. However, we do need the modex in a framework somewhere as we will need to modify it to support tight integration with various environments. I cannot see doing every tight integration with yet another RSL component as the code duplication gets absurd - there isn't enough difference to support it. I also, though, don't want to be forced to use the same modex in every case if the native environment can provide an alternative method - having the modex in the framework solves that problem. So I guess I don't grok the issue here - what is wrong with having a modex API in the pub/sub framework??? Other than causing you some additional merge issues within RSL, I fail to understand why this is a problem. > > Before anyone misunderstands, I am *not* proposing that we add a modex > framework to ompi. Rather, I think this is a case where the RSL could make > things really easy. > > The RSL defines a process attribute system. One of the original ideas (later > retracted, but now that I think about it I may re-add it) was to have some > predefined attribute keys, that the runtime would set so we could look up > information about any process. > > So in the case of the cray, the rsl_init function would query to get all the > info it wants, and then populate the info into its (local) process attribute > data store. > > In other systems each process would set the information in rsl_init and it > would be exchanged in the normal modex method. > > Then, the information would be looked up (locally) using the 'get' function on > both platforms. > > Simple, eh? Maybe - and maybe not. The devil is always in the details. My concerns with the RSL have been documented and wildly misunderstood. I still fail to see the overall advantage as it seems we get different explanations every time we ask. But I'll set that aside here. FWIW: The publish/lookup interface was specifically required to support both local and remote data storage operations, though that doesn't really apply to the modex. > > As an alternative to this, I think we could apply these same ideas into a > specialized ORTE system, but it would not be as clean, and would tie our > system closer to ORTE. I am not going to argue whether this is good or bad, > but I am just mentioning it as a consequence. My concern right now is that doin
