Re: [OMPI devel] Segfault in 1.3 branch
It looks like a new issue to me, Pasha. Possibly a side consequence of the IOF change made by Jeff and I the other day. From what I can see, it looks like you app was a simple "hello" - correct? If you look at the error, the problem occurs when mpirun is trying to route a message. Since the app is clearly running at this time, the problem is probably in the IOF. The error message shows that mpirun is attempting to route a message to a jobid that doesn't exist. We have a test in the RML that forces an "abort" if that occurs. I would guess that there is either a race condition or memory corruption occurring somewhere, but I have no idea where. This may be the "new hole in the dyke" I cautioned about in earlier notes regarding the IOF... :-) Still, given that this hits rarely, it probably is a more acceptable bug to leave in the code than the one we just fixed (duplicated stdin)... Ralph On 7/14/08 1:11 AM, "Pavel Shamis (Pasha)" wrote: > Please see http://www.open-mpi.org/mtt/index.php?do_redir=764 > > The error is not consistent. It takes a lot of iteration to reproduce it. > In my MTT testing I seen it few times. > > Is it know issue ? > > Regards, > Pasha > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] IBCM error
On Jul 14, 2008, at 5:48 PM, Sean Hefty wrote: Is there a service ID range that is guaranteed to be available for user apps? I need to check on this. You may want to look at section A3.2.3 of the spec. If you set the first byte (network order) to 0x00, and the 2nd byte to 0x01, then you hit a 'reserved' range that probably isn't being used currently. If you don't care what the service ID is, you can specify 0, and the kernel will assign one. The assigned value can be retrieved by calling ib_cm_attr_id(). (I'm assuming that you communicate the IDs out of band somehow.) Ok; we'll need to check into this. I don't remember the ordering -- we might actually be communicating the IDs before calling ib_cm_listen() (since we were simply using the PIDs, we could do that). Thanks for the tip! Pasha -- can you look into this? -- Jeff Squyres Cisco Systems
Re: [OMPI devel] IBCM error
>Ah! I did not realize that there were other services on the machine >that were using / reserving IBCM service ID's. Intel MPI hit a similar problem a long, long time ago. >Is there a service ID range that is guaranteed to be available for >user apps? I need to check on this. You may want to look at section A3.2.3 of the spec. If you set the first byte (network order) to 0x00, and the 2nd byte to 0x01, then you hit a 'reserved' range that probably isn't being used currently. If you don't care what the service ID is, you can specify 0, and the kernel will assign one. The assigned value can be retrieved by calling ib_cm_attr_id(). (I'm assuming that you communicate the IDs out of band somehow.) - Sean
Re: [OMPI devel] IBCM error
On Jul 14, 2008, at 5:18 PM, Sean Hefty wrote: Open MPI certainly could be buggy with IBCM, of course -- but it's fishy that the same exact "mpirun ..." command line works one time and fails the next (it's kinda random when the problem occurs). I just want to make sure that service ID collision isn't the issue. (It may be unlikely, but it could happen.) Using the PID is random, and could cause conflicts with other services, depending on the value that's used. I know SDP reserve ranges of service ID values. Ah! I did not realize that there were other services on the machine that were using / reserving IBCM service ID's. Is there a service ID range that is guaranteed to be available for user apps? Is the service ID specified in host or network order? Host order -- just the result of getpid(). Do you know the range of PIDs? I can see if any well known apps might collide. I never looked at the range of PIDs that failed. Pasha / Brad -- could you look into this? It might be that simple... -- Jeff Squyres Cisco Systems
Re: [OMPI devel] IBCM error
>The service ID that it uses is its PID and the mask is always 0. >There will only be one call to ib_cm_listen() per device per MPI >process. > >Open MPI certainly could be buggy with IBCM, of course -- but it's >fishy that the same exact "mpirun ..." command line works one time and >fails the next (it's kinda random when the problem occurs). I just want to make sure that service ID collision isn't the issue. (It may be unlikely, but it could happen.) Using the PID is random, and could cause conflicts with other services, depending on the value that's used. I know SDP reserve ranges of service ID values. Is the service ID specified in host or network order? Do you know the range of PIDs? I can see if any well known apps might collide. - Sean
Re: [OMPI devel] IBCM error
On Jul 14, 2008, at 1:17 PM, Sean Hefty wrote: I talked to Sean Hefty about it, but we never figured out a definitive cause or solution. My best guess is that there is something wonky about multiple processes simultaneously interacting with the IBCM kernel driver from userspace; but I don't know jack about kernel stuff, so that's a total SWAG. The only reason I can think of why ib_cm_listen() fails is if there's a conflict with the service_id and/or service_mask from multiple threads. What does OMPI pass in for these parameters? The service ID that it uses is its PID and the mask is always 0. There will only be one call to ib_cm_listen() per device per MPI process. Open MPI certainly could be buggy with IBCM, of course -- but it's fishy that the same exact "mpirun ..." command line works one time and fails the next (it's kinda random when the problem occurs). -- Jeff Squyres Cisco Systems
Re: [OMPI devel] IBCM error
I've been quietly following this discussion, but now feel a need to jump in here. I really must disagree with the idea of building either IBCM or RDMACM support by default. Neither of these has been proven to reliably work, or to be advantageous. Our own experiences in testing them have been slightly negative at best. When the did work, they were slower, didn't scale well, and unreliable. I'm not trying to rain on anyone's parade. These are worthwhile in the long term. However, they clearly need further work to be "ready for prime time". Accordingly, I would recommend that they -only- be built if specifically requested. Remember, most of our users just build blindly. It makes no sense to have them build support for what can only be classed as an experimental capability at this time. Also, note that the OFED install is less-than-reliable wrt IBCM and RDMACM. We have spent considerable time chasing down installation problems that allowed the system to build, but then caused it to crash-and-burn if we attempted to use it. We have gained experience at knowing when/where to look now, but that doesn't lessen the reputation impact OMPI is getting as a "buggy, cantankerous beast" according to our sys admins. Not a reputation we should be encouraging. Turning this off by default allows those more adventurous souls to explore this capability, while letting our production-oriented customers install and run in peace. Ralph > On Jul 14, 2008, at 9:21 AM, Pavel Shamis (Pasha) wrote: > >>> Should we not even build support for it? >> I think IBCM CPC build should be enabled by default. The IBCM is >> supplied with OFED so it should not be any problem during install. > > Ok. But remember that there are at least some OS's where /dev/ucm* do > *not* get created by default for some unknown reason (even though IBCM > is installed). > >>> PRO: don't even allow the possibility of running with it, because >>> we know that there are issues with the ibcm userspace library >>> (i.e., reduce problem reports from users) >>> >>> PRO: users don't have to have libibcm installed on compute nodes >>> (we've actually gotten some complaints about this) >> We got compliances only for case when ompi was build on platform >> with IBCM and after it was run on platform without IBCM. Also we >> did not have option to disable >> the ibcm during compilation. So actually it was no way to install >> OMPI on compute node. We added the option and the problem was >> resolved. >> In most cases the OFED install is the same on all nodes and it >> should not be any problem to build IBCM support by default. > > > Ok, sounds good. > > -- > Jeff Squyres > Cisco Systems > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel >
Re: [OMPI devel] IBCM error
>I talked to Sean Hefty about it, but we never figured out a definitive >cause or solution. My best guess is that there is something wonky >about multiple processes simultaneously interacting with the IBCM >kernel driver from userspace; but I don't know jack about kernel >stuff, so that's a total SWAG. The only reason I can think of why ib_cm_listen() fails is if there's a conflict with the service_id and/or service_mask from multiple threads. What does OMPI pass in for these parameters? - Sean
Re: [OMPI devel] IBCM error
On Jul 14, 2008, at 9:21 AM, Pavel Shamis (Pasha) wrote: Should we not even build support for it? I think IBCM CPC build should be enabled by default. The IBCM is supplied with OFED so it should not be any problem during install. Ok. But remember that there are at least some OS's where /dev/ucm* do *not* get created by default for some unknown reason (even though IBCM is installed). PRO: don't even allow the possibility of running with it, because we know that there are issues with the ibcm userspace library (i.e., reduce problem reports from users) PRO: users don't have to have libibcm installed on compute nodes (we've actually gotten some complaints about this) We got compliances only for case when ompi was build on platform with IBCM and after it was run on platform without IBCM. Also we did not have option to disable the ibcm during compilation. So actually it was no way to install OMPI on compute node. We added the option and the problem was resolved. In most cases the OFED install is the same on all nodes and it should not be any problem to build IBCM support by default. Ok, sounds good. -- Jeff Squyres Cisco Systems
Re: [OMPI devel] IBCM error
Should we not even build support for it? I think IBCM CPC build should be enabled by default. The IBCM is supplied with OFED so it should not be any problem during install. PRO: don't even allow the possibility of running with it, because we know that there are issues with the ibcm userspace library (i.e., reduce problem reports from users) PRO: users don't have to have libibcm installed on compute nodes (we've actually gotten some complaints about this) We got compliances only for case when ompi was build on platform with IBCM and after it was run on platform without IBCM. Also we did not have option to disable the ibcm during compilation. So actually it was no way to install OMPI on compute node. We added the option and the problem was resolved. In most cases the OFED install is the same on all nodes and it should not be any problem to build IBCM support by default. Pasha
Re: [OMPI devel] SM latency regression
George Bosilca wrote: I'm tracked the SM performance over the last couple of months and I didn't notice any major change on the performance side. I guess there is the architecture factor involved in this. My tests are performed on a PPC (MAC OS X) and on a dual core AMD. What is the architecture where you noticed the performance degradation ? This is on a 2 year old system with 2 chips single AMD cores each chip running both Solaris and Linux under 64 bit addressing. --td Thanks, george. On Jul 11, 2008, at 1:32 PM, Terry Dontje wrote: Has anyone else seen the trunk incur approximately a 10% increase in latency? I think this has happened in the last couple weeks. I have verified that it isn't due to the recheck put into the sm_component_progress. I am about ready to try and track this down but wanted to throw this out there just in case someone is ahead of me. --td ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] IBCM error
On Jul 14, 2008, at 7:55 AM, Pavel Shamis (Pasha) wrote: I can add in head of query function something like : if (!mca_btl_openib_component.cpc_explicitly_defined) return OMPI_ERR_NOT_SUPPORTED; That sounds reasonable until the ibcm userspace library issues can be sorted out. Then perhaps this check can be removed. Should we not even build support for it? PRO: don't even allow the possibility of running with it, because we know that there are issues with the ibcm userspace library (i.e., reduce problem reports from users) PRO: users don't have to have libibcm installed on compute nodes (we've actually gotten some complaints about this) CON: OMPI is not release-synchronized with OFED; OFED could be released with a fixed ibcm userspace library, but it still wouldn't be built by default in OMPI CON: users already have to have librdmacm installed on the compute notes for the RDMA CM (e.g., probably mainly for iWARP support); adding ibcm and rdmacm user libs at the same time might actually be better (rather than rdmacm in v1.3 and ibcm in a future version) Thoughts? -- Jeff Squyres Cisco Systems
Re: [OMPI devel] IBCM error
I can add in head of query function something like : if (!mca_btl_openib_component.cpc_explicitly_defined) return OMPI_ERR_NOT_SUPPORTED; Jeff Squyres wrote: On Jul 14, 2008, at 3:59 AM, Lenny Verkhovsky wrote: Seems to be fixed. Well, it's "fixed" in that Pasha turned off the error message. But the same issue is undoubtedly happening. I was asking for something a little stronger: perhaps we should actually have IBCM not try to be used unless it's specifically asked for. Or maybe it shouldn't even build itself unless specifically asked for (which would obviously take care of the run-time issues as well). The whole point of doing IBCM was to have a simple and fast mechanism for IB wireup. But with these two problems (IBCM not properly installed on all systems, and ib_cm_listen() fails periodically), it more or less makes it unusable. Therefore we shouldn't recommend it to production customers, and per precedent elsewhere in the code base, we should either not build it by default and/or not use it unless specifically asked for.
Re: [OMPI devel] IBCM error
On Jul 14, 2008, at 3:59 AM, Lenny Verkhovsky wrote: Seems to be fixed. Well, it's "fixed" in that Pasha turned off the error message. But the same issue is undoubtedly happening. I was asking for something a little stronger: perhaps we should actually have IBCM not try to be used unless it's specifically asked for. Or maybe it shouldn't even build itself unless specifically asked for (which would obviously take care of the run-time issues as well). The whole point of doing IBCM was to have a simple and fast mechanism for IB wireup. But with these two problems (IBCM not properly installed on all systems, and ib_cm_listen() fails periodically), it more or less makes it unusable. Therefore we shouldn't recommend it to production customers, and per precedent elsewhere in the code base, we should either not build it by default and/or not use it unless specifically asked for. -- Jeff Squyres Cisco Systems
Re: [OMPI devel] IBCM error
Right about when Brad and I discovered that issue, I ran out of time. This made IBCM more-or-less unusable for many installations -- we were kinda hoping for an OpenFabrics fix... On Jul 13, 2008, at 12:43 PM, Pavel Shamis (Pasha) wrote: Fixed in https://svn.open-mpi.org/trac/ompi/changeset/18897 Is it any other know IBCM issue ? Regards, Pasha Jeff Squyres wrote: I think you said opposite things: Lenny's command line did not specifically ask for ibcm, but it was used anyway. Lenny -- did you explicitly request it somewhere else (e.g., env var or MCA param file)? I suspect that you did not; I suspect (without looking at the code again) that ibcm tried to select itself and failed on the ibcm_listen() call, so it fell back to oob. This might have to be another workaround in OMPI, perhaps something like this: if (ibcm_listen() fails) if (ibcm explicitly requested) print_warning() fail to use ibcm Has this been filed as a bug at openfabrics.org? I don't think that I filed it when Brad and I were testing on RoadRunner -- it would probably be good if someone filed it. On Jul 13, 2008, at 8:56 AM, Lenny Verkhovsky wrote: Pasha is right, I didn't disabled it. On 7/13/08, Pavel Shamis (Pasha) wrote: Jeff Squyres wrote: Brad and I did some scale testing of IBCM and saw this error sometimes. It seemed to happen with higher frequency when you increased the number of processes on a single node. I talked to Sean Hefty about it, but we never figured out a definitive cause or solution. My best guess is that there is something wonky about multiple processes simultaneously interacting with the IBCM kernel driver from userspace; but I don't know jack about kernel stuff, so that's a total SWAG. Thanks for reminding me of this issue; I admit that I had forgotten about it. :-( Pasha -- should IBCM not be the default? It is not default. I guess Lenny configured it explicitly, is not it ? Pasha. On Jul 13, 2008, at 7:08 AM, Lenny Verkhovsky wrote: Hi, I am getting this error sometimes. /home/USERS/lenny/OMPI_COMP_PATH/bin/mpirun -np 100 -hostfile / home/USERS/lenny/TESTS/COMPILERS/hostfile /home/USERS/lenny/TESTS/ COMPILERS/hello [witch24][[32428,1],96][../../../../../ompi/mca/btl/openib/connect/ btl_openib_connect_ibcm.c:769:ibcm_component_query] failed to ib_cm_listen 10 times: rc=-1, errno=22 Hello world! I'm 0 of 100 on witch2 Best Regards Lenny. ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres Cisco Systems
Re: [OMPI devel] IBCM error
Seems to be fixed. On 7/14/08, Lenny Verkhovsky wrote: > > ../configure --with-memory-manager=ptmalloc2 --with-openib > > I guess not. I always use same configure line, and only recently I started > to see this error. > > On 7/13/08, Jeff Squyres wrote: >> >> I think you said opposite things: Lenny's command line did not >> specifically ask for ibcm, but it was used anyway. Lenny -- did you >> explicitly request it somewhere else (e.g., env var or MCA param file)? >> >> I suspect that you did not; I suspect (without looking at the code again) >> that ibcm tried to select itself and failed on the ibcm_listen() call, so it >> fell back to oob. This might have to be another workaround in OMPI, perhaps >> something like this: >> >> if (ibcm_listen() fails) >> if (ibcm explicitly requested) >> print_warning() >> fail to use ibcm >> >> Has this been filed as a bug at openfabrics.org? I don't think that I >> filed it when Brad and I were testing on RoadRunner -- it would probably be >> good if someone filed it. >> >> >> >> On Jul 13, 2008, at 8:56 AM, Lenny Verkhovsky wrote: >> >> Pasha is right, I didn't disabled it. >>> >>> On 7/13/08, Pavel Shamis (Pasha) wrote: Jeff >>> Squyres wrote: >>> Brad and I did some scale testing of IBCM and saw this error sometimes. >>> It seemed to happen with higher frequency when you increased the number of >>> processes on a single node. >>> >>> I talked to Sean Hefty about it, but we never figured out a definitive >>> cause or solution. My best guess is that there is something wonky about >>> multiple processes simultaneously interacting with the IBCM kernel driver >>> from userspace; but I don't know jack about kernel stuff, so that's a total >>> SWAG. >>> >>> Thanks for reminding me of this issue; I admit that I had forgotten about >>> it. :-( Pasha -- should IBCM not be the default? >>> It is not default. I guess Lenny configured it explicitly, is not it ? >>> >>> Pasha. >>> >>> >>> >>> >>> >>> On Jul 13, 2008, at 7:08 AM, Lenny Verkhovsky wrote: >>> >>> Hi, >>> >>> I am getting this error sometimes. >>> >>> /home/USERS/lenny/OMPI_COMP_PATH/bin/mpirun -np 100 -hostfile >>> /home/USERS/lenny/TESTS/COMPILERS/hostfile >>> /home/USERS/lenny/TESTS/COMPILERS/hello >>> [witch24][[32428,1],96][../../../../../ompi/mca/btl/openib/connect/btl_openib_connect_ibcm.c:769:ibcm_component_query] >>> failed to ib_cm_listen 10 times: rc=-1, errno=22 >>> Hello world! I'm 0 of 100 on witch2 >>> >>> >>> Best Regards >>> >>> Lenny. >>> >>> >>> ___ >>> devel mailing list >>> de...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>> >>> >>> >>> ___ >>> devel mailing list >>> de...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>> >>> ___ >>> devel mailing list >>> de...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>> >> >> >> -- >> Jeff Squyres >> Cisco Systems >> >> ___ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> > >
[OMPI devel] Segfault in 1.3 branch
Please see http://www.open-mpi.org/mtt/index.php?do_redir=764 The error is not consistent. It takes a lot of iteration to reproduce it. In my MTT testing I seen it few times. Is it know issue ? Regards, Pasha
Re: [OMPI devel] IBCM error
../configure --with-memory-manager=ptmalloc2 --with-openib I guess not. I always use same configure line, and only recently I started to see this error. On 7/13/08, Jeff Squyres wrote: > > I think you said opposite things: Lenny's command line did not specifically > ask for ibcm, but it was used anyway. Lenny -- did you explicitly request > it somewhere else (e.g., env var or MCA param file)? > > I suspect that you did not; I suspect (without looking at the code again) > that ibcm tried to select itself and failed on the ibcm_listen() call, so it > fell back to oob. This might have to be another workaround in OMPI, perhaps > something like this: > > if (ibcm_listen() fails) > if (ibcm explicitly requested) > print_warning() > fail to use ibcm > > Has this been filed as a bug at openfabrics.org? I don't think that I > filed it when Brad and I were testing on RoadRunner -- it would probably be > good if someone filed it. > > > > On Jul 13, 2008, at 8:56 AM, Lenny Verkhovsky wrote: > > Pasha is right, I didn't disabled it. >> >> On 7/13/08, Pavel Shamis (Pasha) wrote: Jeff >> Squyres wrote: >> Brad and I did some scale testing of IBCM and saw this error sometimes. >> It seemed to happen with higher frequency when you increased the number of >> processes on a single node. >> >> I talked to Sean Hefty about it, but we never figured out a definitive >> cause or solution. My best guess is that there is something wonky about >> multiple processes simultaneously interacting with the IBCM kernel driver >> from userspace; but I don't know jack about kernel stuff, so that's a total >> SWAG. >> >> Thanks for reminding me of this issue; I admit that I had forgotten about >> it. :-( Pasha -- should IBCM not be the default? >> It is not default. I guess Lenny configured it explicitly, is not it ? >> >> Pasha. >> >> >> >> >> >> On Jul 13, 2008, at 7:08 AM, Lenny Verkhovsky wrote: >> >> Hi, >> >> I am getting this error sometimes. >> >> /home/USERS/lenny/OMPI_COMP_PATH/bin/mpirun -np 100 -hostfile >> /home/USERS/lenny/TESTS/COMPILERS/hostfile >> /home/USERS/lenny/TESTS/COMPILERS/hello >> [witch24][[32428,1],96][../../../../../ompi/mca/btl/openib/connect/btl_openib_connect_ibcm.c:769:ibcm_component_query] >> failed to ib_cm_listen 10 times: rc=-1, errno=22 >> Hello world! I'm 0 of 100 on witch2 >> >> >> Best Regards >> >> Lenny. >> >> >> ___ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> >> >> >> ___ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> >> ___ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> > > > -- > Jeff Squyres > Cisco Systems > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel >
