with curiosity i looked at this issue. here's my 2 cents
I think issue is because of BTL components is opened&closed
twice(ompi_init, yoda) which leading to incorrect usage of var
groups.
The following sequence of events creating invalid memory
1) all openib component parameters registered in ompi_mpi_init
main > start_pes> shmem_init -> oshmem_shmem_init -> ompi_mpi_init ->
mca_base_framework_open -> mca_pml_base_open .....
mca_bml_base_open...
-> btl_openib_component_register()
* for all string variables it allocated a memory block
(var->mbv_storage
= PTR)
At this time a new var group id:114 (of parent group id: 112) is
created
for all openib component variables.
2) This var group is de-registered in ompi_mpi_init. It marks all
variables as invalid. but, the group&vars is still exist
main > start_pes> shmem_init -> oshmem_shmem_init ->
mca_pml_base_select
-> mca_base_components_close -> ... -> mca_bml_base_close ->
mca_base_framework_close -> mca_base_var_group_deregister(groupid:
114) *
all string variables memory is deallocated ( set var->mbv_storage =
NULL;)
3) because of step 2). btl_openib.so shared lib dlclosed
4) Now we are reopening openib in yoda and registering the openib
variables again.
main > start_pes> shmem_init > oshmem_shmem_init -> _shmem_init ->
mca_base_framework_open -> mca_spml_base_open>
mca_spml_yoda_component_open-> ..... mca_bml_base_open... ->
btl_openib_component_register -> register_variables()
* In register_variables(), var_find() finds this variable( from the
same
old group: 114) and reset the variables.
* For string variables, it allocated the buffers again (
(var->mbv_storage = PTR)
* note that group:114 is not belongs to yoda component.
5) In yoda component close, it never finds above group(114) because
this
is not belongs to this component. So, do not call
mca_base_var_group_deregister() again on the var group. string var
memory
is not deallocated.
main > start_pes> shmem_init > oshmem_shmem_init -> _shmem_init ->
mca_spml_base_select ->..> mca_spml_yoda_component_close ->
mca_bml_base_close -> mca_base_var_group_find().
6) because of step 5), the btl_openib.so is dlclosed(). This step
invalidates, all openib string vars memory ( var->mbv_storage = PTR)
allocated in step 4)
7) in ompi_mpi_finalize(), it will loop through all vars and
finalizes
and deallocate the string var memory (var->mbv_storage = PTR)
ompi_mpi_finalize >...> mca_base_var_finalize * var->mbv_storage =
PTR
is
invalid at this stage and causing the SEGFAULT.
This also explains why Dinar's patch, kostul_fix.patch
(http://bgate.mellanox.com/redmine/attachments/1643/kostul_fix.patch),
resolves the issue. His patch prevents you from finding the invalid
already opened params.
So, I see in a lot of these registration functions the signature has
an
entry for the project name, but now, NULL, is always passed. I see a
note
by Nathan in
../opal/mca/base/mca_base_var.c +1311
{
/* XXX -- component_update -- We will stash the project name in the
component */
return mca_base_var_register (NULL, component->mca_type_name,
Seems knowing the project name, oshmem, would allow us to distinguish
between the different BMLs.
Nathan, please advise.
Josh
-----Original Message-----
From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Nathan
Hjelm
Sent: Monday, December 16, 2013 12:44 PM
To: Open MPI Developers
Subject: Re: [OMPI devel] bug in mca framework?
On Mon, Dec 16, 2013 at 05:21:05PM +0000, Joshua Ladd wrote:
After speaking with Igor Ivanov about this this morning, he
summarized
his findings as follows:
1. Valgrind comes up clean.
Thats good to hear but unfortunate since this seems really like a
stomping-on-memory problem.
2. The issue is not reproduced with a static build.
This is a red-herring. The variable itself contains garbage. The
mbv_storage pointer looked like it was on the stack, the name was not
valid, etc. Not sure how we got an mca_base_var_t into that state
since
the only time we touch anything in them is in mca_base_var_finalize.
That
functions cleans up all of the state to two calls to it should be
harmless.
3. A bisection study reveals that problems first appear after
commit:
https://svn.open-mpi.org/trac/ompi/changeset/28800/trunk/opal/mca/bas
e
/mca_base_var.c
Possibly also a coincidence. That commit only 1) moves the group
stuff
into its own file, and 2) adds the mca_base_pvar interface. Its
possible
I messed something up in the rest of the code but unlikely. I will
take
another look though.
-Nathan
Josh
-----Original Message-----
From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Jeff
Squyres (jsquyres)
Sent: Monday, December 16, 2013 12:15 PM
To: Open MPI Developers
Subject: Re: [OMPI devel] bug in mca framework?
It might be worthwhile to run this through valgrind and see if
something is being freed incorrectly...?
On Dec 16, 2013, at 12:11 PM, Nathan Hjelm <hje...@lanl.gov> wrote:
I took a look at the stacktraces last week and could not identify
where the bug is. I will dig deeper this week and see if I can come
up with the correct fix.
-Nathan
On Mon, Dec 09, 2013 at 03:17:36PM +0200, Mike Dubman wrote:
Nathan,
Could you please comment on the Igor`s observations?
Thanks
On Wed, Dec 4, 2013 at 4:44 PM, Igor Ivanov
<igor.iva...@itseez.com>
wrote:
On 04.12.2013 17:56, Jeff Squyres (jsquyres) wrote:
On Dec 4, 2013, at 2:52 AM, Igor Ivanov
<igor.iva...@itseez.com>
wrote:
It is the first mca variable with type as string from
btl/openib as
'device_param_files'. Actually you can disable it and
get
failure on
the second.
Description of case we see:
1. openib mca variables are registered during startup as
stage at
select component phase;
2. but a winner is cm component and openib mca variables
are
deregistered as part of mca group;
3. mca variables are not removed from global mca array
but
they
marked as invalid and memory for string is freed;
4. shmem needs openib for yoda and does bml
initialization;
5. openib mca variables are registered againusing light
mode
as
searching itself in global array and refreshing their
fields again;
Can you explain what you mean by step 5? I.e., what does
"using light
mode" mean? Is the openib component register function
invoked
again?
It is correct, it is called twice. "light mode" means that
mca_base_var_register() does not allocate mca variable
object
again, it
seeks this variable in global array and finding it updates
fields in
mca_base_var_t structure (at least mbv_storage).
6. for unknown reason bml finalization does not clean
these
vars as
it is done in step 2;
7. mca_btl_openib.so is unloaded;
8. opal_finalize() destroys mca variables form global
array,
observes openib`s variable, try destroy using non
accessed
address;
So a code that is under discussion fixes step 6.
Nathan: it sounds like an MCA var (and entire group) is
registered,
unregistered, and then registered again. Does the MCA var
system get
confused here when it tries to unregister the group a 2nd
time?
Probably issue relates incorrect recognition if variable
valid/invalid
during second call of mca_base_var_deregister().
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
