Re: atexit(), dlclose() and more atexit()
On 06.07.2020 14:31, Robert Elz wrote: > Date:Sun, 5 Jul 2020 23:11:44 +0200 > From:Kamil Rytarowski > Message-ID: <57c15085-dc7d-6c71-1b26-a402839ba...@netbsd.org> > > | This is extended to the behavior of "at dlclose() or a normal program > | termination". > > Extended by whom or what? What I quoted was from a draft if POSIX issue 8 > which is forthcoming (probably sometime next year). There is no change > like the one you postulate. > > | atexit() is a direct subset of __cxa_atexit() and they are asked to > | share the same internal implementation. > > That's fine. What __cta_atexit() does, and how it does it, is its > own business - but what atexit does is specified, and users are entitled > to reply upon it working the way the standard says it works. > > That some implementations (apparently) don't just means that somewhere, > someone was lazy, and didn't do what the inventors of __cta_atexit() did, > and define a new function to implement new functionality. > > Copying that would be a huge disservice to everyone. > > kre > Can we add atdlclose()? (I don't have strong opinions on the final name.) int atdlclose(void (*function)(void *)); void* as a parameter is important. > Funny that you should mention Windows. To quote MSDN: > https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/atexit?view=vs-2019 > >"The code in the atexit function should not contain any dependency on >any DLL which could have already been unloaded when the atexit function >is called." > > ...and Microsoft specifically introduced a different function for that > purpose. > Windows uses per-DLL atexit stacks and they work similarly to GNU/Linux. They are safe to use in loadable modules, unless they cross call functions from other DLLs that could be gone (this does not include itself, which is safe). signature.asc Description: OpenPGP digital signature
Re: atexit(), dlclose() and more atexit()
On Sun, Jul 05, 2020 at 11:11:44PM +0200, Kamil Rytarowski wrote: > Literal and unextended implementation of the standard happened to be > unpractical. All/most mainstream users (all other BSDs, Win, Mac, GNU, > Solaris, ...) diverged from it within the last 20 years. Funny that you should mention Windows. To quote MSDN: https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/atexit?view=vs-2019 "The code in the atexit function should not contain any dependency on any DLL which could have already been unloaded when the atexit function is called." ...and Microsoft specifically introduced a different function for that purpose. Joerg
Re: atexit(), dlclose() and more atexit()
Date:Sun, 5 Jul 2020 23:11:44 +0200 From:Kamil Rytarowski Message-ID: <57c15085-dc7d-6c71-1b26-a402839ba...@netbsd.org> | This is extended to the behavior of "at dlclose() or a normal program | termination". Extended by whom or what? What I quoted was from a draft if POSIX issue 8 which is forthcoming (probably sometime next year). There is no change like the one you postulate. | atexit() is a direct subset of __cxa_atexit() and they are asked to | share the same internal implementation. That's fine. What __cta_atexit() does, and how it does it, is its own business - but what atexit does is specified, and users are entitled to reply upon it working the way the standard says it works. That some implementations (apparently) don't just means that somewhere, someone was lazy, and didn't do what the inventors of __cta_atexit() did, and define a new function to implement new functionality. Copying that would be a huge disservice to everyone. kre
Re: atexit(), dlclose() and more atexit()
On 05.07.2020 19:42, Robert Elz wrote: > Date:Tue, 30 Jun 2020 13:43:00 +0200 > From:Kamil Rytarowski > Message-ID: > > I had been ignoring this discussion, but on cleaning up some > unread list e-mail, I saw this nonsense, and this is just going too far. > > | This is an extension and extensions are allowed. > > That's absolutely true, but that doesn't relieve the implementation of > the need to follow what the standard does require. > > And in this case that is: > > At normal program termination, all functions registered by the > atexit( ) function shall be called, in the reverse order of their > registration, > This is extended to the behavior of "at dlclose() or a normal program termination". > That is, when the program ends, *every* function registered by atexit() > must be called - there is nothing there which ever suggests "except if it > has already been called". That isn't there, because atexit() functions > are only expected to be called when the process exits (code can explicitly > call such a function, independently, if it wants to of course). > > Not only must the functions be called, the order in which they are to be > called is specified, so if program does atexit(A), then dlopen(L), and in > the init function for L, we get atexit(B), after which (after the dlopen and > the init functions are done) the program does atexit(C), then at > program termination time, the atexit processing must call C, and then B, > and then A; B must not be called (as part of atexit processing) before C. > > I really cannot see how you can possibly mangle the operations and remain > compliant with the standard (nor how any other implementation can). > Literal and unextended implementation of the standard happened to be unpractical. All/most mainstream users (all other BSDs, Win, Mac, GNU, Solaris, ...) diverged from it within the last 20 years. > In another message ka...@netbsd.org said: > | Technically atexit() != __cxa_atexit(), but the "atexit-registered > function" > | mechanism is in place and defined for early DSO unload in C++. > > No-one cares who invented what when, but the very existence of > cta_atexit (which not just technically is != atexit, it simply > isn't atexit) would be because atexit() could not be sanely coerced > to work for the purpose intended. Don't you think that if atexit() > would work, they wouldn't simply have used it, instead of inventing > a new (similarly named) function for the purpose? atexit() is a direct subset of __cxa_atexit() and they are asked to share the same internal implementation. > > Back to the initial message: > | Another option would be to make dlclose() no-op and keep atexit(3) > | operational, but this is certainly not what we want. > > Actually, that one is a possible solution. dlclose() is not required > to do anything at all. While having it never do anything isn't what > we'd want, having it do nothing if there is a pending atexit function > from the dynamic object (or even simply one registered by the dynamic > object - though the problematic case, as I understand it, is when the > function has been removed and so can no longer sensibly be called) is > not a ridiculous suggestion. > > If a dynamic library has registered an atexit function, its obvious > intent is that it will remain loaded until the program exits, and so > in that case making dlclose(), if called, do nothing seems like an > entirely sensible idea. > That would be a progress over our current behavior that crashes always... but it would still be harmful in serious usage. > kre > signature.asc Description: OpenPGP digital signature
Re: atexit(), dlclose() and more atexit()
On Mon, Jul 06, 2020 at 12:42:55AM +0700, Robert Elz wrote: > Actually, that one is a possible solution. dlclose() is not required > to do anything at all. While having it never do anything isn't what > we'd want, having it do nothing if there is a pending atexit function > from the dynamic object (or even simply one registered by the dynamic > object - though the problematic case, as I understand it, is when the > function has been removed and so can no longer sensibly be called) is > not a ridiculous suggestion. There is precedence for dlclose doing nothing - it's a no-op in musl libc, by design. There are obvious downsides (servers with reloadable modules suddenly have memory leaks) but that's arguably not critical. Their justification can be found here: https://wiki.musl-libc.org/functional-differences-from-glibc.html#Unloading-libraries
Re: atexit(), dlclose() and more atexit()
Date:Tue, 30 Jun 2020 13:43:00 +0200 From:Kamil Rytarowski Message-ID: I had been ignoring this discussion, but on cleaning up some unread list e-mail, I saw this nonsense, and this is just going too far. | This is an extension and extensions are allowed. That's absolutely true, but that doesn't relieve the implementation of the need to follow what the standard does require. And in this case that is: At normal program termination, all functions registered by the atexit( ) function shall be called, in the reverse order of their registration, That is, when the program ends, *every* function registered by atexit() must be called - there is nothing there which ever suggests "except if it has already been called". That isn't there, because atexit() functions are only expected to be called when the process exits (code can explicitly call such a function, independently, if it wants to of course). Not only must the functions be called, the order in which they are to be called is specified, so if program does atexit(A), then dlopen(L), and in the init function for L, we get atexit(B), after which (after the dlopen and the init functions are done) the program does atexit(C), then at program termination time, the atexit processing must call C, and then B, and then A; B must not be called (as part of atexit processing) before C. I really cannot see how you can possibly mangle the operations and remain compliant with the standard (nor how any other implementation can). In another message ka...@netbsd.org said: | Technically atexit() != __cxa_atexit(), but the "atexit-registered function" | mechanism is in place and defined for early DSO unload in C++. No-one cares who invented what when, but the very existence of cta_atexit (which not just technically is != atexit, it simply isn't atexit) would be because atexit() could not be sanely coerced to work for the purpose intended. Don't you think that if atexit() would work, they wouldn't simply have used it, instead of inventing a new (similarly named) function for the purpose? Back to the initial message: | Another option would be to make dlclose() no-op and keep atexit(3) | operational, but this is certainly not what we want. Actually, that one is a possible solution. dlclose() is not required to do anything at all. While having it never do anything isn't what we'd want, having it do nothing if there is a pending atexit function from the dynamic object (or even simply one registered by the dynamic object - though the problematic case, as I understand it, is when the function has been removed and so can no longer sensibly be called) is not a ridiculous suggestion. If a dynamic library has registered an atexit function, its obvious intent is that it will remain loaded until the program exits, and so in that case making dlclose(), if called, do nothing seems like an entirely sensible idea. kre
Re: atexit(), dlclose() and more atexit()
On 30.06.2020 15:49, Valery Ushakov wrote: > On Tue, Jun 30, 2020 at 15:09:14 +0200, Kamil Rytarowski wrote: > >> On 30.06.2020 14:24, Valery Ushakov wrote: >>> On Tue, Jun 30, 2020 at 13:43:00 +0200, Kamil Rytarowski wrote: >>> On 30.06.2020 05:16, Jason Thorpe wrote: > >> On Jun 29, 2020, at 5:13 PM, Kamil Rytarowski wrote: >> >>> >>> The atexit() function shall register the function pointed to by func, >>> to be called without arguments at normal program termination. At normal >>> program termination, all functions registered by the atexit() function >>> shall be called, in the reverse order of their registration, except >>> that a function is called after any previously registered functions >>> that had already been called at the time it was registered. Normal >>> termination occurs either by a call to exit() or a return from main(). >>> >>> >>> My reading of the standard here is that atexit() handlers are called at >>> "normal program termination", and that "normal program termination" is >>> explicitly defined as either a call to exit() or returning from main(), >>> and thus any other call to atexit() handlers is expressly forbidden by >>> the standard. >>> >> >> There is no word "only", so it's unspecified. > > Sorry, but that seems like a huge stretch. Everything seems tied to the > process "exit" path in the description of atexit(). Even in the > APPLICATION USAGE section, they have the following informative text: > > > All functions registered by the atexit() function are called at normal > process termination, which occurs by a call to the exit() function or a > return from main() or on the last thread termination, when the behavior > is as if the implementation called exit() with a zero argument at thread > termination time. > > > ...specifically, the "is as if" qualifier. In my reading, if the > enclosing program is not terminating, then atexit() handlers should not > be called. > > dlclose() does not initiate "normal program termination" (it's also > specified in Issue 7, so I double-checked!), nor does it mention anything > about being considered "normal program termination" from the perspective > of the shared object that is being closed. > > Can you point to another place in the standard that uses the "only" type > wording to justify your reading of atexit()? > This is an extension and extensions are allowed. There is also no better alternative as __cxa_atexit() besides of being C++ ABI specific, it is documented as internal only: "No user interface to __cxa_atexit is supported, so the user is not able to register an atexit function with a parameter or a home DSO." https://itanium-cxx-abi.github.io/cxx-abi/abi.html This is only me, but DSO can be treated as a subprogram loaded after dlopen() and terminated upon dlclose(). atexit(3) in this metaphor naturally associates to dlclose(). >>> >>> That's an enticing line of reasoning, and yes one can see how it >>> caused the current atexit abuse (heck, I would have done it myself, >>> people are lazy :), but as all analogies it can only be taken so far. >>> A program termination means the program will be gone very soon, it's >>> memory freed, file descriptors closed, etc. In contrast, the program >>> continues to work after dlclose, so resource leaks are a real concern. >>> So cleanup code that runs at exit time and at the dlclose time have >>> very different operational constraints. atexit-for-dlclose really >>> pushes you further back into MSDOS-like environment where programs are >>> not insulated from each other. >> >> Dynamic loading and unloading code predates MSDOS. It also predates >> shared libraries in UNIX (e.g. Lisp C bindings, predating MSDOS). > > What are you even talking about?! You go out of your way to > misinterpret ~anything said to you and/or to put/steer it into the > context that was obviously not intended. If talking to you requires > math like precision in specifying every tiny detail then expect people > to dissmiss you and your arguments regardless of whatever technical > merits they might have. > > Subprograms are not the invention of MSDOS and predate them. This feature was available in other OSs like BeOS. atexit() as a mechanism can be newer. >> atexit-for-dlclose is already done in C++ behind the scenes for Objects >> and nobody calls it MSDOS-like environment (even if it is, it's not a >> bad design). > > Two can play that game... What does that sentence mean? Which c++ > implementation are you talking about? What specific aspect of that > implementation do you refer to as atexit-for-dlclose? Please provide > specific examples. > > The already linked page from Itanium C++ ABI documents this: 3.3.6.3 Runtime
Re: atexit(), dlclose() and more atexit()
On Tue, Jun 30, 2020 at 15:09:14 +0200, Kamil Rytarowski wrote: > On 30.06.2020 14:24, Valery Ushakov wrote: > > On Tue, Jun 30, 2020 at 13:43:00 +0200, Kamil Rytarowski wrote: > > > >> On 30.06.2020 05:16, Jason Thorpe wrote: > >>> > On Jun 29, 2020, at 5:13 PM, Kamil Rytarowski wrote: > > > > > The atexit() function shall register the function pointed to by func, > > to be called without arguments at normal program termination. At normal > > program termination, all functions registered by the atexit() function > > shall be called, in the reverse order of their registration, except > > that a function is called after any previously registered functions > > that had already been called at the time it was registered. Normal > > termination occurs either by a call to exit() or a return from main(). > > > > > > My reading of the standard here is that atexit() handlers are called at > > "normal program termination", and that "normal program termination" is > > explicitly defined as either a call to exit() or returning from main(), > > and thus any other call to atexit() handlers is expressly forbidden by > > the standard. > > > > There is no word "only", so it's unspecified. > >>> > >>> Sorry, but that seems like a huge stretch. Everything seems tied to the > >>> process "exit" path in the description of atexit(). Even in the > >>> APPLICATION USAGE section, they have the following informative text: > >>> > >>> > >>> All functions registered by the atexit() function are called at normal > >>> process termination, which occurs by a call to the exit() function or a > >>> return from main() or on the last thread termination, when the behavior > >>> is as if the implementation called exit() with a zero argument at thread > >>> termination time. > >>> > >>> > >>> ...specifically, the "is as if" qualifier. In my reading, if the > >>> enclosing program is not terminating, then atexit() handlers should not > >>> be called. > >>> > >>> dlclose() does not initiate "normal program termination" (it's also > >>> specified in Issue 7, so I double-checked!), nor does it mention anything > >>> about being considered "normal program termination" from the perspective > >>> of the shared object that is being closed. > >>> > >>> Can you point to another place in the standard that uses the "only" type > >>> wording to justify your reading of atexit()? > >>> > >> > >> This is an extension and extensions are allowed. > >> > >> There is also no better alternative as __cxa_atexit() besides of being > >> C++ ABI specific, it is documented as internal only: > >> > >> "No user interface to __cxa_atexit is supported, so the user is not able > >> to register an atexit function with a parameter or a home DSO." > >> > >> https://itanium-cxx-abi.github.io/cxx-abi/abi.html > >> > >> This is only me, but DSO can be treated as a subprogram loaded after > >> dlopen() and terminated upon dlclose(). atexit(3) in this metaphor > >> naturally associates to dlclose(). > > > > That's an enticing line of reasoning, and yes one can see how it > > caused the current atexit abuse (heck, I would have done it myself, > > people are lazy :), but as all analogies it can only be taken so far. > > A program termination means the program will be gone very soon, it's > > memory freed, file descriptors closed, etc. In contrast, the program > > continues to work after dlclose, so resource leaks are a real concern. > > So cleanup code that runs at exit time and at the dlclose time have > > very different operational constraints. atexit-for-dlclose really > > pushes you further back into MSDOS-like environment where programs are > > not insulated from each other. > > Dynamic loading and unloading code predates MSDOS. It also predates > shared libraries in UNIX (e.g. Lisp C bindings, predating MSDOS). What are you even talking about?! You go out of your way to misinterpret ~anything said to you and/or to put/steer it into the context that was obviously not intended. If talking to you requires math like precision in specifying every tiny detail then expect people to dissmiss you and your arguments regardless of whatever technical merits they might have. > atexit-for-dlclose is already done in C++ behind the scenes for Objects > and nobody calls it MSDOS-like environment (even if it is, it's not a > bad design). Two can play that game... What does that sentence mean? Which c++ implementation are you talking about? What specific aspect of that implementation do you refer to as atexit-for-dlclose? Please provide specific examples. -uwe
Re: atexit(), dlclose() and more atexit()
On 30.06.2020 14:24, Valery Ushakov wrote: > On Tue, Jun 30, 2020 at 13:43:00 +0200, Kamil Rytarowski wrote: > >> On 30.06.2020 05:16, Jason Thorpe wrote: >>> On Jun 29, 2020, at 5:13 PM, Kamil Rytarowski wrote: > > The atexit() function shall register the function pointed to by func, to > be called without arguments at normal program termination. At normal > program termination, all functions registered by the atexit() function > shall be called, in the reverse order of their registration, except that > a function is called after any previously registered functions that had > already been called at the time it was registered. Normal termination > occurs either by a call to exit() or a return from main(). > > > My reading of the standard here is that atexit() handlers are called at > "normal program termination", and that "normal program termination" is > explicitly defined as either a call to exit() or returning from main(), > and thus any other call to atexit() handlers is expressly forbidden by > the standard. > There is no word "only", so it's unspecified. >>> >>> Sorry, but that seems like a huge stretch. Everything seems tied to the >>> process "exit" path in the description of atexit(). Even in the >>> APPLICATION USAGE section, they have the following informative text: >>> >>> >>> All functions registered by the atexit() function are called at normal >>> process termination, which occurs by a call to the exit() function or a >>> return from main() or on the last thread termination, when the behavior is >>> as if the implementation called exit() with a zero argument at thread >>> termination time. >>> >>> >>> ...specifically, the "is as if" qualifier. In my reading, if the enclosing >>> program is not terminating, then atexit() handlers should not be called. >>> >>> dlclose() does not initiate "normal program termination" (it's also >>> specified in Issue 7, so I double-checked!), nor does it mention anything >>> about being considered "normal program termination" from the perspective of >>> the shared object that is being closed. >>> >>> Can you point to another place in the standard that uses the "only" type >>> wording to justify your reading of atexit()? >>> >> >> This is an extension and extensions are allowed. >> >> There is also no better alternative as __cxa_atexit() besides of being >> C++ ABI specific, it is documented as internal only: >> >> "No user interface to __cxa_atexit is supported, so the user is not able >> to register an atexit function with a parameter or a home DSO." >> >> https://itanium-cxx-abi.github.io/cxx-abi/abi.html >> >> This is only me, but DSO can be treated as a subprogram loaded after >> dlopen() and terminated upon dlclose(). atexit(3) in this metaphor >> naturally associates to dlclose(). > > That's an enticing line of reasoning, and yes one can see how it > caused the current atexit abuse (heck, I would have done it myself, > people are lazy :), but as all analogies it can only be taken so far. > A program termination means the program will be gone very soon, it's > memory freed, file descriptors closed, etc. In contrast, the program > continues to work after dlclose, so resource leaks are a real concern. > So cleanup code that runs at exit time and at the dlclose time have > very different operational constraints. atexit-for-dlclose really > pushes you further back into MSDOS-like environment where programs are > not insulated from each other. > Dynamic loading and unloading code predates MSDOS. It also predates shared libraries in UNIX (e.g. Lisp C bindings, predating MSDOS). atexit-for-dlclose is already done in C++ behind the scenes for Objects and nobody calls it MSDOS-like environment (even if it is, it's not a bad design). I've found out that MacOS and OpenBSD also call atexit callbacks upon dlclose(). Analogous behavior is in Windows for FreeLibrary(). NetBSD is special here with the minimalism in implementation. > -uwe > signature.asc Description: OpenPGP digital signature
Re: atexit(), dlclose() and more atexit()
On Tue, Jun 30, 2020 at 13:43:00 +0200, Kamil Rytarowski wrote: > On 30.06.2020 05:16, Jason Thorpe wrote: > > > >> On Jun 29, 2020, at 5:13 PM, Kamil Rytarowski wrote: > >> > >>> > >>> The atexit() function shall register the function pointed to by func, to > >>> be called without arguments at normal program termination. At normal > >>> program termination, all functions registered by the atexit() function > >>> shall be called, in the reverse order of their registration, except that > >>> a function is called after any previously registered functions that had > >>> already been called at the time it was registered. Normal termination > >>> occurs either by a call to exit() or a return from main(). > >>> > >>> > >>> My reading of the standard here is that atexit() handlers are called at > >>> "normal program termination", and that "normal program termination" is > >>> explicitly defined as either a call to exit() or returning from main(), > >>> and thus any other call to atexit() handlers is expressly forbidden by > >>> the standard. > >>> > >> > >> There is no word "only", so it's unspecified. > > > > Sorry, but that seems like a huge stretch. Everything seems tied to the > > process "exit" path in the description of atexit(). Even in the > > APPLICATION USAGE section, they have the following informative text: > > > > > > All functions registered by the atexit() function are called at normal > > process termination, which occurs by a call to the exit() function or a > > return from main() or on the last thread termination, when the behavior is > > as if the implementation called exit() with a zero argument at thread > > termination time. > > > > > > ...specifically, the "is as if" qualifier. In my reading, if the enclosing > > program is not terminating, then atexit() handlers should not be called. > > > > dlclose() does not initiate "normal program termination" (it's also > > specified in Issue 7, so I double-checked!), nor does it mention anything > > about being considered "normal program termination" from the perspective of > > the shared object that is being closed. > > > > Can you point to another place in the standard that uses the "only" type > > wording to justify your reading of atexit()? > > > > This is an extension and extensions are allowed. > > There is also no better alternative as __cxa_atexit() besides of being > C++ ABI specific, it is documented as internal only: > > "No user interface to __cxa_atexit is supported, so the user is not able > to register an atexit function with a parameter or a home DSO." > > https://itanium-cxx-abi.github.io/cxx-abi/abi.html > > This is only me, but DSO can be treated as a subprogram loaded after > dlopen() and terminated upon dlclose(). atexit(3) in this metaphor > naturally associates to dlclose(). That's an enticing line of reasoning, and yes one can see how it caused the current atexit abuse (heck, I would have done it myself, people are lazy :), but as all analogies it can only be taken so far. A program termination means the program will be gone very soon, it's memory freed, file descriptors closed, etc. In contrast, the program continues to work after dlclose, so resource leaks are a real concern. So cleanup code that runs at exit time and at the dlclose time have very different operational constraints. atexit-for-dlclose really pushes you further back into MSDOS-like environment where programs are not insulated from each other. -uwe
Re: atexit(), dlclose() and more atexit()
On 30.06.2020 05:16, Jason Thorpe wrote: > >> On Jun 29, 2020, at 5:13 PM, Kamil Rytarowski wrote: >> >>> >>> The atexit() function shall register the function pointed to by func, to be >>> called without arguments at normal program termination. At normal program >>> termination, all functions registered by the atexit() function shall be >>> called, in the reverse order of their registration, except that a function >>> is called after any previously registered functions that had already been >>> called at the time it was registered. Normal termination occurs either by a >>> call to exit() or a return from main(). >>> >>> >>> My reading of the standard here is that atexit() handlers are called at >>> "normal program termination", and that "normal program termination" is >>> explicitly defined as either a call to exit() or returning from main(), and >>> thus any other call to atexit() handlers is expressly forbidden by the >>> standard. >>> >> >> There is no word "only", so it's unspecified. > > Sorry, but that seems like a huge stretch. Everything seems tied to the > process "exit" path in the description of atexit(). Even in the APPLICATION > USAGE section, they have the following informative text: > > > All functions registered by the atexit() function are called at normal > process termination, which occurs by a call to the exit() function or a > return from main() or on the last thread termination, when the behavior is as > if the implementation called exit() with a zero argument at thread > termination time. > > > ...specifically, the "is as if" qualifier. In my reading, if the enclosing > program is not terminating, then atexit() handlers should not be called. > > dlclose() does not initiate "normal program termination" (it's also specified > in Issue 7, so I double-checked!), nor does it mention anything about being > considered "normal program termination" from the perspective of the shared > object that is being closed. > > Can you point to another place in the standard that uses the "only" type > wording to justify your reading of atexit()? > This is an extension and extensions are allowed. There is also no better alternative as __cxa_atexit() besides of being C++ ABI specific, it is documented as internal only: "No user interface to __cxa_atexit is supported, so the user is not able to register an atexit function with a parameter or a home DSO." https://itanium-cxx-abi.github.io/cxx-abi/abi.html This is only me, but DSO can be treated as a subprogram loaded after dlopen() and terminated upon dlclose(). atexit(3) in this metaphor naturally associates to dlclose(). Another option would be to make dlclose() no-op and keep atexit(3) operational, but this is certainly not what we want. > -- thorpej > signature.asc Description: OpenPGP digital signature
Re: atexit(), dlclose() and more atexit()
> On Jun 29, 2020, at 5:13 PM, Kamil Rytarowski wrote: > >> >> The atexit() function shall register the function pointed to by func, to be >> called without arguments at normal program termination. At normal program >> termination, all functions registered by the atexit() function shall be >> called, in the reverse order of their registration, except that a function >> is called after any previously registered functions that had already been >> called at the time it was registered. Normal termination occurs either by a >> call to exit() or a return from main(). >> >> >> My reading of the standard here is that atexit() handlers are called at >> "normal program termination", and that "normal program termination" is >> explicitly defined as either a call to exit() or returning from main(), and >> thus any other call to atexit() handlers is expressly forbidden by the >> standard. >> > > There is no word "only", so it's unspecified. Sorry, but that seems like a huge stretch. Everything seems tied to the process "exit" path in the description of atexit(). Even in the APPLICATION USAGE section, they have the following informative text: All functions registered by the atexit() function are called at normal process termination, which occurs by a call to the exit() function or a return from main() or on the last thread termination, when the behavior is as if the implementation called exit() with a zero argument at thread termination time. ...specifically, the "is as if" qualifier. In my reading, if the enclosing program is not terminating, then atexit() handlers should not be called. dlclose() does not initiate "normal program termination" (it's also specified in Issue 7, so I double-checked!), nor does it mention anything about being considered "normal program termination" from the perspective of the shared object that is being closed. Can you point to another place in the standard that uses the "only" type wording to justify your reading of atexit()? -- thorpej
Re: atexit(), dlclose() and more atexit()
On 29.06.2020 21:36, Jason Thorpe wrote: > >> On Jun 29, 2020, at 9:09 AM, Rhialto wrote: >> >> But I wonder if there is any standards text that >> describes whether this particular scenario is supposed to work. > > Quoting from "The Open Group Base Specifications Issue 7, 2018 edition" > > > The atexit() function shall register the function pointed to by func, to be > called without arguments at normal program termination. At normal program > termination, all functions registered by the atexit() function shall be > called, in the reverse order of their registration, except that a function is > called after any previously registered functions that had already been called > at the time it was registered. Normal termination occurs either by a call to > exit() or a return from main(). > > > My reading of the standard here is that atexit() handlers are called at > "normal program termination", and that "normal program termination" is > explicitly defined as either a call to exit() or returning from main(), and > thus any other call to atexit() handlers is expressly forbidden by the > standard. > There is no word "only", so it's unspecified. > -- thorpej > signature.asc Description: OpenPGP digital signature
Re: atexit(), dlclose() and more atexit()
> On Jun 29, 2020, at 9:09 AM, Rhialto wrote: > > But I wonder if there is any standards text that > describes whether this particular scenario is supposed to work. Quoting from "The Open Group Base Specifications Issue 7, 2018 edition" The atexit() function shall register the function pointed to by func, to be called without arguments at normal program termination. At normal program termination, all functions registered by the atexit() function shall be called, in the reverse order of their registration, except that a function is called after any previously registered functions that had already been called at the time it was registered. Normal termination occurs either by a call to exit() or a return from main(). My reading of the standard here is that atexit() handlers are called at "normal program termination", and that "normal program termination" is explicitly defined as either a call to exit() or returning from main(), and thus any other call to atexit() handlers is expressly forbidden by the standard. -- thorpej
Re: atexit(), dlclose() and more atexit() (with reproducer)
I made a small program reproducing the problem, also with a base system
library. I randomly chose /usr/lib/libtermcap.so as default, but I think
everything that turns up by grep _fini /usr/lib/*.so will do.
Actual output:
$ ./dl
main thread 0x706e490a2800: calling exit()
main thread 0x706e490a2800: telling thread to exit
main thread 0x706e490a2800: calling pthread_join()
worker thread 0x706e4909f000: calling dlclose()
(deadlock)
Expected output:
$ ./dl
main thread 0x706d25e35800: calling exit()
main thread 0x706d25e35800: telling thread to exit
main thread 0x706d25e35800: calling pthread_join()
worker thread 0x706d25e32000: calling dlclose()
worker thread 0x706d25e32000: dlclose() returned
worker thread 0x706d25e32000: exiting
main thread 0x706d25e35800: pthread_join() returned
$
dl.c: (compile with "sh dl.c")
--><>---cut here---<><--
# /*
cc dl.c -o dl -pthread -ggdb
exit $?
*/
#include
#include
#include
#include
#include
void *library_handle;
pthread_t worker_thread = 0;
int worker_thread_should_exit = 0;
void cleanup_from_main_thread()
{
printf("main thread %p: telling thread to exit\n", pthread_self());
worker_thread_should_exit = 1;
printf("main thread %p: calling pthread_join()\n", pthread_self());
pthread_join(worker_thread, NULL);
printf("main thread %p: pthread_join() returned\n", pthread_self());
}
void *worker_thread_main(void *arg)
{
/* Do useful work */
while (!worker_thread_should_exit) {
sleep(1);
}
printf("worker thread %p: calling dlclose()\n", pthread_self());
dlclose(library_handle);
printf("worker thread %p: dlclose() returned\n", pthread_self());
printf("worker thread %p: exiting\n", pthread_self());
pthread_exit(NULL);
}
int
main(int argc, char **argv)
{
char *library = "/usr/lib/libtermcap.so";
if (argc > 1) {
library = argv[1];
}
library_handle = dlopen(library, RTLD_LAZY|RTLD_LOCAL);
pthread_create(_thread, NULL, worker_thread_main, NULL);
atexit(cleanup_from_main_thread);
/* Do useful work */
sleep(1);
printf("main thread %p: calling exit()\n", pthread_self());
exit(0);
}
--><>---cut here---<><--
-Olaf.
--
Olaf 'Rhialto' Seibert -- rhialto at falu dot nl
___ Anyone who is capable of getting themselves made President should on
\X/ no account be allowed to do the job. --Douglas Adams, "THGTTG"
signature.asc
Description: PGP signature
Re: atexit(), dlclose() and more atexit()
On Mon 29 Jun 2020 at 09:55:10 +0200, Rhialto wrote: > I've looked at __cxa_finalize a bit better, and it seems that the lock > mutex_lock(&__atexit_mutex); isn't just used to protect running the > atexit handlers, but even to protect looking at the list of handlers: > atexit_handler_stack. > So the fact that libavformat establishes a handler may be a red herring. > I will try to test this later today somehow. So I did a small experiment. Since VICE dlopens several of the ffmpeg's shared libraries, I changed the order that it is closing them. I get now: ffmpegdrv_shutdown: entered; calling ffmpeglib_close(); ffmpeglib_close: free_avcodec() vice_dynlib_close: pthread=0x72abf78b8000 ffmpeglib_close: free_avutil() vice_dynlib_close: pthread=0x72abf78b8000 ffmpeglib_close: free_swscale() vice_dynlib_close: pthread=0x72abf78b8000 ^C^\Quit (core dumped) meaning that it managed to dlclose libavcodec, libavutil, and hangs during libswscale. None of those contain calls to atexit that I could find in the source; only libavformat/avisynth.c refers to atexit. With my revised chain of events, I was expecting that it would deadlock while dlclosing the first shared library, which is now libavcodec. Instead it deadlocks at the third. I'm not sure yet what to make of that. I can only say that in gdb, breaking on __cxa_finalize, the first two libraries don't seem to get there at all and the first library to get there is Thread 9 "" hit Breakpoint 2, __cxa_finalize ( dso=0x7b02ccc7b940 <__dso_handle>) at /usr/src/lib/libc/stdlib/atexit.c:192 (gdb) bt #0 __cxa_finalize (dso=0x7b02ccc7b940 <__dso_handle>) at /usr/src/lib/libc/stdlib/atexit.c:192 #1 0x7b02cca02318 in ?? () from /usr/pkg/lib/ffmpeg4/libswscale.so.5 #2 0x7b02e49fe1a0 in ?? () #3 0x7b02cca6c579 in _fini () from /usr/pkg/lib/ffmpeg4/libswscale.so.5 #4 0x in ?? () I'm not sure why this library has a _fini() function which gets called. This would make the chain of events (3rd version): 1. In the original thread, it dlopen()s a library with a _fini function which calls __cxa_finalize(). [what causes this???] 2. There is no 2. 3. The main thread starts a new thread, and registers an atexit() handler to clean up that thread. 3a. Both threads run for a while doing their main jobs. 4. main thread exit()s. 5. atexit() handler obtains its lock (which protects the handler list), and calls the handler established in 3. 6. said handler tells the new thread to clean up and finish. One of the last things the thread does, is to dlclose() libavformat. 7. __cxa_finalize() gets called on behalf of the library, which tries to obtain the same lock that was already obtained, in a different thread, in step 5. 8. deadlock. In this case, the fault cannot be with libavformat or libswscale, right? -Olaf. -- Olaf 'Rhialto' Seibert -- rhialto at falu dot nl ___ Anyone who is capable of getting themselves made President should on \X/ no account be allowed to do the job. --Douglas Adams, "THGTTG" signature.asc Description: PGP signature
Re: atexit(), dlclose() and more atexit()
On Mon 29 Jun 2020 at 10:39:45 +0200, Martin Husemann wrote: > On Mon, Jun 29, 2020 at 09:55:10AM +0200, Rhialto wrote: > > 6. said handler tells the new thread to clean up and finish. One of the last > >things the thread does, is to dlclose() libavformat. > > How can an atexit() handler (or a destructor) defer work to a thread > (w/o waiting for the thread to complete, but then the thread makes no > sense)? I'm boiling down things to the essence here. The "new thread" isn't just there to do cleanup. It has been running the CPU emulation of VICE for potentially a long time. When the GTK3 GUI (which runs on the main thread) lets the user choose the Quit menu entry, it starts shutting things down. Part of the shutdown happens in the atexit handler. When this CPU emulating thread gets notified that it should finish up, it also handles the dlclose(), which in turn deadlocks. It is basically an unfortunate distribution of cleanup tasks which causes a deadlock. But I wonder if there is any standards text that describes whether this particular scenario is supposed to work. > Martin -Olaf. -- Olaf 'Rhialto' Seibert -- rhialto at falu dot nl ___ Anyone who is capable of getting themselves made President should on \X/ no account be allowed to do the job. --Douglas Adams, "THGTTG" signature.asc Description: PGP signature
Re: atexit(), dlclose() and more atexit()
On Sun 28 Jun 2020 at 23:29:05 +0200, Joerg Sonnenberger wrote: > On Sun, Jun 28, 2020 at 10:56:01PM +0200, Rhialto wrote: > > The funny thing is that libavformat uses an atexit handler due to issues > > with dynamic (un)loading (or so they claim). This is from their file > > ffmpeg-4.2.2/libavformat/avisynth.c: > > > > /* A conflict between C++ global objects, atexit, and dynamic loading > > requires > > * us to register our own atexit handler to prevent double freeing. */ > > It is fundamentally wrong to use a handler in a library that can be > unloaded. Some systems hack around that problem by looping over all > atexit handlers on dlclose, but that's exactly that, a costly hack. The > most common way this triggers is a segfault, actually. > The code should be using either a C dtor with the appropiate attribute > or __cxa_atexit directly, but the former is preferable. I've looked at __cxa_finalize a bit better, and it seems that the lock mutex_lock(&__atexit_mutex); isn't just used to protect running the atexit handlers, but even to protect looking at the list of handlers: atexit_handler_stack. So the fact that libavformat establishes a handler may be a red herring. I will try to test this later today somehow. This would make the chain of events: 1. In the original thread, it dlopen()s libavformat. 2. There is no 2. 3. The main thread starts a new thread, and registers an atexit() handler to clean up that thread. 4. main thread exit()s. 5. atexit() handler obtains its lock (which protects the handler list), and calls the handler established in 3. 6. said handler tells the new thread to clean up and finish. One of the last things the thread does, is to dlclose() libavformat. 7. __cxa_finalize() gets called on behalf of libavformat (as always happens when a library is unloaded), which tries to obtain the same lock that was already obtained, in a different thread, in step 5. 8. deadlock. In this case, the fault cannot be with libavformat, right? -Olaf. -- Olaf 'Rhialto' Seibert -- rhialto at falu dot nl ___ Anyone who is capable of getting themselves made President should on \X/ no account be allowed to do the job. --Douglas Adams, "THGTTG" signature.asc Description: PGP signature
Re: atexit(), dlclose() and more atexit()
On 29.06.2020 00:50, Joerg Sonnenberger wrote: > On Mon, Jun 29, 2020 at 12:34:31AM +0200, Kamil Rytarowski wrote: >> On 28.06.2020 23:57, Joerg Sonnenberger wrote: >>> On Sun, Jun 28, 2020 at 11:48:15PM +0200, Kamil Rytarowski wrote: On 28.06.2020 23:29, Joerg Sonnenberger wrote: > It is fundamentally wrong to use a handler in a library that can be > unloaded. Some systems hack around that problem by looping over all > atexit handlers on dlclose, but that's exactly that, a costly hack. The > most common way this triggers is a segfault, actually. The world disagrees and NetBSD is different for no good reason. >>> >>> You sound like a broken record. Have you *thought* about the reasons at >>> all? Like for example the very definition of atexit: "Run a handler at >>> process exit". The Linux variant does not do that. Heck, they only >>> document it as a side note. >>> >> >> atexit is implemented today as 'The atexit() function registers the >> function pointed to by func to be called without arguments on normal >> termination of the program or when the object defining the function is >> unloaded.' for around 20 years now. >> >> NetBSD is a leftover with a broken implementation. > > Funny, neither POSIX nor ISO C agree with you, but I guess neither is > the relevant standard. But I'll stop here. It isn't productive. > There is no disagreement, but a field not specified. This behavior is suggested in Itanium C++ ABI with so called reasonable treatment of atexit handlers upon dlclose). I've submitted in 2018 a request to Itanium C++ ABI people to clarify the wording, but they redirected me to POSIX. POSIX kind of intends to support no-op dlclose, but perhaps I will need to try to reach them. So this is a dominant convention (Linux, FreeBSD, Solaris) and we will keep observing recurring NetBSD-specific behavior crashes. > Joerg > signature.asc Description: OpenPGP digital signature
Re: atexit(), dlclose() and more atexit()
On Mon, Jun 29, 2020 at 12:34:31AM +0200, Kamil Rytarowski wrote: > On 28.06.2020 23:57, Joerg Sonnenberger wrote: > > On Sun, Jun 28, 2020 at 11:48:15PM +0200, Kamil Rytarowski wrote: > >> On 28.06.2020 23:29, Joerg Sonnenberger wrote: > >>> It is fundamentally wrong to use a handler in a library that can be > >>> unloaded. Some systems hack around that problem by looping over all > >>> atexit handlers on dlclose, but that's exactly that, a costly hack. The > >>> most common way this triggers is a segfault, actually. > >> > >> The world disagrees and NetBSD is different for no good reason. > > > > You sound like a broken record. Have you *thought* about the reasons at > > all? Like for example the very definition of atexit: "Run a handler at > > process exit". The Linux variant does not do that. Heck, they only > > document it as a side note. > > > > atexit is implemented today as 'The atexit() function registers the > function pointed to by func to be called without arguments on normal > termination of the program or when the object defining the function is > unloaded.' for around 20 years now. > > NetBSD is a leftover with a broken implementation. Funny, neither POSIX nor ISO C agree with you, but I guess neither is the relevant standard. But I'll stop here. It isn't productive. Joerg
Re: atexit(), dlclose() and more atexit()
On 28.06.2020 23:57, Joerg Sonnenberger wrote: > On Sun, Jun 28, 2020 at 11:48:15PM +0200, Kamil Rytarowski wrote: >> On 28.06.2020 23:29, Joerg Sonnenberger wrote: >>> It is fundamentally wrong to use a handler in a library that can be >>> unloaded. Some systems hack around that problem by looping over all >>> atexit handlers on dlclose, but that's exactly that, a costly hack. The >>> most common way this triggers is a segfault, actually. >> >> The world disagrees and NetBSD is different for no good reason. > > You sound like a broken record. Have you *thought* about the reasons at > all? Like for example the very definition of atexit: "Run a handler at > process exit". The Linux variant does not do that. Heck, they only > document it as a side note. > atexit is implemented today as 'The atexit() function registers the function pointed to by func to be called without arguments on normal termination of the program or when the object defining the function is unloaded.' for around 20 years now. NetBSD is a leftover with a broken implementation. >> We shall add support for this. > > We shall not. It is a bad design. > The world standardized on integration with dlclose(3). >> On 28.06.2020 23:29, Joerg Sonnenberger wrote: >>> The code should be using either a C dtor with the appropiate attribute >>> or __cxa_atexit directly, but the former is preferable. >> >> This replacement forces to use a hack with a GCC extension (destructor >> attribute) instead of C complaint code. > > So what? It can be spelled out differently and the library here > certainly contains enough dependencies on GCC extensions already. > Destructor is a C++ feature, available in C as a GCC extension. >> __cxa_atexit is a C++ thing so another hack for our atexit(3). > > Where the heck did you arrive at the conclusion that __cxa_atexit is a > C++ thing? It is support infrastructure having dynamic cleanup handlers > on a per DSO base. Just because C++ was the first language to desire > that doesn't mean it is C++ only. In fact, the above extension is using > exactly the same code. > __cxa_atexit is an internal symbol in Itanium C++ ABI, nothing to do with C. > Joerg > signature.asc Description: OpenPGP digital signature
Re: atexit(), dlclose() and more atexit()
On Sun, Jun 28, 2020 at 11:48:15PM +0200, Kamil Rytarowski wrote: > On 28.06.2020 23:29, Joerg Sonnenberger wrote: > > It is fundamentally wrong to use a handler in a library that can be > > unloaded. Some systems hack around that problem by looping over all > > atexit handlers on dlclose, but that's exactly that, a costly hack. The > > most common way this triggers is a segfault, actually. > > The world disagrees and NetBSD is different for no good reason. You sound like a broken record. Have you *thought* about the reasons at all? Like for example the very definition of atexit: "Run a handler at process exit". The Linux variant does not do that. Heck, they only document it as a side note. > We shall add support for this. We shall not. It is a bad design. > On 28.06.2020 23:29, Joerg Sonnenberger wrote: > > The code should be using either a C dtor with the appropiate attribute > > or __cxa_atexit directly, but the former is preferable. > > This replacement forces to use a hack with a GCC extension (destructor > attribute) instead of C complaint code. So what? It can be spelled out differently and the library here certainly contains enough dependencies on GCC extensions already. > __cxa_atexit is a C++ thing so another hack for our atexit(3). Where the heck did you arrive at the conclusion that __cxa_atexit is a C++ thing? It is support infrastructure having dynamic cleanup handlers on a per DSO base. Just because C++ was the first language to desire that doesn't mean it is C++ only. In fact, the above extension is using exactly the same code. Joerg
Re: atexit(), dlclose() and more atexit()
On 28.06.2020 23:29, Joerg Sonnenberger wrote: > It is fundamentally wrong to use a handler in a library that can be > unloaded. Some systems hack around that problem by looping over all > atexit handlers on dlclose, but that's exactly that, a costly hack. The > most common way this triggers is a segfault, actually. The world disagrees and NetBSD is different for no good reason. We shall add support for this. On 28.06.2020 23:29, Joerg Sonnenberger wrote: > The code should be using either a C dtor with the appropiate attribute > or __cxa_atexit directly, but the former is preferable. This replacement forces to use a hack with a GCC extension (destructor attribute) instead of C complaint code. __cxa_atexit is a C++ thing so another hack for our atexit(3). signature.asc Description: OpenPGP digital signature
Re: atexit(), dlclose() and more atexit()
On Sun, Jun 28, 2020 at 10:56:01PM +0200, Rhialto wrote: > On Sun 28 Jun 2020 at 22:39:28 +0200, Joerg Sonnenberger wrote: > > On Sun, Jun 28, 2020 at 10:35:27PM +0200, Rhialto wrote: > > > I have at hand a program (the current svn trunk of VICE, to be exact) > > > which does the following: > > > > > > 1. In the original thread, it dlopen()s libavformat. > > > 2. libavformat establishes an atexit() handler. > > > 3. The main thread starts a new thread, and registers an atexit() > > >handler to clean up that thread. > > > 4. main thread exit()s. > > > 5. atexit() handler obtains its lock, and calls the handler established > > > in 3. > > > 6. said handler tells the new thread to clean up and finish. One of the > > > last > > >things the thread does, is to dlclose() libavformat. > > > 7. libavformat's atexit handling gets called, which tries to obtain the > > >same lock that was already obtained, in a different thread, in step > > >5. > > > 8. deadlock. > > > > > > Who is in the wrong here? > > > > libavformat. Never use atexit() with a handler in a library that can be > > closed. > > The funny thing is that libavformat uses an atexit handler due to issues > with dynamic (un)loading (or so they claim). This is from their file > ffmpeg-4.2.2/libavformat/avisynth.c: > > /* A conflict between C++ global objects, atexit, and dynamic loading requires > * us to register our own atexit handler to prevent double freeing. */ It is fundamentally wrong to use a handler in a library that can be unloaded. Some systems hack around that problem by looping over all atexit handlers on dlclose, but that's exactly that, a costly hack. The most common way this triggers is a segfault, actually. The code should be using either a C dtor with the appropiate attribute or __cxa_atexit directly, but the former is preferable. Joerg
Re: atexit(), dlclose() and more atexit()
On Sun 28 Jun 2020 at 22:39:28 +0200, Joerg Sonnenberger wrote: > On Sun, Jun 28, 2020 at 10:35:27PM +0200, Rhialto wrote: > > I have at hand a program (the current svn trunk of VICE, to be exact) > > which does the following: > > > > 1. In the original thread, it dlopen()s libavformat. > > 2. libavformat establishes an atexit() handler. > > 3. The main thread starts a new thread, and registers an atexit() > >handler to clean up that thread. > > 4. main thread exit()s. > > 5. atexit() handler obtains its lock, and calls the handler established in > > 3. > > 6. said handler tells the new thread to clean up and finish. One of the last > >things the thread does, is to dlclose() libavformat. > > 7. libavformat's atexit handling gets called, which tries to obtain the > >same lock that was already obtained, in a different thread, in step > >5. > > 8. deadlock. > > > > Who is in the wrong here? > > libavformat. Never use atexit() with a handler in a library that can be > closed. The funny thing is that libavformat uses an atexit handler due to issues with dynamic (un)loading (or so they claim). This is from their file ffmpeg-4.2.2/libavformat/avisynth.c: /* A conflict between C++ global objects, atexit, and dynamic loading requires * us to register our own atexit handler to prevent double freeing. */ > Joerg -Olaf. -- Olaf 'Rhialto' Seibert -- rhialto at falu dot nl ___ Anyone who is capable of getting themselves made President should on \X/ no account be allowed to do the job. --Douglas Adams, "THGTTG" signature.asc Description: PGP signature
Re: atexit(), dlclose() and more atexit()
On Sun, Jun 28, 2020 at 10:35:27PM +0200, Rhialto wrote: > I have at hand a program (the current svn trunk of VICE, to be exact) > which does the following: > > 1. In the original thread, it dlopen()s libavformat. > 2. libavformat establishes an atexit() handler. > 3. The main thread starts a new thread, and registers an atexit() >handler to clean up that thread. > 4. main thread exit()s. > 5. atexit() handler obtains its lock, and calls the handler established in 3. > 6. said handler tells the new thread to clean up and finish. One of the last >things the thread does, is to dlclose() libavformat. > 7. libavformat's atexit handling gets called, which tries to obtain the >same lock that was already obtained, in a different thread, in step >5. > 8. deadlock. > > Who is in the wrong here? libavformat. Never use atexit() with a handler in a library that can be closed. Joerg
