Re: [Firebird-devel] MacOS problem with dlclose
On 1/25/22 11:11, Paul Beach wrote: Should this be maintained in FB 3? No, Min version now for 3.0.8 is 10.15 Ahh - OK, have not known when suggested a fix a few minutes ago. This makes things just simpler. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] MacOS problem with dlclose
On 1/24/22 21:21, Adriano dos Santos Fernandes wrote:
As I talked earlier, FB 3 apparently was not affected.
Now looking for details I see that configure.ac has:
case $host in
*-darwin*)
ac_cv_func_clock_gettime=no
;;
*)
AC_SEARCH_LIBS(clock_gettime, rt)
AC_CHECK_FUNCS(clock_gettime)
;;
esac
Leading to imprecise curTime() (MasterImplementation), not firing timers
in correct moment.
Or not firing at all: "If the processor time used is not available or
its value cannot be represented, the function returns the value
(clock_t) -1." That's enough for plugins to be not unloaded.
Can you check code based on this method? ( I do not provide URL cause
it's russian page)
#if defined(__MACH__) && !defined(CLOCK_REALTIME)
#include
#define CLOCK_REALTIME 0
// clock_gettime is not implemented on older versions of OS X (< 10.12).
// If implemented, CLOCK_REALTIME will have already been defined.
int clock_gettime(int /*clk_id*/, struct timespec* t) {
struct timeval now;
int rv = gettimeofday(, NULL);
if (rv) return rv;
t->tv_sec = now.tv_sec;
t->tv_nsec = now.tv_usec * 1000;
return 0;
}
#endif
This is introduced in this commit:
commit 20542beae18d13c3d2453a9ac2af51b095136c16
Author: paulbeach
Date: Mon Jul 17 14:13:23 2017 +0200
clock_gettime is only supported on 10.12, but lazy binding means
that the code will compile even though version-min is set to 10.7 and
Firebird will then throw a run time error on any versions of OSX post
10.7 and pre 10.12 - disable the function
Should this be maintained in FB 3?
Breaking support of old OS versions in dot release is bad idea on my mind.
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] MacOS problem with dlclose
> On 24/01/2022 06:50, Alex Peshkoff via Firebird-devel wrote: > > On 1/24/22 12:28, Alex Peshkoff via Firebird-devel wrote: > > > >> for FB3 and may be fb4 option 1 is a way to go. > > > > BTW, looks like we have 3.0.9 release soon. > > > > As I talked earlier, FB 3 apparently was not affected. > > Now looking for details I see that configure.ac has: > > case $host in > *-darwin*) > ac_cv_func_clock_gettime=no > ;; > *) > AC_SEARCH_LIBS(clock_gettime, rt) > AC_CHECK_FUNCS(clock_gettime) > ;; > esac > > Leading to imprecise curTime() (MasterImplementation), not firing timers > in correct moment. > > This is introduced in this commit: > > commit 20542beae18d13c3d2453a9ac2af51b095136c16 > Author: paulbeach > Date: Mon Jul 17 14:13:23 2017 +0200 > > clock_gettime is only supported on 10.12, but lazy binding means > that the code will compile even though version-min is set to 10.7 and > Firebird will then throw a run time error on any versions of OSX post > 10.7 and pre 10.12 - disable the function > > > Should this be maintained in FB 3? No, Min version now for 3.0.8 is 10.15 Regards Paul Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] MacOS problem with dlclose
On 24/01/2022 06:50, Alex Peshkoff via Firebird-devel wrote: > On 1/24/22 12:28, Alex Peshkoff via Firebird-devel wrote: > >> for FB3 and may be fb4 option 1 is a way to go. > > BTW, looks like we have 3.0.9 release soon. > As I talked earlier, FB 3 apparently was not affected. Now looking for details I see that configure.ac has: case $host in *-darwin*) ac_cv_func_clock_gettime=no ;; *) AC_SEARCH_LIBS(clock_gettime, rt) AC_CHECK_FUNCS(clock_gettime) ;; esac Leading to imprecise curTime() (MasterImplementation), not firing timers in correct moment. This is introduced in this commit: commit 20542beae18d13c3d2453a9ac2af51b095136c16 Author: paulbeach Date: Mon Jul 17 14:13:23 2017 +0200 clock_gettime is only supported on 10.12, but lazy binding means that the code will compile even though version-min is set to 10.7 and Firebird will then throw a run time error on any versions of OSX post 10.7 and pre 10.12 - disable the function Should this be maintained in FB 3? Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] MacOS problem with dlclose
On 24/01/2022 07:51, Dimitry Sibiryakov wrote: > Adriano dos Santos Fernandes wrote 24.01.2022 11:37: >> If Firebird had an Engine class with state inside that instance, things >> would certainly be easy. >> >> But we rely on many global/static state. >> >> It's not very simple to just reload engine after shutdown when the >> library was not unloaded. > > Shouldn't it to be enough if we don't unload/kill these global states > on shutdown? > Except in trace (maybe), things are not killed. Some state variables are marked as shutdown. But this requires many tests to be reliable. For example, it's possible to relive engine for attachments but not services. It's two different states. Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] MacOS problem with dlclose
Adriano dos Santos Fernandes wrote 24.01.2022 11:37: If Firebird had an Engine class with state inside that instance, things would certainly be easy. But we rely on many global/static state. It's not very simple to just reload engine after shutdown when the library was not unloaded. Shouldn't it to be enough if we don't unload/kill these global states on shutdown? -- WBR, SD. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] MacOS problem with dlclose
On 22/01/2022 12:48, Kovalenko Dmitry wrote: > Hello Adriano, > > If I understood correctly in this topic described same problem: > > https://groups.google.com/g/firebird-support/c/oO9c_ppmfRg/m/-ukO9L37DQAJ > > ? > It appears to be the same error, but AFAIK it currently happens only in MacOS. Or, if someone tries to dlopen/LoadLibrary directly the plugins library... > --- > The problem with correct shutdown of module with shutdown function (or > component with "Dispose" method) is solved through counter of active calls. > Or counter of alive objects in case of OO-API. > > I use this technology in my DLL servers. > If Firebird had an Engine class with state inside that instance, things would certainly be easy. But we rely on many global/static state. It's not very simple to just reload engine after shutdown when the library was not unloaded. Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] MacOS problem with dlclose
On 1/24/22 12:28, Alex Peshkoff via Firebird-devel wrote: for FB3 and may be fb4 option 1 is a way to go. BTW, looks like we have 3.0.9 release soon. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] MacOS problem with dlclose
On 1/21/22 23:19, Dimitry Sibiryakov wrote: Adriano dos Santos Fernandes wrote 21.01.2022 20:08: 1) Disable unload timer in PluginManager in MacOS. Fastest and simplest (or any other way to disable plugins unload). 2) Makes it possible for engine plugin to relive when it was shut down and its entry point is called again. I have tested this solution and it seems to work, but some details must be more tested, like shutdown/relive of trace's ConfigStorage. In any case, with active sandbox code it will never really unload the libraries, just the plugins. 3) Disable things related to sandbox (https://paulbeachsblog.blogspot.com/2021/01/firebird-embedded-in-sandboxed-macos-app.html) Sandbox was not added as a toy just to play with ObjC - it's really needed to users (embedded use in sandbox is nice thing), i.e. disabling it is not a way to go. I think 1 is the way to go. Opinions? Option 2 seems to be right one from my POV. Every plugin should not rely on undocumented unloading moment worse here is that it's documented but broken in some builds... but I agree that we can't rely on unloading currently , so initialization in the entry point and finalization in doCleanup for all plugin-related things and initialization in constructor and finalization in destructor for every library-related things should be the right behavior for every plugin including engine itself. It would be nice if these time frames weren't crossed but it seems that there is nothing we can do for that. Longterm option 2 should be used. But for FB3 and may be fb4 option 1 is a way to go. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] MacOS problem with dlclose
On 21/01/2022 17:57, Dimitry Sibiryakov wrote: > Dimitry Sibiryakov wrote 21.01.2022 21:19: >> initialization in the entry point and finalization in doCleanup for >> all plugin-related things and initialization in constructor and >> finalization in destructor for every library-related things should be >> the right behavior for every plugin including engine itself. > > In addition, not directly related to subj: > > If client library was copied with different name and both libraries > loaded into the same application (nothing unusual for IBExpert for > example), sequence of events in a plugin may be following: > > 1) Global constructors > 2) Entry point from plugin manager 1 > 3) Entry point from plugin manager 2 > 4) doCleanup from plugin manager 2 > 5) Global destructors > 6) doCleanup from plugin manager 1 > > Two issues are here: > > a) If (as per every example and engine code itself) master interface is > cached in a global variable, then starting from 3) the plugin's methods, > called from library 1 will use wrong master interface. > b) After 4) the cached pointer is invalid and if used may lead to crash. > So, this is not supported usage. Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] MacOS problem with dlclose
Hello Adriano, If I understood correctly in this topic described same problem: https://groups.google.com/g/firebird-support/c/oO9c_ppmfRg/m/-ukO9L37DQAJ ? --- The problem with correct shutdown of module with shutdown function (or component with "Dispose" method) is solved through counter of active calls. Or counter of alive objects in case of OO-API. I use this technology in my DLL servers. Regards, Dmitry Kovalenko -Original Message- From: Adriano dos Santos Fernandes [mailto:adrian...@gmail.com] Sent: Friday, January 21, 2022 10:08 PM To: For discussion among Firebird Developers Subject: Re: [Firebird-devel] MacOS problem with dlclose Hi Alex and Paul! I see three way to resolve this: On 10/01/2022 16:19, Adriano dos Santos Fernandes wrote: > Hi! > > As reported in #7041, MacOS build (even in Intel) is returning errors > after some time (1 min) used: > > Authentication error > connection shutdown > > I had debugged the problem, it happens when plugin manager tries to > unload plugins. > > When it calls dlclose no error is returned but library is not really > unloaded. > > Next time it loads the plugin again, it is already in memory and the > memory is not reinitialized nor constructors run again. Only the > plugin entry point is called. > > That makes things unusable. > > I have tracked why it is not unloading. > > There is two reasons for it: > 1) Usage of framework APIs / Objective-C > 2) Usage of __thread variables > > 1) https://developer.apple.com/forums/thread/122591 > 2) > https://github.com/rust-lang/rust/issues/28794#issuecomment-368693049 > > In our code (1) is happening with mac_utils.m and (2) due to > HAVE___THREAD being defined. > > I have tested fixing (2) changing fb_tls.h define: > > -#if defined(HAVE___THREAD) > +#if defined(HAVE___THREAD) && !defined(DARWIN) > > And reverting mac_utils changes it worked. > > There is also suspicious code in > src/yvalve/config/os/darwin/config_root.cpp but it has not needed to > be changed. > > I have not deeply tested why this works in v3. It seems to not try to > unload plugins. > > > Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] MacOS problem with dlclose
Dimitry Sibiryakov wrote 21.01.2022 21:19: initialization in the entry point and finalization in doCleanup for all plugin-related things and initialization in constructor and finalization in destructor for every library-related things should be the right behavior for every plugin including engine itself. In addition, not directly related to subj: If client library was copied with different name and both libraries loaded into the same application (nothing unusual for IBExpert for example), sequence of events in a plugin may be following: 1) Global constructors 2) Entry point from plugin manager 1 3) Entry point from plugin manager 2 4) doCleanup from plugin manager 2 5) Global destructors 6) doCleanup from plugin manager 1 Two issues are here: a) If (as per every example and engine code itself) master interface is cached in a global variable, then starting from 3) the plugin's methods, called from library 1 will use wrong master interface. b) After 4) the cached pointer is invalid and if used may lead to crash. -- WBR, SD. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] MacOS problem with dlclose
Adriano dos Santos Fernandes wrote 21.01.2022 20:08: 1) Disable unload timer in PluginManager in MacOS. 2) Makes it possible for engine plugin to relive when it was shut down and its entry point is called again. I have tested this solution and it seems to work, but some details must be more tested, like shutdown/relive of trace's ConfigStorage. In any case, with active sandbox code it will never really unload the libraries, just the plugins. 3) Disable things related to sandbox (https://paulbeachsblog.blogspot.com/2021/01/firebird-embedded-in-sandboxed-macos-app.html) I think 1 is the way to go. Opinions? Option 2 seems to be right one from my POV. Every plugin should not rely on undocumented unloading moment, so initialization in the entry point and finalization in doCleanup for all plugin-related things and initialization in constructor and finalization in destructor for every library-related things should be the right behavior for every plugin including engine itself. It would be nice if these time frames weren't crossed but it seems that there is nothing we can do for that. -- WBR, SD. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] MacOS problem with dlclose
Hi Alex and Paul! I see three way to resolve this: 1) Disable unload timer in PluginManager in MacOS. 2) Makes it possible for engine plugin to relive when it was shut down and its entry point is called again. I have tested this solution and it seems to work, but some details must be more tested, like shutdown/relive of trace's ConfigStorage. In any case, with active sandbox code it will never really unload the libraries, just the plugins. 3) Disable things related to sandbox (https://paulbeachsblog.blogspot.com/2021/01/firebird-embedded-in-sandboxed-macos-app.html) I think 1 is the way to go. Opinions? Adriano On 10/01/2022 16:19, Adriano dos Santos Fernandes wrote: > Hi! > > As reported in #7041, MacOS build (even in Intel) is returning errors > after some time (1 min) used: > > Authentication error > connection shutdown > > I had debugged the problem, it happens when plugin manager tries to > unload plugins. > > When it calls dlclose no error is returned but library is not really > unloaded. > > Next time it loads the plugin again, it is already in memory and the > memory is not reinitialized nor constructors run again. Only the plugin > entry point is called. > > That makes things unusable. > > I have tracked why it is not unloading. > > There is two reasons for it: > 1) Usage of framework APIs / Objective-C > 2) Usage of __thread variables > > 1) https://developer.apple.com/forums/thread/122591 > 2) https://github.com/rust-lang/rust/issues/28794#issuecomment-368693049 > > In our code (1) is happening with mac_utils.m and (2) due to > HAVE___THREAD being defined. > > I have tested fixing (2) changing fb_tls.h define: > > -#if defined(HAVE___THREAD) > +#if defined(HAVE___THREAD) && !defined(DARWIN) > > And reverting mac_utils changes it worked. > > There is also suspicious code in > src/yvalve/config/os/darwin/config_root.cpp but it has not needed to be > changed. > > I have not deeply tested why this works in v3. It seems to not try to > unload plugins. > > > Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
