Re: [Oorexx-devel] THE Function registration problem
Rick, Reading your previous post I was going to reply that, on Windows, there is a reference count and MSDN says the library won't be unloaded until the count goes to 0. But, it looks like you've already seen that. ;-) -- Mark Miesfeld On Mon, Dec 23, 2013 at 5:41 AM, Rick McGuire wrote: > Ok, it looks like solution #1 will work. Both dlopen() and Windows > LoadLibrary maintain reference counts, so it is possible to lock library in > place if it needs to maintain a local copy. This could be a fairly simple > fix (which can probably wait until Thursday). > > Rick > > > On Mon, Dec 23, 2013 at 8:23 AM, Rick McGuire wrote: > >> Still waiting for Walter to open a bug report on how to recreate the THE >> problem so I can verify, but I have a theory on what's going on here. >> >> I wrote some test cases and added them to the test suite for function EXE >> registrations, and they are definitely working. However, the information >> from Mark Hessling gave me a hint. There was a bug I fixed that was a >> memory leak with these function registrations. The leak would occur if a >> process did local registrations and then did not deregister them before >> terminating. rxapi would then accumulate these dead resistrations every >> time the application was run. I tried a number of different methods of >> forcing these to get cleaned up at process termination, but it turned out >> to be very difficult to do everything needed at process termination time >> because things were going away. >> >> In the end, I fixed the problem by keeping all *Exe registrations in the >> local process rather than using rxapi. Since these were only callable from >> the process registered them, this took rxapi out of the picture entirely. >> The table got nicely cleaned up when the process went away, and was even a >> bit of a performance boost since an IPC was no longer called to invoke >> these functions. It seemed like a nice clean solution to the problem. >> >> Until THE (or more specifically, RexxWrapper) threw a monkey wrench into >> the whole mess. My thinking here was applications would load rexxapi and >> leave it loaded. I'm betting that RexxWrapper is dynamically loading the >> library, calling the function, then unloading the library afterwards >> (again, I need to be able to recreate Walter's problem to verify). The >> unloading process, of course, nukes the local table, so when rexx.dll is >> loaded, rexxapi is reloaded with a fresh table and the functions cannot be >> located. >> >> So, I'm looking for some way to protect these tables within a process >> that will work across platforms, so I'm going to need some help from the >> platform experts. >> >> The first thing that comes to mind is it possible to lock rexxapi in once >> it is loaded? Do library loadings have a usage count that could prevent >> one from getting unloaded? If rexxapi could issue a load against itself >> when first loaded (or maybe only when the local tables are used), that >> could solve the problem. >> >> Another thought is to move the table to another library that rexxapi >> could load dynamically and leave loaded. This could be something quite >> tiny. Or we could add this to the rexx library and dynamically load that >> to obtain the table. One would assume that these APIs would only be used >> if it is intended that Rexx is going to get used. Again, this would only >> work if loads nest, since I assume RexxWrapper will dynamically load >> rexx.dll as well, and we would not want the table to go away if that is >> unloaded. And if the loads, nest, then solution #1 of locking the library >> into the process would be a better solution. >> >> I'm going to be a bit limited in internet access until Thursday evening, >> but I wanted to get this discussion rolling on how to potentially solve >> this problem. >> >> Rick >> > > > > -- > Rapidly troubleshoot problems before they affect your business. Most IT > organizations don't have a clear picture of how application performance > affects their revenue. With AppDynamics, you get 100% visibility into your > Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics > Pro! > http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk > ___ > Oorexx-devel mailing list > Oorexx-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/oorexx-devel > > -- Rapidly troubleshoot problems before they affect your business. Most IT organizations don't have a clear picture of how application performance affects their revenue. With AppDynamics, you get 100% visibility into your Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro! http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk_
Re: [Oorexx-devel] THE Function registration problem
Only now (after entering the ticket) I do see this and understand some (well, hardly anything) of it. But if I can test any modified rexx.exe: I'll be happy to do so Walter Von: Rick McGuire [mailto:object.r...@gmail.com] Gesendet: Montag, 23. Dezember 2013 14:23 An: Open Object Rexx Developer Mailing List Betreff: [Oorexx-devel] THE Function registration problem Still waiting for Walter to open a bug report on how to recreate the THE problem so I can verify, but I have a theory on what's going on here. I wrote some test cases and added them to the test suite for function EXE registrations, and they are definitely working. However, the information from Mark Hessling gave me a hint. There was a bug I fixed that was a memory leak with these function registrations. The leak would occur if a process did local registrations and then did not deregister them before terminating. rxapi would then accumulate these dead resistrations every time the application was run. I tried a number of different methods of forcing these to get cleaned up at process termination, but it turned out to be very difficult to do everything needed at process termination time because things were going away. In the end, I fixed the problem by keeping all *Exe registrations in the local process rather than using rxapi. Since these were only callable from the process registered them, this took rxapi out of the picture entirely. The table got nicely cleaned up when the process went away, and was even a bit of a performance boost since an IPC was no longer called to invoke these functions. It seemed like a nice clean solution to the problem. Until THE (or more specifically, RexxWrapper) threw a monkey wrench into the whole mess. My thinking here was applications would load rexxapi and leave it loaded. I'm betting that RexxWrapper is dynamically loading the library, calling the function, then unloading the library afterwards (again, I need to be able to recreate Walter's problem to verify). The unloading process, of course, nukes the local table, so when rexx.dll is loaded, rexxapi is reloaded with a fresh table and the functions cannot be located. So, I'm looking for some way to protect these tables within a process that will work across platforms, so I'm going to need some help from the platform experts. The first thing that comes to mind is it possible to lock rexxapi in once it is loaded? Do library loadings have a usage count that could prevent one from getting unloaded? If rexxapi could issue a load against itself when first loaded (or maybe only when the local tables are used), that could solve the problem. Another thought is to move the table to another library that rexxapi could load dynamically and leave loaded. This could be something quite tiny. Or we could add this to the rexx library and dynamically load that to obtain the table. One would assume that these APIs would only be used if it is intended that Rexx is going to get used. Again, this would only work if loads nest, since I assume RexxWrapper will dynamically load rexx.dll as well, and we would not want the table to go away if that is unloaded. And if the loads, nest, then solution #1 of locking the library into the process would be a better solution. I'm going to be a bit limited in internet access until Thursday evening, but I wanted to get this discussion rolling on how to potentially solve this problem. Rick -- Rapidly troubleshoot problems before they affect your business. Most IT organizations don't have a clear picture of how application performance affects their revenue. With AppDynamics, you get 100% visibility into your Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro! http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk___ Oorexx-devel mailing list Oorexx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/oorexx-devel
Re: [Oorexx-devel] THE Function registration problem
Ok, it looks like solution #1 will work. Both dlopen() and Windows LoadLibrary maintain reference counts, so it is possible to lock library in place if it needs to maintain a local copy. This could be a fairly simple fix (which can probably wait until Thursday). Rick On Mon, Dec 23, 2013 at 8:23 AM, Rick McGuire wrote: > Still waiting for Walter to open a bug report on how to recreate the THE > problem so I can verify, but I have a theory on what's going on here. > > I wrote some test cases and added them to the test suite for function EXE > registrations, and they are definitely working. However, the information > from Mark Hessling gave me a hint. There was a bug I fixed that was a > memory leak with these function registrations. The leak would occur if a > process did local registrations and then did not deregister them before > terminating. rxapi would then accumulate these dead resistrations every > time the application was run. I tried a number of different methods of > forcing these to get cleaned up at process termination, but it turned out > to be very difficult to do everything needed at process termination time > because things were going away. > > In the end, I fixed the problem by keeping all *Exe registrations in the > local process rather than using rxapi. Since these were only callable from > the process registered them, this took rxapi out of the picture entirely. > The table got nicely cleaned up when the process went away, and was even a > bit of a performance boost since an IPC was no longer called to invoke > these functions. It seemed like a nice clean solution to the problem. > > Until THE (or more specifically, RexxWrapper) threw a monkey wrench into > the whole mess. My thinking here was applications would load rexxapi and > leave it loaded. I'm betting that RexxWrapper is dynamically loading the > library, calling the function, then unloading the library afterwards > (again, I need to be able to recreate Walter's problem to verify). The > unloading process, of course, nukes the local table, so when rexx.dll is > loaded, rexxapi is reloaded with a fresh table and the functions cannot be > located. > > So, I'm looking for some way to protect these tables within a process that > will work across platforms, so I'm going to need some help from the > platform experts. > > The first thing that comes to mind is it possible to lock rexxapi in once > it is loaded? Do library loadings have a usage count that could prevent > one from getting unloaded? If rexxapi could issue a load against itself > when first loaded (or maybe only when the local tables are used), that > could solve the problem. > > Another thought is to move the table to another library that rexxapi could > load dynamically and leave loaded. This could be something quite tiny. Or > we could add this to the rexx library and dynamically load that to obtain > the table. One would assume that these APIs would only be used if it is > intended that Rexx is going to get used. Again, this would only work if > loads nest, since I assume RexxWrapper will dynamically load rexx.dll as > well, and we would not want the table to go away if that is unloaded. And > if the loads, nest, then solution #1 of locking the library into the > process would be a better solution. > > I'm going to be a bit limited in internet access until Thursday evening, > but I wanted to get this discussion rolling on how to potentially solve > this problem. > > Rick > -- Rapidly troubleshoot problems before they affect your business. Most IT organizations don't have a clear picture of how application performance affects their revenue. With AppDynamics, you get 100% visibility into your Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro! http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk___ Oorexx-devel mailing list Oorexx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/oorexx-devel
[Oorexx-devel] THE Function registration problem
Still waiting for Walter to open a bug report on how to recreate the THE problem so I can verify, but I have a theory on what's going on here. I wrote some test cases and added them to the test suite for function EXE registrations, and they are definitely working. However, the information from Mark Hessling gave me a hint. There was a bug I fixed that was a memory leak with these function registrations. The leak would occur if a process did local registrations and then did not deregister them before terminating. rxapi would then accumulate these dead resistrations every time the application was run. I tried a number of different methods of forcing these to get cleaned up at process termination, but it turned out to be very difficult to do everything needed at process termination time because things were going away. In the end, I fixed the problem by keeping all *Exe registrations in the local process rather than using rxapi. Since these were only callable from the process registered them, this took rxapi out of the picture entirely. The table got nicely cleaned up when the process went away, and was even a bit of a performance boost since an IPC was no longer called to invoke these functions. It seemed like a nice clean solution to the problem. Until THE (or more specifically, RexxWrapper) threw a monkey wrench into the whole mess. My thinking here was applications would load rexxapi and leave it loaded. I'm betting that RexxWrapper is dynamically loading the library, calling the function, then unloading the library afterwards (again, I need to be able to recreate Walter's problem to verify). The unloading process, of course, nukes the local table, so when rexx.dll is loaded, rexxapi is reloaded with a fresh table and the functions cannot be located. So, I'm looking for some way to protect these tables within a process that will work across platforms, so I'm going to need some help from the platform experts. The first thing that comes to mind is it possible to lock rexxapi in once it is loaded? Do library loadings have a usage count that could prevent one from getting unloaded? If rexxapi could issue a load against itself when first loaded (or maybe only when the local tables are used), that could solve the problem. Another thought is to move the table to another library that rexxapi could load dynamically and leave loaded. This could be something quite tiny. Or we could add this to the rexx library and dynamically load that to obtain the table. One would assume that these APIs would only be used if it is intended that Rexx is going to get used. Again, this would only work if loads nest, since I assume RexxWrapper will dynamically load rexx.dll as well, and we would not want the table to go away if that is unloaded. And if the loads, nest, then solution #1 of locking the library into the process would be a better solution. I'm going to be a bit limited in internet access until Thursday evening, but I wanted to get this discussion rolling on how to potentially solve this problem. Rick -- Rapidly troubleshoot problems before they affect your business. Most IT organizations don't have a clear picture of how application performance affects their revenue. With AppDynamics, you get 100% visibility into your Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro! http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk___ Oorexx-devel mailing list Oorexx-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/oorexx-devel
