Re: Haskell DLL (using FFI) causes memory leak.

[Ольга Филиппская]

пн, 27 авг. 2018 г. в 19:46, Phyx :

> Hi,
>
> You're likely just hitting a memory leak, Haskell DLLs and the RTS aren't
> designed to be unload-able, which is why you can't call hs_init again after
> you stop.
>
> The leak happens only when you do a foreign export because foreign exports
> create C wrappers to initializers for each function you export.
>
> https://github.com/ghc/ghc/blob/21f0f56164f50844c2150c62f950983b2376f8b6/compiler/deSugar/DsForeign.hs#L668
>
>
These are marked as static initializers, and as such the CRT will
> initialize them on DLL_PROCESS_ATTACH time. At DLL_PROCESS_DETACH
> time the destructors would be run, but we don't have destructors for these
> initializers, so whatever they did will always persist.
>

Hello!
Thank you very much for such a detailed explanation! Now I gain an insight
into what's happening.



>
> Your use case is fairly uncommon, but file a bug report against the leaks
> anyway. Why do you need to keep loading and unloading the dll?
>

Our goal was to understand whether we're cooking it in a wrong way or
that's the implementation details of the Haskell RTS.
And your answer just suits our goal perfectly. Anyway I will submit a bug
report as you suggested.

Kind regards,
-- 
*Olga Philippskaya.*
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Haskell DLL (using FFI) causes memory leak.

[Ben Gamari]

Phyx  writes:

> Oh and also, neither of those tickets you linked to should prevent you from
> using a newer GHC to make shared libraries.
>
> the Rts.h headers are nothing but convenience functions and you can just
> create the prototypes you need in your own headers.
>
That being said, I think it would be much better if GHC's headers would
accommodate C++ compilers, as Herbert suggested in #14784. I suspect we
aren't using much C99 in the headers anyways.

Perhaps we should just start by trying a conservative C++11 guard and
tighten it later if we find earlier language versions would work.

Cheers,

- Ben



signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Haskell DLL (using FFI) causes memory leak.

[Phyx]

Oh and also, neither of those tickets you linked to should prevent you from
using a newer GHC to make shared libraries.

the Rts.h headers are nothing but convenience functions and you can just
create the prototypes you need in your own headers.

On Mon, Aug 27, 2018 at 5:46 PM Phyx  wrote:

> Hi,
>
> You're likely just hitting a memory leak, Haskell DLLs and the RTS aren't
> designed to be unload-able, which is why you can't call hs_init again after
> you stop.
>
> The leak happens only when you do a foreign export because foreign exports
> create C wrappers to initializers for each function you export.
>
> https://github.com/ghc/ghc/blob/21f0f56164f50844c2150c62f950983b2376f8b6/compiler/deSugar/DsForeign.hs#L668
>
> These are marked as static initializers, and as such the CRT will
> initialize them on DLL_PROCESS_ATTACH time. At DLL_PROCESS_DETACH
> time the destructors would be run, but we don't have destructors for these
> initializers, so whatever they did will always persist.
>
> Your use case is fairly uncommon, but file a bug report against the leaks
> anyway. Why do you need to keep loading and unloading the dll?
>
> Note that FreeLibrary like dlclose on unix does not guarantee that the
> library is freed, it just decrements the reference count. and when it
> reaches 0 it will *eventually* unload it. Notice that if you use
> UnmapViewOfFile instead of FreeLibrary the leak doesn't happen as that
> *actually* immediately unmaps the image.  But this will get you in all
> sorts of trouble later on when terminating (you'll likely segfault at that
> point).
>
> Things will get worse when you actually do call hs_init, because hs_init
> will call initLinker which will load libraries of it's own. Unloading the
> top level
> does not decrease the reference counts of the dependencies, and we do no
> extra work to ensure this.
>
> If you really need to load/reload libraries, then I fear your only choice
> is some kind of process isolation and communicating back using share
> memory/pipes
> or some sort of IPC.
>
> Regards,
> Tamar
>
> On Mon, Aug 27, 2018 at 4:33 PM Ольга Филиппская 
> wrote:
>
>> It's true that one must initialize the runtime when calling Haskell from
>> C/C++. We do this in the real project and just a pair of calls
>> hs_init/hs_exit causes memory to be leaked even faster. But I tried to
>> construct the minimal example to describe the bug, so I reduced all the
>> extra-code.
>>
>> пн, 27 авг. 2018 г. в 18:00, Vanessa McHale :
>>
>>> I don't know about C++, but I do know that when calling Haskell from C
>>> code one must initialize the runtime and then exit it once finished. I also
>>> know that one must only initialize the runtime once over the course of the
>>> program's run. As far as I can tell, this does not occur in your example.
>>>
>>> On 08/27/2018 09:44 AM, Ольга Филиппская wrote:
>>>
>>> Hello.
>>>
>>> *Summary: *memory is consumed without releasing when a Haskell DLL
>>> (that uses FFI) is loaded and unloaded in an endless loop.
>>>
>>> *Details*: I'm working on a C++ project relying on a DLL that uses
>>> Haskell code. The DLL was built with GHC 8.0.1 x64, OS is Windows 7. (Newer
>>> versions of GHC - 8.2.1 and later - do not allow you to build such DLLs due
>>> to some bugs: ticket #1
>>> , ticker #2
>>> ). The C++ project
>>> was built with MSVC compiler 2015.
>>>
>>> We noticed that memory is not released when the library is unloaded.
>>> I've constructed a baseline example to reproduce the bug. It has three
>>> files: the first one is *HaskellSources.hs*(see attachments). It
>>> contains one foreign export function foo. This foreign export function is
>>> necessary to reproduce the bug. The second file is *CWrapper.cpp*. I
>>> intentionally didn't include any Haskell function export because they make
>>> no difference for this example. The last file is the code of the main
>>> program (see *main.cpp*) that loads the library and unloads it at once
>>> in an endless loop.
>>>
>>> There are two cases: whether HaskellExports.o is included into the
>>> library or not.
>>> 1. HaskellExports.o is not included into the library (see attached build
>>> script* build_no_hs.sh*). Everything works just fine (i.e. amount of
>>> memory consumed by the app is the same before DLL loading and right after
>>> unloading.)
>>> 2. HaskellExports.o is included into the library (see *build_w_hs.sh*).
>>> Memory is not released after unloading the DLL, after repeated load/unload
>>> cycles memory consumption keeps growing until finally the application
>>> crashes.
>>>
>>> Is this a known problem? Is it tracked in your bugtracker? (Or maybe it
>>> is not a problem at all and I'm doing it wrong).
>>>
>>> Thank you for your time.
>>>
>>> --
>>> * Olga Philippskaya .*
>>>
>>>
>>> ___
>>> ghc-devs mailing 
>>> 

Re: Haskell DLL (using FFI) causes memory leak.

[Phyx]

Hi,

You're likely just hitting a memory leak, Haskell DLLs and the RTS aren't
designed to be unload-able, which is why you can't call hs_init again after
you stop.

The leak happens only when you do a foreign export because foreign exports
create C wrappers to initializers for each function you export.
https://github.com/ghc/ghc/blob/21f0f56164f50844c2150c62f950983b2376f8b6/compiler/deSugar/DsForeign.hs#L668

These are marked as static initializers, and as such the CRT will
initialize them on DLL_PROCESS_ATTACH time. At DLL_PROCESS_DETACH
time the destructors would be run, but we don't have destructors for these
initializers, so whatever they did will always persist.

Your use case is fairly uncommon, but file a bug report against the leaks
anyway. Why do you need to keep loading and unloading the dll?

Note that FreeLibrary like dlclose on unix does not guarantee that the
library is freed, it just decrements the reference count. and when it
reaches 0 it will *eventually* unload it. Notice that if you use
UnmapViewOfFile instead of FreeLibrary the leak doesn't happen as that
*actually* immediately unmaps the image.  But this will get you in all
sorts of trouble later on when terminating (you'll likely segfault at that
point).

Things will get worse when you actually do call hs_init, because hs_init
will call initLinker which will load libraries of it's own. Unloading the
top level
does not decrease the reference counts of the dependencies, and we do no
extra work to ensure this.

If you really need to load/reload libraries, then I fear your only choice
is some kind of process isolation and communicating back using share
memory/pipes
or some sort of IPC.

Regards,
Tamar

On Mon, Aug 27, 2018 at 4:33 PM Ольга Филиппская 
wrote:

> It's true that one must initialize the runtime when calling Haskell from
> C/C++. We do this in the real project and just a pair of calls
> hs_init/hs_exit causes memory to be leaked even faster. But I tried to
> construct the minimal example to describe the bug, so I reduced all the
> extra-code.
>
> пн, 27 авг. 2018 г. в 18:00, Vanessa McHale :
>
>> I don't know about C++, but I do know that when calling Haskell from C
>> code one must initialize the runtime and then exit it once finished. I also
>> know that one must only initialize the runtime once over the course of the
>> program's run. As far as I can tell, this does not occur in your example.
>>
>> On 08/27/2018 09:44 AM, Ольга Филиппская wrote:
>>
>> Hello.
>>
>> *Summary: *memory is consumed without releasing when a Haskell DLL (that
>> uses FFI) is loaded and unloaded in an endless loop.
>>
>> *Details*: I'm working on a C++ project relying on a DLL that uses
>> Haskell code. The DLL was built with GHC 8.0.1 x64, OS is Windows 7. (Newer
>> versions of GHC - 8.2.1 and later - do not allow you to build such DLLs due
>> to some bugs: ticket #1
>> , ticker #2
>> ). The C++ project
>> was built with MSVC compiler 2015.
>>
>> We noticed that memory is not released when the library is unloaded. I've
>> constructed a baseline example to reproduce the bug. It has three files:
>> the first one is *HaskellSources.hs*(see attachments). It contains one
>> foreign export function foo. This foreign export function is necessary to
>> reproduce the bug. The second file is *CWrapper.cpp*. I intentionally
>> didn't include any Haskell function export because they make no difference
>> for this example. The last file is the code of the main program (see
>> *main.cpp*) that loads the library and unloads it at once in an endless
>> loop.
>>
>> There are two cases: whether HaskellExports.o is included into the
>> library or not.
>> 1. HaskellExports.o is not included into the library (see attached build
>> script* build_no_hs.sh*). Everything works just fine (i.e. amount of
>> memory consumed by the app is the same before DLL loading and right after
>> unloading.)
>> 2. HaskellExports.o is included into the library (see *build_w_hs.sh*).
>> Memory is not released after unloading the DLL, after repeated load/unload
>> cycles memory consumption keeps growing until finally the application
>> crashes.
>>
>> Is this a known problem? Is it tracked in your bugtracker? (Or maybe it
>> is not a problem at all and I'm doing it wrong).
>>
>> Thank you for your time.
>>
>> --
>> * Olga Philippskaya .*
>>
>>
>> ___
>> ghc-devs mailing 
>> listghc-devs@haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
>
>
> --
> *Филиппская Ольга.*
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>

Re: Haskell DLL (using FFI) causes memory leak.

[Ольга Филиппская]

It's true that one must initialize the runtime when calling Haskell from
C/C++. We do this in the real project and just a pair of calls
hs_init/hs_exit causes memory to be leaked even faster. But I tried to
construct the minimal example to describe the bug, so I reduced all the
extra-code.

пн, 27 авг. 2018 г. в 18:00, Vanessa McHale :

> I don't know about C++, but I do know that when calling Haskell from C
> code one must initialize the runtime and then exit it once finished. I also
> know that one must only initialize the runtime once over the course of the
> program's run. As far as I can tell, this does not occur in your example.
>
> On 08/27/2018 09:44 AM, Ольга Филиппская wrote:
>
> Hello.
>
> *Summary: *memory is consumed without releasing when a Haskell DLL (that
> uses FFI) is loaded and unloaded in an endless loop.
>
> *Details*: I'm working on a C++ project relying on a DLL that uses
> Haskell code. The DLL was built with GHC 8.0.1 x64, OS is Windows 7. (Newer
> versions of GHC - 8.2.1 and later - do not allow you to build such DLLs due
> to some bugs: ticket #1
> , ticker #2
> ). The C++ project was
> built with MSVC compiler 2015.
>
> We noticed that memory is not released when the library is unloaded. I've
> constructed a baseline example to reproduce the bug. It has three files:
> the first one is *HaskellSources.hs*(see attachments). It contains one
> foreign export function foo. This foreign export function is necessary to
> reproduce the bug. The second file is *CWrapper.cpp*. I intentionally
> didn't include any Haskell function export because they make no difference
> for this example. The last file is the code of the main program (see
> *main.cpp*) that loads the library and unloads it at once in an endless
> loop.
>
> There are two cases: whether HaskellExports.o is included into the library
> or not.
> 1. HaskellExports.o is not included into the library (see attached build
> script* build_no_hs.sh*). Everything works just fine (i.e. amount of
> memory consumed by the app is the same before DLL loading and right after
> unloading.)
> 2. HaskellExports.o is included into the library (see *build_w_hs.sh*).
> Memory is not released after unloading the DLL, after repeated load/unload
> cycles memory consumption keeps growing until finally the application
> crashes.
>
> Is this a known problem? Is it tracked in your bugtracker? (Or maybe it is
> not a problem at all and I'm doing it wrong).
>
> Thank you for your time.
>
> --
> * Olga Philippskaya .*
>
>
> ___
> ghc-devs mailing 
> listghc-devs@haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>


-- 
*Филиппская Ольга.*
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Haskell DLL (using FFI) causes memory leak.

[Vanessa McHale]

I don't know about C++, but I do know that when calling Haskell from C
code one must initialize the runtime and then exit it once finished. I
also know that one must only initialize the runtime once over the course
of the program's run. As far as I can tell, this does not occur in your
example.


On 08/27/2018 09:44 AM, Ольга Филиппская wrote:
> Hello.
>
> /Summary: /memory is consumed without releasing when a Haskell DLL
> (that uses FFI) is loaded and unloaded in an endless loop.
>
> /Details/: I'm working on a C++ project relying on a DLL that uses
> Haskell code. The DLL was built with GHC 8.0.1 x64, OS is Windows 7.
> (Newer versions of GHC - 8.2.1 and later - do not allow you to build
> such DLLs due to some bugs: ticket #1
> , ticker #2
> ). The C++ project
> was built with MSVC compiler 2015.
>
> We noticed that memory is not released when the library is unloaded.
> I've constructed a baseline example to reproduce the bug. It has three
> files: the first one is *HaskellSources.hs*(see attachments). It
> contains one foreign export function foo. This foreign export function
> is necessary to reproduce the bug. The second file is *CWrapper.cpp*.
> I intentionally didn't include any Haskell function export because
> they make no difference for this example. The last file is the code of
> the main program (see *main.cpp*) that loads the library and unloads
> it at once in an endless loop.
>
> There are two cases: whether HaskellExports.o is included into the
> library or not.
> 1. HaskellExports.o is not included into the library (see attached
> build script/ build_no_hs.sh/). Everything works just fine (i.e.
> amount of memory consumed by the app is the same before DLL loading
> and right after unloading.)
> 2. HaskellExports.o is included into the library
> (see /build_w_hs.sh/). Memory is not released after unloading the DLL,
> after repeated load/unload cycles memory consumption keeps growing
> until finally the application crashes.
>
> Is this a known problem? Is it tracked in your bugtracker? (Or maybe
> it is not a problem at all and I'm doing it wrong).
>
> Thank you for your time.
>
> -- 
> /
> Olga Philippskaya
> ./
>
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


signature.asc
Description: OpenPGP digital signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Haskell DLL (using FFI) causes memory leak.

[Ольга Филиппская]

Hello.

*Summary: *memory is consumed without releasing when a Haskell DLL (that
uses FFI) is loaded and unloaded in an endless loop.

*Details*: I'm working on a C++ project relying on a DLL that uses Haskell
code. The DLL was built with GHC 8.0.1 x64, OS is Windows 7. (Newer
versions of GHC - 8.2.1 and later - do not allow you to build such DLLs due
to some bugs: ticket #1 
, ticker #2 ). The C++
project was built with MSVC compiler 2015.

We noticed that memory is not released when the library is unloaded. I've
constructed a baseline example to reproduce the bug. It has three files:
the first one is *HaskellSources.hs*(see attachments). It contains one
foreign export function foo. This foreign export function is necessary to
reproduce the bug. The second file is *CWrapper.cpp*. I intentionally
didn't include any Haskell function export because they make no difference
for this example. The last file is the code of the main program (see
*main.cpp*) that loads the library and unloads it at once in an endless
loop.

There are two cases: whether HaskellExports.o is included into the library
or not.
1. HaskellExports.o is not included into the library (see attached build
script* build_no_hs.sh*). Everything works just fine (i.e. amount of memory
consumed by the app is the same before DLL loading and right after
unloading.)
2. HaskellExports.o is included into the library (see *build_w_hs.sh*).
Memory is not released after unloading the DLL, after repeated load/unload
cycles memory consumption keeps growing until finally the application
crashes.

Is this a known problem? Is it tracked in your bugtracker? (Or maybe it is
not a problem at all and I'm doing it wrong).

Thank you for your time.

-- 
*Olga Philippskaya.*
#define DLLExport extern "C" __declspec(dllexport)

DLLExport void* c_smth (const int num)
{
return 0;
}
ghc -c HaskellExports.hs
ghc -c CWrapper.cpp 
ghc -shared -no-hs-main -o HaskellExports.dll CWrapper.o HaskellExports.oghc -c HaskellExports.hs
ghc -c CWrapper.cpp 
ghc -shared -no-hs-main -o HaskellExports.dll CWrapper.o

HaskellExports.hs
Description: Binary data
#include 

int main()
{
for (;;)
{
HINSTANCE module = ::LoadLibrary(L"HaskellExports.dll");
::FreeLibrary(module);
}
return 0;
}___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs