[issue14597] Cannot unload dll in ctypes until script exits

[Eryk Sun]

Eryk Sun  added the comment:

Automatically decrementing the reference count of a shared library (e.g. via 
WinAPI FreeLibrary or POSIX dlclose) when a CDLL instance is finalized would 
require significant design changes to ensure that all ctypes pointers, scalars, 
and aggregates that reference memory in the DLL also always reference the CDLL 
instance in their _objects attribute. I don't think this feature request is 
worth the work that involves, or the risk of crashing if some case is 
overlooked in the implementation. Nothing prevents a script from implementing 
this manually on a case by case basis.

--
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Eric Snow]

Change by Eric Snow :


--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Mateusz Loskot]

Changes by Mateusz Loskot :


--
nosy: +mloskot

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Mark Mikofski]

Mark Mikofski added the comment:

Snippets of "proprietary" code

`setup.py`
--
# run clean or build libraries if they don't exist
if 'clean' in sys.argv:
try:
os.remove(os.path.join(LIB_DIR, SOLPOSAM_LIB_FILE))
os.remove(os.path.join(LIB_DIR, SPECTRL2_LIB_FILE))
except OSError as err:
sys.stderr.write('%s\n' % err)

`core.py`
-
# load the DLL
solposAM_dll = ctypes.cdll.LoadLibrary(SOLPOSAMDLL)
_solposAM = solposAM_dll.solposAM

After deleting files, setup.py tries to build, then runs test_cdlls.py a 
unittest that contains test_solposAM() that calls solposAM() and imports 
core.py that contains solposAM.

Case 1:
---
I put the load library calls inside the function solposAM().
Function calls works as expected.
I am able to delete dlls after interpreter exits and restarts using os.remove()

Case 2:
---
I put the load library calls at the module level.
Function calls works as expected.
If I try to delete dlls after interpreter exits and restarts using os.remove() 
I get access denied, but I can delete them from windows explorer.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Brian Curtin]

Changes by Brian Curtin :


--
nosy:  -brian.curtin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Eryk Sun]

Eryk Sun added the comment:

As to not being able to delete a loaded DLL on Windows, a workaround that may 
help in some instances is to rename it to a temporary name on the same volume. 
This is useful for upgrading in place. If you have admin privileges you can 
even flag the renamed DLL to be deleted when the system reboots:

MoveFileExW(renamed_dll_path, NULL, MOVEFILE_DELAY_UNTIL_REBOOT)

Renaming a loaded DLL succeeds because the system opens the file with delete 
sharing, which allows renaming (i.e. relinking) an open file. 

To verify this, let's check the associated file object that's used by the 
mapping for python35.dll. We can see this in a local kernel debugging session. 
The following is in 64-bit Windows 7.

Attach to the Python process address space:

lkd> .tlist python.exe
 0n2296 python.exe
lkd> !process 0n2296 0
Searching for Process with Cid == 8f8
Cid handle table at f8a003506000 with 951 entries in use

PROCESS fa800712fb10
SessionId: 0  Cid: 08f8Peb: 7fdf000  ParentCid: 02b8
DirBase: 20780d000  ObjectTable: f8a0189fe1a0  HandleCount:  67.
Image: python.exe

lkd> .process /p /r 0xfa800712fb10 
Implicit process is now fa80`0712fb10
Loading User Symbols
...

Get the base address for python35.dll.

lkd> lm m python35
start end module name
`66fb `67393000   python35   (deferred)

Get the associated Virtual Address Descriptor (VAD).

lkd> !vad 66fb
VAD level  start  endcommit
fa800855f230 (-1)  66fb0  67392  151 Mapped  Exe
 EXECUTE_WRITECOPY
 \Program Files\Python35\python35.dll

Get the File object reference from the associated control area.

lkd> ?? ((nt!_MMVAD 
*)0xfa800855f230)->Subsection->ControlArea->FilePointer
struct _EX_FAST_REF
   +0x000 Object   : 0xfa80`093f1503 Void
   +0x000 RefCnt   : 0y0011
   +0x000 Value: 0xfa80`093f1503

The low nibble in the above address is information that needs to be masked out, 
so the address is actually 0xfa80`093f1500. Verify we have the correct file 
object by checking the filename.
   
lkd> ?? ((nt!_FILE_OBJECT *)0xfa80`093f1500)->FileName
struct _UNICODE_STRING
 "\Program Files\Python35\python35.dll"
   +0x000 Length   : 0x48
   +0x002 MaximumLength: 0x78
   +0x008 Buffer   : 0xf8a0`04b23690
 "\Program Files\Python35\python35.dll"

Finally, check the sharing mode. We see below that the file for a DLL mapping 
is loaded with read and delete sharing, but not write sharing.

lkd> ?? ((nt!_FILE_OBJECT *)0xfa80`093f1500)->SharedRead
unsigned char 0x01 ''
lkd> ?? ((nt!_FILE_OBJECT *)0xfa80`093f1500)->SharedDelete
unsigned char 0x01 ''
lkd> ?? ((nt!_FILE_OBJECT *)0xfa80`093f1500)->SharedWrite
unsigned char 0x00 ''

Thus when deleting a DLL fails, it's not due to ERROR_SHARING_VIOLATION (32). 
The sharing mode would actually permit deleting the file. Instead the error is 
ERROR_ACCESS_DENIED (5), which results from the system call 
NtSetInformationFile returning STATUS_CANNOT_DELETE (0xC121), because the 
file currently has a mapped image or data section. But that doesn't stop us 
from renaming it to get it out of the way of an in-place upgrade.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Eryk Sun]

Eryk Sun added the comment:

The _ctypes extension module provides dlclose on POSIX and FreeLibrary on 
Windows. For the reasons already stated in this issue, ctypes should not 
automatically call these functions, but maybe they should be documented and 
imported to the ctypes module instead of being buried in the private _ctypes 
extension module. You could also implement them with ctypes itself, but the 
existing functions in the _ctypes module already set a proper exception if the 
call fails and properly use pointers (an HMODULE is a pointer).

--
assignee:  -> docs@python
components: +Documentation
nosy: +docs@python
versions: +Python 2.7, Python 3.5, Python 3.6 -Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Eryk Sun]

Eryk Sun added the comment:

> My setup.py creates a .dll then later loads the dll for testing 
> using ctypes. Subsequent runs of setup.py that would force 
> rebuilding the .dll attempt to delete the old dll first if it 
> exists, but I get permission denied.
>
> Oddly, if I put the load library call inside a function, then, 
> after exiting the interpreter the dll can be deleted.

I'm having trouble imagining what the problem could be here. Can you attach a 
minimal example script that reproduces the problem?

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Mark Mikofski]

Mark Mikofski added the comment:

I think I have this issue even after interpreter exits. My setup.py creates a 
.dll then later loads the dll for testing using ctypes. Subsequent runs of 
setup.py that would force rebuilding the .dll attempt to delete the old dll 
first if it exists, but I get permission denied.

Oddly, if I put the load library call inside a function, then, after exiting 
the interpreter the dll can be deleted.

Windos 7 x64
Python 2.7.10

Sorry if this is by design, a Windows feature, unrelated or the wrong issue. I 
search SO and these bugs, and only found answers related to unloading dll 
during script.

--
nosy: +bwanamarko

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc added the comment:

Did you consider creating a copy with a random suffix?

--
nosy: +amaury.forgeotdarc

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14597
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Piotr Lopusiewicz]

Piotr Lopusiewicz added the comment:

For most ctypes uses, closing is irrelevant, since people typically access 
system libraries that are independently loaded anyway, so closing them would 
not have any effect.

My use case is this: I test some things, then I need to recompile my .dll 
without closing the interpreter (as building stuff to begin tests is quiet 
expensive) then reload it. I can't recompile it because Windows blocks the .dll 
file and gcc can't overwrite it. Deleting the reference and running 
gc.collect() doesn't work.
It would be very nice to have a way to do this.

--
nosy: +Piotr.Lopusiewicz

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14597
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14597
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Pat Lynch]

New submission from Pat Lynch plync...@gmail.com:

If I load a dll in ctypes, then delete that loaded DLL instance, the DLL is not 
unloaded until the script finishes and exits.

I'm trying to write some unit tests in python to exercise that DLL where each 
test case loads a DLL, does some work, then unloads the DLL.  Unfortunately the 
DLL only gets unloaded when the unit tests finish.

I've tried forcing the garbage collector to run to get the DLL to unload.  It 
did nothing...

# load the DLL
parser_dll = CDLL(dllpath)

# do some work here

# 'unload' the dll (or as close as I can get it to it)
if (parser_dll):
del parser_dll

--
components: ctypes
messages: 158433
nosy: plynch76
priority: normal
severity: normal
status: open
title: Cannot unload dll in ctypes until script exits
type: enhancement
versions: Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14597
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Pat Lynch]

Pat Lynch plync...@gmail.com added the comment:

I should mention also, that this is mostly an issue for me on Win7 x64.  It
does behave 'slightly' better on WinXP x86.

(I have the 64-bit version of python installed on Win7 x64  the 32-bit
version installed on WinXP)

thanks,
Pat.

On 16 April 2012 14:09, Pat Lynch rep...@bugs.python.org wrote:


 New submission from Pat Lynch plync...@gmail.com:

 If I load a dll in ctypes, then delete that loaded DLL instance, the DLL
 is not unloaded until the script finishes and exits.

 I'm trying to write some unit tests in python to exercise that DLL where
 each test case loads a DLL, does some work, then unloads the DLL.
  Unfortunately the DLL only gets unloaded when the unit tests finish.

 I've tried forcing the garbage collector to run to get the DLL to unload.
  It did nothing...

 # load the DLL
 parser_dll = CDLL(dllpath)

 # do some work here

 # 'unload' the dll (or as close as I can get it to it)
 if (parser_dll):
del parser_dll

 --
 components: ctypes
 messages: 158433
 nosy: plynch76
 priority: normal
 severity: normal
 status: open
 title: Cannot unload dll in ctypes until script exits
 type: enhancement
 versions: Python 2.7

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue14597
 ___


--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14597
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

In general it is difficult to impossible to get Python2 to unload modules 
before the interpreter shuts down.  See issue 9072.  I'm not savvy enough with 
the C stuff to know if the fact that you loaded it via ctypes changes anything, 
but I doubt it.

Note that the implication of that issue is that if you could move to Python3 
there might be a way to do it, but that would indeed be an enhancement as there 
is no direct support for it yet.

--
nosy: +r.david.murray
versions: +Python 3.3 -Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14597
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +brian.curtin, meador.inge, tim.golden

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14597
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Pat Lynch]

Pat Lynch plync...@gmail.com added the comment:

thanks for the very quick response.

Since LoadLibrary is called in the constructor, why can't FreeLibrary be
called in the destructor?  or at least expose a function to unload that
calls FreeLibrary?

http://msdn.microsoft.com/en-us/library/windows/desktop/ms683152%28v=vs.85%29.aspx

thanks again,
Pat.

On 16 April 2012 14:30, R. David Murray rep...@bugs.python.org wrote:


 R. David Murray rdmur...@bitdance.com added the comment:

 In general it is difficult to impossible to get Python2 to unload modules
 before the interpreter shuts down.  See issue 9072.  I'm not savvy enough
 with the C stuff to know if the fact that you loaded it via ctypes changes
 anything, but I doubt it.

 Note that the implication of that issue is that if you could move to
 Python3 there might be a way to do it, but that would indeed be an
 enhancement as there is no direct support for it yet.

 --
 nosy: +r.david.murray
 versions: +Python 3.3 -Python 2.7

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue14597
 ___


--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14597
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Martin v . Löwis]

Martin v. Löwis mar...@v.loewis.de added the comment:

In principle, it should be possible (but perhaps not desirable, see below) to 
call FreeLibrary in a CDLL's __del__. However, since this would be a new 
feature, it can't go into 2.7. Patches are welcome; make sure to support both 
FreeLIbrary and dlclose.

There is a general issue with closing/freeing DLLs: if they are still 
referenced somewhere (e.g. in an atexit function, a C++ virtual method table, 
or on the call stack of another thread), then a later access to the code will 
crash the interpreter. In a typical DLL today (including all Python extension 
modules), the likelihood of crashes is close to 100%. For that reason, it's 
probably not a good idea to have ctypes auto-close DLLs; instead, it should be 
an opt-in mechanism.

For most ctypes uses, closing is irrelevant, since people typically access 
system libraries that are independently loaded anyway, so closing them would 
not have any effect.

--
nosy: +loewis

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14597
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Pat Lynch]

Pat Lynch plync...@gmail.com added the comment:

ok, that's fair enough if most usage of ctypes is from people accessing
system libraries :)

I wouldn't have thought my usage was that weird though (given the strength
of using python for unit testing).

In local tests, adding a function CDLL::ForceUnloadDll  (which just calls
FreeLibrary(self._handle)) seems to work well.  I haven't done intensive
testing though at this point.  I could be missing something though.

thanks,
Pat.

On 16 April 2012 14:45, Martin v. Löwis rep...@bugs.python.org wrote:


 Martin v. Löwis mar...@v.loewis.de added the comment:

 In principle, it should be possible (but perhaps not desirable, see below)
 to call FreeLibrary in a CDLL's __del__. However, since this would be a new
 feature, it can't go into 2.7. Patches are welcome; make sure to support
 both FreeLIbrary and dlclose.

 There is a general issue with closing/freeing DLLs: if they are still
 referenced somewhere (e.g. in an atexit function, a C++ virtual method
 table, or on the call stack of another thread), then a later access to the
 code will crash the interpreter. In a typical DLL today (including all
 Python extension modules), the likelihood of crashes is close to 100%. For
 that reason, it's probably not a good idea to have ctypes auto-close DLLs;
 instead, it should be an opt-in mechanism.

 For most ctypes uses, closing is irrelevant, since people typically access
 system libraries that are independently loaded anyway, so closing them
 would not have any effect.

 --
 nosy: +loewis

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue14597
 ___


--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14597
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[Pat Lynch]

Pat Lynch plync...@gmail.com added the comment:

Just to update:-

I've run this pretty extensively on multiple systems (XP x86  Win7 64-bit)
and it appears to behave as expected (haven't checked it on Linux).  I have
that code being called in 100s of unit tests.

For python 3.1, would it make sense to add it as a ForceUnload function??
- for safety bail out if handle was not None when passed into the
constructor?  i.e. if somebody has accessed an independently loaded DLL,
they will pass in the handle when constructing the CDLL object.  Disallow
ForceUnload in that case.  ForceUnload will only be allowed in cases where
we created that type by passing in the path to the DLL.

I'll be using this code as a local patch, so no rush to put it into 3.1 etc.

thanks for all the info - much appreciated :)

Pat.

On 16 April 2012 15:04, Pat Lynch rep...@bugs.python.org wrote:


 Pat Lynch plync...@gmail.com added the comment:

 ok, that's fair enough if most usage of ctypes is from people accessing
 system libraries :)

 I wouldn't have thought my usage was that weird though (given the strength
 of using python for unit testing).

 In local tests, adding a function CDLL::ForceUnloadDll  (which just calls
 FreeLibrary(self._handle)) seems to work well.  I haven't done intensive
 testing though at this point.  I could be missing something though.

 thanks,
 Pat.

 On 16 April 2012 14:45, Martin v. Löwis rep...@bugs.python.org wrote:

 
  Martin v. Löwis mar...@v.loewis.de added the comment:
 
  In principle, it should be possible (but perhaps not desirable, see
 below)
  to call FreeLibrary in a CDLL's __del__. However, since this would be a
 new
  feature, it can't go into 2.7. Patches are welcome; make sure to support
  both FreeLIbrary and dlclose.
 
  There is a general issue with closing/freeing DLLs: if they are still
  referenced somewhere (e.g. in an atexit function, a C++ virtual method
  table, or on the call stack of another thread), then a later access to
 the
  code will crash the interpreter. In a typical DLL today (including all
  Python extension modules), the likelihood of crashes is close to 100%.
 For
  that reason, it's probably not a good idea to have ctypes auto-close
 DLLs;
  instead, it should be an opt-in mechanism.
 
  For most ctypes uses, closing is irrelevant, since people typically
 access
  system libraries that are independently loaded anyway, so closing them
  would not have any effect.
 
  --
  nosy: +loewis
 
  ___
  Python tracker rep...@bugs.python.org
  http://bugs.python.org/issue14597
  ___
 

 --

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue14597
 ___


--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14597
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14597] Cannot unload dll in ctypes until script exits

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Current default will become 3.3.  3.1 has been out for a while :)

Your thought sounds reasonable, though Martin may have further input.

Would you are to propose a patch?  Otherwise most like nothing will happen with 
this issue.  3.3 Beta is scheduled for mid-June, so that would be the deadline 
for new features.

(PS: Could you please trim your replies when replying to tracker messages?  The 
quoted text makes the tracker issue hard to read.)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14597
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com