Re: [python-win32] Opening existing memory mapped files with pywin32

[Eryk Sun]

On 2/21/21, rhyslloyd1  wrote:
>
> In my head my ideal outcome is being able to map the whole memory mapped
> file and not have to specify a size for it incase it changes size however I
> don't understand the concept of a memory mapped file so I maybe my very
> limited understanding is just incorrect, thanks in advance for any response!

The address space of a process is managed in pages of a given fixed
size (e.g. 4 KiB). The state of each page is tracked in an entry in
the process page table, i.e. a page-table entry (PTE). When a page is
sharable between processes, its PTE in a particular process references
a prototype PTE that manages the overall state of the page (e.g.
whether the page is resident in RAM or paged out to a filesystem
paging file or data file).

The system uses a kernel object called a Section to reference the
prototype PTEs in a block of shared memory. In the Windows API, a
Section is called a "file mapping" because the pages mapped into
memory are always a byte range in a filesystem paging file or data
file. At the time of creation via CreateFileMappingW, a file mapping
can be assigned a name, either a global name for all sessions, or a
local name in the current desktop session. This allows a process to
easily try to open a file mapping (e.g. via OpenFileMapping) and use
it to map all or a subset of the shared pages into its address space
(e.g. via MapViewOfFile).

In your case, I gather that you need to open a file mapping named
"{8BA1E16C-FC54-4595-9782-E370A5FBE8DA}" and map the whole block of
shared memory into the current process, without knowing how large it
is. You'll need to query the size of the mapped region (i.e. the
RegionSize) via VirtualQuery. Preferably also wrap the memory in an
easy interface such as a memoryview, which should automatically unmap
the memory when finalized.

Here's a map_section(name, mode) function that implements this. Note
that for reliable operation it requires memoryview.toreadonly(), which
was added in Python 3.8. The underlying ctypes array allows writing to
readonly memory, which will crash the process with an access
violation. That's avoided in 3.8+ by returning a readonly memoryview
when the shared memory is mapped without write access.

import warnings
import ctypes
from ctypes import wintypes

kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

FILE_MAP_READ = SECTION_MAP_READ = 0x0004
FILE_MAP_WRITE = SECTION_MAP_WRITE = 0x0002

class MEMORY_BASIC_INFORMATION(ctypes.Structure):
_fields_ = (('BaseAddress', wintypes.LPVOID),
   ('AllocationBase', wintypes.LPVOID),
   ('AllocationProtect', wintypes.DWORD),
   ('PartitionId', wintypes.WORD),
   ('RegionSize', ctypes.c_size_t),
   ('State', wintypes.DWORD),
   ('Protect', wintypes.DWORD),
   ('Type', wintypes.DWORD))

PMEMORY_BASIC_INFORMATION = ctypes.POINTER(MEMORY_BASIC_INFORMATION)

kernel32.OpenFileMappingW.restype = wintypes.HANDLE
kernel32.OpenFileMappingW.argtypes = (wintypes.DWORD, wintypes.BOOL,
wintypes.LPWSTR)

kernel32.MapViewOfFile.restype = wintypes.LPVOID
kernel32.MapViewOfFile.argtypes = (wintypes.HANDLE, wintypes.DWORD,
wintypes.DWORD, wintypes.DWORD, ctypes.c_size_t)

kernel32.UnmapViewOfFile.restype = wintypes.BOOL
kernel32.UnmapViewOfFile.argtypes = (wintypes.LPCVOID,)

kernel32.VirtualQuery.restype = ctypes.c_size_t
kernel32.VirtualQuery.argtypes = (wintypes.LPCVOID,
PMEMORY_BASIC_INFORMATION, ctypes.c_size_t)

class BaseSharedMem(ctypes.Array):
_type_ = ctypes.c_char
_length_ = 0

def __del__(self, *, UnmapViewOfFile=kernel32.UnmapViewOfFile,
warn=warnings.warn):
if not UnmapViewOfFile(self):
warn("UnmapViewOfFile failed", ResourceWarning, source=self)


def map_section(name, mode='r'):
mbi = MEMORY_BASIC_INFORMATION()

if mode == 'r':
access = FILE_MAP_READ
elif mode == 'w':
access = FILE_MAP_WRITE
elif mode in ('rw', 'wr'):
access = FILE_MAP_READ | FILE_MAP_WRITE
else:
raise ValueError('invalid mode')

h = kernel32.OpenFileMappingW(access, False, name)
if not h:
raise ctypes.WinError(ctypes.get_last_error())

try:
address = kernel32.MapViewOfFile(h, access, 0, 0, 0)
if not address:
raise ctypes.WinError(ctypes.get_last_error())
finally:
kernel32.CloseHandle(h)

result = kernel32.VirtualQuery(address, ctypes.byref(mbi),
ctypes.sizeof(mbi))
if not result:
ex = ctypes.WinError(ctypes.get_last_error())
if not kernel32.UnmapViewOfFile(address):
ex2 = ctypes.WinError(ctypes.get_last_error())
raise ex2 from ex
raise ex

array_t = type('SharedMem_{}'.format(mbi.RegionSize),
(BaseSharedMem,), {'_length_': mbi.RegionSize})
mv = memoryview(array_t.from_address(address)).cast('B')

if 'w' not in mode:
return 

Re: [python-win32] Opening existing memory mapped files with pywin32

[Tim Roberts]

On Feb 16, 2021, at 5:49 PM, rhyslloyd1  wrote:
> 
> Thanks for the quick response! I've made some progress since reading. My goal 
> is to be able to talk to the NVIDIA Share API using Python. I found this 
>  
> post on GitHub the other day and I wanted to implement it myself. I've 
> obviously bitten off more than I could chew.

I was wrong.  What you have there is the name of a mapping object, not the name 
of a file.


> Also the "read_line" method doesn't exist apparently. This 
>  is the output now.

Your snippet is private; we can’t read it.  No, “read_line” isn’t going to 
work.  This is not a file, it’s just a chunk of memory — a string of bytes.  
The implication from your link is that this is a JSON string.  If so, you 
should be able to read() the whole thing, do .decode(‘utf-8’) and pass the 
result to a JSON decoder.
— 
Tim Roberts, t...@probo.com
Providenza & Boekelheide, Inc.

___
python-win32 mailing list
python-win32@python.org
https://mail.python.org/mailman/listinfo/python-win32

Re: [python-win32] Opening existing memory mapped files with pywin32

[Eryk Sun]

On 2/16/21, Tim Roberts  wrote:
> Eryk Sun wrote:
>>
>> I'm sorry to say that the only reliable solution is to directly call
>> WinAPI OpenFileMappingW() and MapViewOfFile() using a foreign-function
>> interface (FFI) package such as ctypes or CFFI. I can write up an
>> example if you need help.
>
> Respectfully, I think you have misread the original message.  He is a
> relative beginner, just trying to do a simple file mapping. Based on his
> description, mmap should be able to do what he needs.

Sorry, I'm sure you're right, now that I think about it. I was
responding reflexively based on the OP's code that attempts to
create/open a file mapping with a name and 0 maximum size (i.e.
unknown size; just map it all and query the size), since I've
previously seen people trying and failing to do that with mmap in
various IPC cases. The GUID name didn't help to clarify matters, since
it's just as likely to be generated as a unique object name in the
current session. I also think at unconscious level I didn't want to
assume a confused misreading of the documented function parameters.

Anyway, maybe the discussion will help someone else that's searching
for information on the topic.
___
python-win32 mailing list
python-win32@python.org
https://mail.python.org/mailman/listinfo/python-win32

Re: [python-win32] Opening existing memory mapped files with pywin32

[Tim Roberts]

Eryk Sun wrote:


I'm sorry to say that the only reliable solution is to directly call
WinAPI OpenFileMappingW() and MapViewOfFile() using a foreign-function
interface (FFI) package such as ctypes or CFFI. I can write up an
example if you need help.


Respectfully, I think you have misread the original message.  He is a 
relative beginner, just trying to do a simple file mapping. Based on his 
description, mmap should be able to do what he needs.


I'm willing to be proven wrong, but I'm afraid you may be sending him on 
an unnecessary wild goose chase.


--
Tim Roberts, t...@probo.com
Providenza & Boekelheide, Inc.




smime.p7s
Description: S/MIME Cryptographic Signature
___
python-win32 mailing list
python-win32@python.org
https://mail.python.org/mailman/listinfo/python-win32

Re: [python-win32] Opening existing memory mapped files with pywin32

[Eryk Sun]

On 2/16/21, rhyslloyd1 via python-win32  wrote:
>
> I am trying to open a memory mapped file using Python. I originally used the
> "mmap" module from Python however I had issues with it because I had to use
> a fixed size for the file even though my goal was to open an existing file
> of which the size would be unknown.

I'm sorry to say that the only reliable solution is to directly call
WinAPI OpenFileMappingW() and MapViewOfFile() using a foreign-function
interface (FFI) package such as ctypes or CFFI. I can write up an
example if you need help.

In theory, shared_memory.SharedMemory() in the multiprocessing package
could support this case. But in practice it's still based on the mmap
module. It tries to improve the situation by opening and mapping an
existing mapping via WinAPI OpenFileMappingW() [1] and MapViewOfFile()
[2] in order to query the region size via VirtualQuery(), but this
isn't reliable because it returns the size rounded up to the nearest
page boundary. Calling mmap.mmap(-1, size, tagname=name) with this
queried size may fail the internal MapViewOfFile() call with
ERROR_ACCESS_DENIED (5) if the size is bigger than the maximum size of
the file mapping. (The underlying status from the NT system call more
clearly describes the failure as STATUS_INVALID_VIEW_SIZE.)

The problem with the standard library mmap module and PyWin32's
mmapfile is that they're designed to open a mapping only via
CreateFileMappingW() [3]. When this function is called with hFile as
INVALID_HANDLE_VALUE (i.e. use the system paging file) and the maximum
size as 0 (i.e. dwMaximumSizeHigh=0, dwMaximumSizeLow=0), it always
fails with ERROR_INVALID_PARAMETER (87). It does not check whether
lpName is the name of an existing mapping before validating the
parameters and failing. Hypothetically, it could have been written to
check for this case, since the file handle is ignored anyway for an
existing mapping. But it is what it is, and has been so for about 30
years.

A high-level library function needs to be aware of this limitation and
handle it appropriately by implementing an open disposition (e.g.
create-new, open-always, open-existing) that can dictate the use of
OpenFileMappingW() in addition to or instead of CreateFileMapping().

---
[1] 
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-openfilemappinga
[2] 
https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-mapviewoffile
[3] 
https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-createfilemappingw
___
python-win32 mailing list
python-win32@python.org
https://mail.python.org/mailman/listinfo/python-win32

Re: [python-win32] Opening existing memory mapped files with pywin32

[Tim Roberts]

rhyslloyd1 via python-win32 wrote:


I am trying to open a memory mapped file using Python. I originally 
used the "mmap" module from Python however I had issues with it 
because I had to use a fixed size for the file even though my goal was 
to open an existing file of which the size would be unknown.


The file size isn't really "unknown".  It's just not known in advance.  
There are many ways to find the size of a file.  Plus, the "mmap" module 
accepts 0 as a size parameter, meaning "map the whole file".



I am now using the "mmapfile.mmapfile" function. My code looks like 
this . In the docs 
 it 
says I can use a file name of "None" if I'm planning on opening an 
existing file. This  is the error I get 
when running my code.


There are a couple of misunderstanding here.  It is important to realize 
that the name of the mapping object is quite different from the name of 
the file being mapped.  The Windows memory-mapping concept is quite 
general, and is often used as a way to share memory between multiple 
processes.  In that case, you just want a chunk of memory without 
bothering with a file on disk.  In that case, you'd create a mapping 
object with a name, but no file name.


If you're just mapping an existing file, you need to supply the file 
name, but you don't need to supply a name for the mapping object.  The 
code you showed creates a shared memory object that does not map a file 
on disk.  The shared memory object has a name (the GUID), and another 
application could open that named object, but it doesn't map to a file.


If you want to open a mapping to an existing file, specify the name of 
the file as File= and specify Name=None.


However, you can do all of this with the mmap module as well.  If you 
have a file called "data.bin", you can do


    import mmap
    fn = open('data.bin','rb')
    data = mmap.mmap( fn.fileno(), 0, access=mmap.ACCESS_READ )

That maps the whole file, and len(data) will tell you how large it is.

--
Tim Roberts, t...@probo.com
Providenza & Boekelheide, Inc.




smime.p7s
Description: S/MIME Cryptographic Signature
___
python-win32 mailing list
python-win32@python.org
https://mail.python.org/mailman/listinfo/python-win32

[python-win32] Opening existing memory mapped files with pywin32

[rhyslloyd1 via python-win32]

Hello. Sorry if this is the wrong place to ask my question.

I am trying to open a memory mapped file using Python. I originally used the 
"mmap" module from Python however I had issues with it because I had to use a 
fixed size for the file even though my goal was to open an existing file of 
which the size would be unknown. I am now using the "mmapfile.mmapfile" 
function. My code looks like [this](https://pastebin.com/QygT2wp6). In the 
[docs](http://timgolden.me.uk/pywin32-docs/mmapfile__mmapfile_meth.html) it 
says I can use a file name of "None" if I'm planning on opening an existing 
file. [This](https://pastebin.com/2FVhpDiB) is the error I get when running my 
code.

I'm a novice Python user and I don't know much at all about the Windows API. 
Thanks in advance for any help!___
python-win32 mailing list
python-win32@python.org
https://mail.python.org/mailman/listinfo/python-win32