Re: [python-win32] Opening existing memory mapped files with pywin32

[Tim Roberts]

On Feb 16, 2021, at 5:49 PM, rhyslloyd1  wrote:
> 
> Thanks for the quick response! I've made some progress since reading. My goal 
> is to be able to talk to the NVIDIA Share API using Python. I found this 
>  
> post on GitHub the other day and I wanted to implement it myself. I've 
> obviously bitten off more than I could chew.

I was wrong.  What you have there is the name of a mapping object, not the name 
of a file.


> Also the "read_line" method doesn't exist apparently. This 
>  is the output now.

Your snippet is private; we can’t read it.  No, “read_line” isn’t going to 
work.  This is not a file, it’s just a chunk of memory — a string of bytes.  
The implication from your link is that this is a JSON string.  If so, you 
should be able to read() the whole thing, do .decode(‘utf-8’) and pass the 
result to a JSON decoder.
— 
Tim Roberts, t...@probo.com
Providenza & Boekelheide, Inc.

___
python-win32 mailing list
python-win32@python.org
https://mail.python.org/mailman/listinfo/python-win32

Re: [python-win32] Opening existing memory mapped files with pywin32

[Eryk Sun]

On 2/16/21, Tim Roberts  wrote:
> Eryk Sun wrote:
>>
>> I'm sorry to say that the only reliable solution is to directly call
>> WinAPI OpenFileMappingW() and MapViewOfFile() using a foreign-function
>> interface (FFI) package such as ctypes or CFFI. I can write up an
>> example if you need help.
>
> Respectfully, I think you have misread the original message.  He is a
> relative beginner, just trying to do a simple file mapping. Based on his
> description, mmap should be able to do what he needs.

Sorry, I'm sure you're right, now that I think about it. I was
responding reflexively based on the OP's code that attempts to
create/open a file mapping with a name and 0 maximum size (i.e.
unknown size; just map it all and query the size), since I've
previously seen people trying and failing to do that with mmap in
various IPC cases. The GUID name didn't help to clarify matters, since
it's just as likely to be generated as a unique object name in the
current session. I also think at unconscious level I didn't want to
assume a confused misreading of the documented function parameters.

Anyway, maybe the discussion will help someone else that's searching
for information on the topic.
___
python-win32 mailing list
python-win32@python.org
https://mail.python.org/mailman/listinfo/python-win32

Re: [python-win32] Opening existing memory mapped files with pywin32

[Tim Roberts]

Eryk Sun wrote:


I'm sorry to say that the only reliable solution is to directly call
WinAPI OpenFileMappingW() and MapViewOfFile() using a foreign-function
interface (FFI) package such as ctypes or CFFI. I can write up an
example if you need help.


Respectfully, I think you have misread the original message.  He is a 
relative beginner, just trying to do a simple file mapping. Based on his 
description, mmap should be able to do what he needs.


I'm willing to be proven wrong, but I'm afraid you may be sending him on 
an unnecessary wild goose chase.


--
Tim Roberts, t...@probo.com
Providenza & Boekelheide, Inc.




smime.p7s
Description: S/MIME Cryptographic Signature
___
python-win32 mailing list
python-win32@python.org
https://mail.python.org/mailman/listinfo/python-win32

Re: [python-win32] Opening existing memory mapped files with pywin32

[Eryk Sun]

On 2/16/21, rhyslloyd1 via python-win32  wrote:
>
> I am trying to open a memory mapped file using Python. I originally used the
> "mmap" module from Python however I had issues with it because I had to use
> a fixed size for the file even though my goal was to open an existing file
> of which the size would be unknown.

I'm sorry to say that the only reliable solution is to directly call
WinAPI OpenFileMappingW() and MapViewOfFile() using a foreign-function
interface (FFI) package such as ctypes or CFFI. I can write up an
example if you need help.

In theory, shared_memory.SharedMemory() in the multiprocessing package
could support this case. But in practice it's still based on the mmap
module. It tries to improve the situation by opening and mapping an
existing mapping via WinAPI OpenFileMappingW() [1] and MapViewOfFile()
[2] in order to query the region size via VirtualQuery(), but this
isn't reliable because it returns the size rounded up to the nearest
page boundary. Calling mmap.mmap(-1, size, tagname=name) with this
queried size may fail the internal MapViewOfFile() call with
ERROR_ACCESS_DENIED (5) if the size is bigger than the maximum size of
the file mapping. (The underlying status from the NT system call more
clearly describes the failure as STATUS_INVALID_VIEW_SIZE.)

The problem with the standard library mmap module and PyWin32's
mmapfile is that they're designed to open a mapping only via
CreateFileMappingW() [3]. When this function is called with hFile as
INVALID_HANDLE_VALUE (i.e. use the system paging file) and the maximum
size as 0 (i.e. dwMaximumSizeHigh=0, dwMaximumSizeLow=0), it always
fails with ERROR_INVALID_PARAMETER (87). It does not check whether
lpName is the name of an existing mapping before validating the
parameters and failing. Hypothetically, it could have been written to
check for this case, since the file handle is ignored anyway for an
existing mapping. But it is what it is, and has been so for about 30
years.

A high-level library function needs to be aware of this limitation and
handle it appropriately by implementing an open disposition (e.g.
create-new, open-always, open-existing) that can dictate the use of
OpenFileMappingW() in addition to or instead of CreateFileMapping().

---
[1] 
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-openfilemappinga
[2] 
https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-mapviewoffile
[3] 
https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-createfilemappingw
___
python-win32 mailing list
python-win32@python.org
https://mail.python.org/mailman/listinfo/python-win32

Re: [python-win32] Opening existing memory mapped files with pywin32

[Tim Roberts]

rhyslloyd1 via python-win32 wrote:


I am trying to open a memory mapped file using Python. I originally 
used the "mmap" module from Python however I had issues with it 
because I had to use a fixed size for the file even though my goal was 
to open an existing file of which the size would be unknown.


The file size isn't really "unknown".  It's just not known in advance.  
There are many ways to find the size of a file.  Plus, the "mmap" module 
accepts 0 as a size parameter, meaning "map the whole file".



I am now using the "mmapfile.mmapfile" function. My code looks like 
this . In the docs 
 it 
says I can use a file name of "None" if I'm planning on opening an 
existing file. This  is the error I get 
when running my code.


There are a couple of misunderstanding here.  It is important to realize 
that the name of the mapping object is quite different from the name of 
the file being mapped.  The Windows memory-mapping concept is quite 
general, and is often used as a way to share memory between multiple 
processes.  In that case, you just want a chunk of memory without 
bothering with a file on disk.  In that case, you'd create a mapping 
object with a name, but no file name.


If you're just mapping an existing file, you need to supply the file 
name, but you don't need to supply a name for the mapping object.  The 
code you showed creates a shared memory object that does not map a file 
on disk.  The shared memory object has a name (the GUID), and another 
application could open that named object, but it doesn't map to a file.


If you want to open a mapping to an existing file, specify the name of 
the file as File= and specify Name=None.


However, you can do all of this with the mmap module as well.  If you 
have a file called "data.bin", you can do


    import mmap
    fn = open('data.bin','rb')
    data = mmap.mmap( fn.fileno(), 0, access=mmap.ACCESS_READ )

That maps the whole file, and len(data) will tell you how large it is.

--
Tim Roberts, t...@probo.com
Providenza & Boekelheide, Inc.




smime.p7s
Description: S/MIME Cryptographic Signature
___
python-win32 mailing list
python-win32@python.org
https://mail.python.org/mailman/listinfo/python-win32

[python-win32] Opening existing memory mapped files with pywin32

[rhyslloyd1 via python-win32]

Hello. Sorry if this is the wrong place to ask my question.

I am trying to open a memory mapped file using Python. I originally used the 
"mmap" module from Python however I had issues with it because I had to use a 
fixed size for the file even though my goal was to open an existing file of 
which the size would be unknown. I am now using the "mmapfile.mmapfile" 
function. My code looks like [this](https://pastebin.com/QygT2wp6). In the 
[docs](http://timgolden.me.uk/pywin32-docs/mmapfile__mmapfile_meth.html) it 
says I can use a file name of "None" if I'm planning on opening an existing 
file. [This](https://pastebin.com/2FVhpDiB) is the error I get when running my 
code.

I'm a novice Python user and I don't know much at all about the Windows API. 
Thanks in advance for any help!___
python-win32 mailing list
python-win32@python.org
https://mail.python.org/mailman/listinfo/python-win32