[Python-ideas] Add a PyObject_VaCallFunction to C API??
The C API has a function PyObject_CallFunction( PyObject*, const char* fmt, ...
). It is a variadic function hence I couldn't pass a va_list to it to invoke
the call. My question is, is it technically possible to provide a companion
PyObject_VaCallFunction which takes a va_list, just like Py_VaBuildValue is to
Py_BuildValue?
Here's what I have found so far. I read the Objects/call.c about
PyObject_CallFunction there, and actually found a function
_PyObject_CallFunctionVa, which is close to what I want, although it has two
additional parameters. By taking a look at the implementation of
PyObject_CallFunction, naively I was thinking a Va version of that could have
been similarly implemented ( I'm ignoring the size_t issue ). I'd appreciate it
if someone could enlighten me on this subject. Question rephrased: is there a
technical reason why there is no PyObject_VaCallFunction in the API? If not,
would it be possible to add it in?
Motivation: I write scientific simulation code to be run on large clusters. The
data generated can be huge. Although I know parallelization using C++, I don't
know it using Python. A direct consequence is formidable data processing time
with python, which is driving me nuts. I have two options, either parallelize
python code or embed the python in C++. I'm favoring the latter, not just
because I don't know about python parallelization, but more importantly because
by embedding python in C++, I get to keep using the highly specialized C++
classes and some calculation routines, without having to duplicate essentially
the same thing in Python, which is very time-saving. I could have just used the
C API, but here is the all-time drawback of linking with Python libraries ---
it messes up the linker runtime search path. Usually a Python installation has
under its lib/ many common libraries such as libz.so, libhdf5.so ( or I
probably should have said earlier that I primarily work on Linux )
. They have different versions to, say, the libhdf5.so I'm using in the code.
The upshot is that by linking my program with both the major library in which
HDF5 links to one version, and the Python libraries in which HDF5 links to
another, the runtime search path is always mixed up, which is not safe.
So what I thought about doing is that I'm gonna create a C++ wrapper on Python,
one that doesn't expose the raw Python to the client code at all. ( For
example, there is no #include in any header of that library. Plus I
could use C++ OOP to automate away keeping tracking of Py_INCREF/Py_DECREF. )
In fact it's very doable. Everything is straightforward, except variadic
functions, such as PyObject_CallFunction, Py_BuildValue, and so on. Let me use
PyObject_CallFunction to illustrate the problem.
Let's say that I want to provide a wrapper to PyObject_CallFunction to the
client, like this ( I'm taking the return type to void for simplicity )
void MyCallFunction(PyObject* obj, const char* fmt, ... );
I go ahead and put this line in the header "my_python_wrapper.h". Then in the
"my_python_wrapper.c" file I would like to write the following implementation.
void MyCallFunction(PyObject* obj, const char* fmt, ... ) {
va_list args;
va_start(args,fmt);
PyObject(obj, fmt, args); // This function doesn't exist in the API ( yet? )
va_end(args);
}
This way, only the "my_python_wrapper.c" needs to link with Python Libraries.
Any user of my_python_wrapper doesn't need to, which seems nice. ( In cmake
lingo, I only need to target_link_libraries( my_python_wrapper PRIVATE
${PYTHON_LIBRARIES} ), instead of PUBLIC ) As one can see, the crux is that not
all variadic functions in the API has a companion Va-ed version. So far I only
found Py_VaBuildValue.
I've worked out MyCallFunction() in my actual code in the same manner described
above, but with Py_VaBuildValue. What I did was I send all variadic arguments
to a MyBuildValue(PyObject*, const char*, ...), the in the .c file,
MyBuildValue will generate a va_list and pass it onto Py_VaBuildValue, then I
force the outcome to be a tuple and pass it to PyObject_CallObject and it
works! Nontheless, this approach seems less straightforward to having a
PyObject_VaCallFunction, so I'm guessing it may have performance penalty.
I really appreciate it whoever takes their time to read this essay of mine! I
apologize if I failed to use the idiomatic mark-up. Any comments, questions on
this subject are welcome!
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/VHAUIILV3I7PULKZVILUVY5JEDK6OR2B/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Python, Be Bold!
On Jan 5, 2020, at 00:17, James Lu wrote: > > > I use macOS, and using Python is very confusing. > > - Apple's bundled Python 2.7. Apple has made a mess of things, but they’ve actually fixed that mess in 10.15—they now give you 3.7 and 2.7, and neither one is broken or weird. That being said, I don’t think anyone trusts them to not to change their minds again and leave everything without updates for half a decade to the point where their pre-installed Python is more of a nuisance than helpful. So everyone is still using Homebrew or Anaconda, and you probably should as well. I think there’s more reason than ever for developers to learn virtual environments early and never have to think about all the different Pythons they have installed except whole setting up a new env. But you’re apparently concerned about end users, not devs here. So: > Python, Be Bold captures the spirit of it should not be a shame to have the > interpreter/VM installed on end-users machines. It also facilitates the > work of other python devs. You installed it because of one Python program, > other programs benefit from it. It also proposes enhancements to the VM to > better facilitate that. There’s already a better solution for end users today. You don’t install Python to run Spam, you just `brew install spam`, and if Spam requires Python 3.7 and you don’t have it, Homebrew installs it for you. And if you just `brew install eggs` and Eggs also requires 3.7 and you’ve already got it from installing Spam, it uses the same 3.7. You don’t even have to know you have Python installed, much less know which versions you have. Of course this means someone has to build a Homebrew recipe—and RPM and DEB and Portage and Choco—for every application, but people have already been doing that for decades, and it works. For users who do want to know they have Python installed, there are a lot of “power user” apps that they can install the same way they do libraries: `pip install cheese`. This also already works today. I suppose it would be even better if the whole world used one single package manager that worked the same on every platform, and pip/setuptools could just be a trivial way to hook into that rather than an entirely separate system. But I don’t see any way to solve that from inside the Python ecosystem. The closest you can get is probably Anaconda—which already exists and already works fine. > One big reason the web is a popular platform today among the general public > > is because it offers strong sandboxing, The web was popular long before it offered strong sandboxing. That sandboxing had to be wedged in after the fact, because the web had become so popular that millions of people who didn’t know how to deal with security issues were visiting dynamic websites with IE4 and Netscape and getting hacked, tracked, scammed, and spammed. > and privelges are granted per-site, opt out by default. > Another reason the web is popular is because it loads quickly. > I suggest python apps feature sandboxing with a way to opt-in to special > permissions. How? In CPython, unlike the browser, you have direct access to the filesystem, and the sockets layer. You even have hooks to load any dylib/so/dll and call functions out of it. So it can only be sandboxed the same way C can be sandboxed. Which means only the OS can do it. There have been attempts to build a useful environment out of Python by removing all of those features, and then adding in other, more restricted ways of doing the things you really need to do (access sandboxed local storage, talk to web services, accept connections as a web service, etc.). The most successful is probably Google App Engine. But GAE Python doesn’t feel the same as Python. It can’t do a lot of the things you can do with a normal Python install. And with version 2.0, Google scrapped all of that and went back to building things around a normal Python instead. If you wanted to do the same thing for end users instead of web services, you’d need to provide some way to run a GUI, to accept local files by some mechanism like drag, etc. At which point you’re just designing a web browser. You might as well just write a Python interpreter that runs in the browser, and then add hooks to talk to the DOM and do WebSockets and make AJAX requests and so on the same way JS does. And there are already multiple projects to do that. > Perhaps, like the web, it could have a uniform distribution mechanism. I > suggest DNS. Why not just use URIs the same way browser JS and most of the existing browser Python projects do? What benefit do you get from building a custom package manager when every app has to be sandboxed and therefore can’t share packages with any other app? > I am suggesting Python compete with the web by implementing strong language > sandboxing. > I imagine a browser that can load regular web and "PyWeb." > > A web browser is both a kernel and a VM for
[Python-ideas] Re: Python, Be Bold!
On Sun, Jan 5, 2020 at 7:19 PM James Lu wrote: > > I use macOS, and using Python is very confusing. > > - Apple's bundled Python 2.7. > - Anaconda (Python scientific stack package manager) Python and conda. > - Homebrew (3rd party package manager for macOS) Python and pip. > I also believe that there is a PSF Python installer, but I am not sure. Use Homebrew. It's a good package manager for the Mac and it gets you away from Apple's ancient version of Python. > Python, Be Bold captures the spirit of it should not be a shame to have the > interpreter/VM installed on end-users machines. It also facilitates the > work of other python devs. You installed it because of one Python program, > other programs benefit from it. It also proposes enhancements to the VM to > better facilitate that. I still don't understand this concept of it "not being a shame" to have Python installed. How is that different from the way it now is? You install Python. Now Python is installed. How does the VM need to be enhanced to change this? Other than *not* bundling Python with your application? > One big reason the web is a popular platform today among the general public > is because it offers strong sandboxing, > and privelges are granted per-site, opt out by default. > Another reason the web is popular is because it loads quickly. > I suggest python apps feature sandboxing with a way to opt-in to special > permissions. > Perhaps, like the web, it could have a uniform distribution mechanism. I > suggest DNS. Eh? > I am suggesting Python compete with the web by implementing strong language > sandboxing. > I imagine a browser that can load regular web and "PyWeb." > > A web browser is both a kernel and a VM for the web. The kernel interfaces > with the underlying OS: Linux, windows, MacOS. > I do not see "PyWeb" as a kernel, but I do see it as a VM. PyWeb would merely > provide a secure gatekeeper to the underlying operating system. > Like the Mac App Store, PyWeb could give each app its own sandboxed file > system. > This would also help introduce young people to Python. Like how the DevTools > console has taught many kids JavaScript. > I imagine being able to run Tensorflow or Calibre inside a Python "browser." You can already run Python code in a web environment. There are a number of web sites that allow this, including pythontutor.com (which also does detailed analysis and visualization, so it's not JUST Python-in-the-web), and if you want to, you can compile PyPy to Asm.js and run the entire interpreter right there in the browser. But what's the point? How does that make it easier to do anything? You just have to live within the restrictions of either a browser tab, or the things you can do remotely. ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EXOVTEYY4TOBH4MGJ7C6ZOOEOI2TASQ7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Python, Be Bold!
I use macOS, and using Python is very confusing. - Apple's bundled Python 2.7. - Anaconda (Python scientific stack package manager) Python and conda. - Homebrew (3rd party package manager for macOS) Python and pip. I also believe that there is a PSF Python installer, but I am not sure. Python, Be Bold captures the spirit of it should not be a shame to have the interpreter/VM installed on end-users machines. It also facilitates the work of other python devs. You installed it because of one Python program, other programs benefit from it. It also proposes enhancements to the VM to better facilitate that. One big reason the web is a popular platform today among the general public is because it offers strong sandboxing, and privelges are granted per-site, opt out by default. Another reason the web is popular is because it loads quickly. I suggest python apps feature sandboxing with a way to opt-in to special permissions. Perhaps, like the web, it could have a uniform distribution mechanism. I suggest DNS. I am suggesting Python compete with the web by implementing strong language sandboxing. I imagine a browser that can load regular web and "PyWeb." A web browser is both a kernel and a VM for the web. The kernel interfaces with the underlying OS: Linux, windows, MacOS. I do not see "PyWeb" as a kernel, but I do see it as a VM. PyWeb would merely provide a secure gatekeeper to the underlying operating system. Like the Mac App Store, PyWeb could give each app its own sandboxed file system. This would also help introduce young people to Python. Like how the DevTools console has taught many kids JavaScript. I imagine being able to run Tensorflow or Calibre inside a Python "browser." > > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4EKYH5T3MZINO2Y7MPH7MTCADEMRE45T/ Code of Conduct: http://python.org/psf/codeofconduct/
