The C API has a function PyObject_CallFunction( PyObject*, const char* fmt, ...
). It is a variadic function hence I couldn't pass a va_list to it to invoke
the call. My question is, is it technically possible to provide a companion
PyObject_VaCallFunction which takes a va_list, just like Py_VaBuildValue is to
Py_BuildValue?
Here's what I have found so far. I read the Objects/call.c about
PyObject_CallFunction there, and actually found a function
_PyObject_CallFunctionVa, which is close to what I want, although it has two
additional parameters. By taking a look at the implementation of
PyObject_CallFunction, naively I was thinking a Va version of that could have
been similarly implemented ( I'm ignoring the size_t issue ). I'd appreciate it
if someone could enlighten me on this subject. Question rephrased: is there a
technical reason why there is no PyObject_VaCallFunction in the API? If not,
would it be possible to add it in?
Motivation: I write scientific simulation code to be run on large clusters. The
data generated can be huge. Although I know parallelization using C++, I don't
know it using Python. A direct consequence is formidable data processing time
with python, which is driving me nuts. I have two options, either parallelize
python code or embed the python in C++. I'm favoring the latter, not just
because I don't know about python parallelization, but more importantly because
by embedding python in C++, I get to keep using the highly specialized C++
classes and some calculation routines, without having to duplicate essentially
the same thing in Python, which is very time-saving. I could have just used the
C API, but here is the all-time drawback of linking with Python libraries ---
it messes up the linker runtime search path. Usually a Python installation has
under its lib/ many common libraries such as libz.so, libhdf5.so ( or I
probably should have said earlier that I primarily work on Linux )
. They have different versions to, say, the libhdf5.so I'm using in the code.
The upshot is that by linking my program with both the major library in which
HDF5 links to one version, and the Python libraries in which HDF5 links to
another, the runtime search path is always mixed up, which is not safe.
So what I thought about doing is that I'm gonna create a C++ wrapper on Python,
one that doesn't expose the raw Python to the client code at all. ( For
example, there is no #include <Python.h> in any header of that library. Plus I
could use C++ OOP to automate away keeping tracking of Py_INCREF/Py_DECREF. )
In fact it's very doable. Everything is straightforward, except variadic
functions, such as PyObject_CallFunction, Py_BuildValue, and so on. Let me use
PyObject_CallFunction to illustrate the problem.
Let's say that I want to provide a wrapper to PyObject_CallFunction to the
client, like this ( I'm taking the return type to void for simplicity )
void MyCallFunction(PyObject* obj, const char* fmt, ... );
I go ahead and put this line in the header "my_python_wrapper.h". Then in the
"my_python_wrapper.c" file I would like to write the following implementation.
void MyCallFunction(PyObject* obj, const char* fmt, ... ) {
va_list args;
va_start(args,fmt);
PyObject(obj, fmt, args); // This function doesn't exist in the API ( yet? )
va_end(args);
}
This way, only the "my_python_wrapper.c" needs to link with Python Libraries.
Any user of my_python_wrapper doesn't need to, which seems nice. ( In cmake
lingo, I only need to target_link_libraries( my_python_wrapper PRIVATE
${PYTHON_LIBRARIES} ), instead of PUBLIC ) As one can see, the crux is that not
all variadic functions in the API has a companion Va-ed version. So far I only
found Py_VaBuildValue.
I've worked out MyCallFunction() in my actual code in the same manner described
above, but with Py_VaBuildValue. What I did was I send all variadic arguments
to a MyBuildValue(PyObject*, const char*, ...), the in the .c file,
MyBuildValue will generate a va_list and pass it onto Py_VaBuildValue, then I
force the outcome to be a tuple and pass it to PyObject_CallObject and it
works! Nontheless, this approach seems less straightforward to having a
PyObject_VaCallFunction, so I'm guessing it may have performance penalty.
I really appreciate it whoever takes their time to read this essay of mine! I
apologize if I failed to use the idiomatic mark-up. Any comments, questions on
this subject are welcome!
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/VHAUIILV3I7PULKZVILUVY5JEDK6OR2B/
Code of Conduct: http://python.org/psf/codeofconduct/
