Re: [GRASS-dev] RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
Hi, 2010/5/16 Glynn Clements gl...@gclements.plus.com: I've already replaced the SWIG bindings in 7.0 with the ctypesgen version. trying ctypesgen version I have problem with G_find_raster2() mapset = grass.G_find_raster2(elevation, ) print mapset WARNING: Illegal filename �pT�. Character not allowed. � Martin -- Martin Landa landa.martin gmail.com * http://gama.fsv.cvut.cz/~landa ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-dev] RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
Martin Landa wrote: I've already replaced the SWIG bindings in 7.0 with the ctypesgen version. trying ctypesgen version I have problem with G_find_raster2() mapset = grass.G_find_raster2(elevation, ) print mapset WARNING: Illegal filename �pT�. Character not allowed. Odd. Can you try it under GDB and check what arguments G_find_raster2() is getting? Also, are you calling G_gisinit() first? -- Glynn Clements gl...@gclements.plus.com ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-dev] RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
Hi, 2010/5/18 Glynn Clements gl...@gclements.plus.com: Odd. Can you try it under GDB and check what arguments G_find_raster2() is getting? Also, are you calling G_gisinit() first? sample script below --- #!/usr/bin/env python import os, sys from grass.lib import grass, raster from ctypes import * input = sys.argv[1] grass.G_gisinit(sys.argv[0]) mapset = grass.G_find_raster2(input, ) print mapset --- Running python example.py elevation I get D1/5: G_find_raster2(): name=� ��� mapset=elevation WARNING: Illegal filename � ���. Character � not allowed. � Martin -- Martin Landa landa.martin gmail.com * http://gama.fsv.cvut.cz/~landa ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-dev] RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
Hi, 2010/5/16 Glynn Clements gl...@gclements.plus.com: [...] Ideally, I'd like to do the same in 6.x in time for 6.4.0, so that we can forget that the SWIG version ever existed. The main issue there is the vdigit and nviz modules in the wxPython GUI (which don't work on Windows in any case). I haven't tried using ctypesgen for C++, though. +1 BTW, compiling SWIG in trunk ctypesgen.py -I/usr/local/src/grass_trunk/dist.i686-pc-linux-gnu/include -I/usr/local/src/grass_trunk/dist.i686-pc-linux-gnu/include -L/usr/local/src/grass_trunk/dist.i686-pc-linux-gnu/lib -L/usr/local/src/grass_trunk/dist.i686-pc-linux-gnu/lib -lgrass_datetime /usr/local/src/grass_trunk/dist.i686-pc-linux-gnu/include/grass/datetime.h /usr/local/src/grass_trunk/dist.i686-pc-linux-gnu/include/grass/P_datetime.h -o date.py make[2]: ctypesgen.py: Command not found make[2]: *** [date.py] Error 127 make[2]: Leaving directory `/usr/local/src/grass_trunk/lib/python/ctypes' make[1]: *** [default] Error 2 make[1]: Leaving directory `/usr/local/src/grass_trunk/lib/python/ctypes' make: *** [default] Error 2 Martin -- Martin Landa landa.martin gmail.com * http://gama.fsv.cvut.cz/~landa ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-dev] RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
Martin Landa wrote: Ideally, I'd like to do the same in 6.x in time for 6.4.0, so that we can forget that the SWIG version ever existed. The main issue there is the vdigit and nviz modules in the wxPython GUI (which don't work on Windows in any case). I haven't tried using ctypesgen for C++, though. +1 BTW, compiling SWIG in trunk ctypesgen.py -I/usr/local/src/grass_trunk/dist.i686-pc-linux-gnu/include -I/usr/local/src/grass_trunk/dist.i686-pc-linux-gnu/include -L/usr/local/src/grass_trunk/dist.i686-pc-linux-gnu/lib -L/usr/local/src/grass_trunk/dist.i686-pc-linux-gnu/lib -lgrass_datetime /usr/local/src/grass_trunk/dist.i686-pc-linux-gnu/include/grass/datetime.h /usr/local/src/grass_trunk/dist.i686-pc-linux-gnu/include/grass/P_datetime.h -o date.py make[2]: ctypesgen.py: Command not found make[2]: *** [date.py] Error 127 make[2]: Leaving directory `/usr/local/src/grass_trunk/lib/python/ctypes' make[1]: *** [default] Error 2 make[1]: Leaving directory `/usr/local/src/grass_trunk/lib/python/ctypes' make: *** [default] Error 2 Building the swig directory is a no-op (the SUBDIRS list is empty). The above comes from building lib/python. I'll modify it to ignore errors when building the ctypes subdirectory. ctypesgen can be obtained via SVN: svn checkout http://ctypesgen.googlecode.com/svn/trunk/ ctypesgen-read-only or you can get a tarball from: http://mirrors.kernel.org/gentoo/distfiles/ctypesgen-0_p72.tar.bz2 To build: python setup.py build install Although it's probably not as widespread as SWIG, it has the advantage that both ctypesgen itself and the wrappers which it generates are pure Python code, with no binary components. I'm also going to look at ctypeslib: http://pypi.python.org/pypi/ctypeslib/ I went with ctypesgen first as there's a supported Gentoo package for it. -- Glynn Clements gl...@gclements.plus.com ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
Glynn Clements wrote:
I also found ctypesgen:
http://code.google.com/p/ctypesgen/
I finally got around to looking at this, and it took less than half an
hour to generate working bindings. Obviously, it will need more
testing, but unless any insurmountable problems show up, I intend to
using this in place of SWIG.
Usage is similar to the example posted earlier, except:
1. The library doesn't need to be loaded manually; this is handled by
the wrapper.
2. The GIS_H_VERSION and G_gisinit() macros are wrapped, so you can
just use e.g. grass.G_gisinit(sys.argv[0]) rather than needing to
extract the version for G__gisinit().
3. The CELL/FCELL/DCELL types are wrapped, so those can be
used instead of c_int, c_float, c_double.
4. The {CELL,FCELL,DCELL}_TYPE constants are wrapped, so those can be
used instead of 0, 1, 2.
The example script is attached.
The wrappers should be largely compatible with those generated by
SWIG, but without the headaches. The automatic type conversions are no
longer there, but converting manually with ctypes is easy enough.
Also, I've discovered that NumPy arrays have conversion methods, e.g.
p = buf.ctypes.data_as(POINTER(raster.CELL))
One caveat: this *doesn't* appear to increment the reference count, so
if you do e.g.:
buf = buf.ctypes.data_as(POINTER(raster.CELL))
the array will be gc'd resulting in buf pointing at garbage.
I've already replaced the SWIG bindings in 7.0 with the ctypesgen
version.
Ideally, I'd like to do the same in 6.x in time for 6.4.0, so that we
can forget that the SWIG version ever existed. The main issue there is
the vdigit and nviz modules in the wxPython GUI (which don't work on
Windows in any case). I haven't tried using ctypesgen for C++, though.
--
Glynn Clements gl...@gclements.plus.com
#!/usr/bin/env python
import os, sys
from grass.lib import grass, raster
from ctypes import *
input = sys.argv[1]
grass.G_gisinit(sys.argv[0])
# determine the inputmap type (CELL/FCELL/DCELL) */
data_type = raster.Rast_map_type(input, )
if data_type == raster.CELL_TYPE:
ptype = POINTER(raster.CELL)
elif data_type == raster.FCELL_TYPE:
ptype = POINTER(raster.FCELL)
elif data_type == raster.DCELL_TYPE:
ptype = POINTER(raster.DCELL)
infd = raster.Rast_open_old(input, )
inrast = raster.Rast_allocate_buf(data_type)
inrast = cast(c_void_p(inrast), ptype)
rows = raster.Rast_window_rows()
cols = raster.Rast_window_cols()
for rown in xrange(rows):
raster.Rast_get_row(infd, inrast, rown, data_type)
print rown, inrast[0:cols]
raster.Rast_close(infd)
grass.G_free(inrast)
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
From: Glynn Clements I also found ctypesgen: http://code.google.com/p/ctypesgen/ I finally got around to looking at this, and it took less than half an hour to generate working bindings. Obviously, it will need more testing, but unless any insurmountable problems show up, I intend to using this in place of SWIG. Glad you found something that you're comfortable with, and that the effort required appears to be so modest. Also, I've discovered that NumPy arrays have conversion methods, e.g. p = buf.ctypes.data_as(POINTER(raster.CELL)) One caveat: this *doesn't* appear to increment the reference count, so if you do e.g.: buf = buf.ctypes.data_as(POINTER(raster.CELL)) the array will be gc'd resulting in buf pointing at garbage. This sort of issue is something that catches people in Python (especially when using ctypes) until they learn what needs to be preserved - its sometimes easy to lose sight of whether you've got deep copies or shallow copies of the original. I haven't tried using ctypesgen for C++, though. The material on the above website makes no mention of C++... Regards, Andrew. - These thoughts are mine alone! - Andrew MacIntyre Operations Branch tel: +61 2 6219 5356 Communications Infrastructure Division fax: +61 2 6253 3277 Australian Communications Media Authority email: andrew.macint...@acma.gov.auhttp://www.acma.gov.au/ If you have received this email in error, please notify the sender immediately and erase all copies of the email and any attachments to it. The information contained in this email and any attachments may be private, confidential and legally privileged or the subject of copyright. If you are not the addressee it may be illegal to review, disclose, use, forward, or distribute this email and/or its contents. Unless otherwise specified, the information in the email and any attachments is intended as a guide only and should not be relied upon as legal or technical advice or regarded as a substitute for legal or technical advice in individual cases. Opinions contained in this email or any of its attachments do not necessarily reflect the opinions of ACMA. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
Andrew MacIntyre wrote: From the SWIG examples I inferred the following type references for the pointers in question: CELL* = SWIGTYPE_p_CELL FCELL* = SWIGTYPE_p_FCELL DCELL* = SWIGTYPE_p_DCELL provided that CELL*/FCELL*/DCELL* are typedef'ed. So I expected that in pyobj_to_ptr() you could do something like But I still don't see what happens when a function returns a CELL* which is subsequently passed to a function accepting a void*, or vice-versa. This is common enough that if SWIG can't handle it, we need to use something other than SWIG. But, to be honest, even if SWIG can handle it, I'd prefer something which generates pure Python wrappers. I think that I'm not going to put any more effort into the SWIG wrapper unless usable documentation materialises. In the meantime, I'm going to look for alternatives. If you decide to investigate using ctypes, the following links might be of interest: - http://starship.python.net/crew/theller/ctypes/old/codegen.html AFAICT, this has been superseded by ctypeslib: http://pypi.python.org/pypi/ctypeslib/ - http://www.language-binding.net/pygccxml/pygccxml.html I wasn't aware of this one. I also found ctypesgen: http://code.google.com/p/ctypesgen/ -- Glynn Clements gl...@gclements.plus.com ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
-Original Message- From: Glynn Clements [mailto:gl...@gclements.plus.com] Sent: Wednesday, 12 May 2010 1:43 AM To: Andrew MacIntyre Cc: GRASS user list; grass-...@lists.osgeo.org Subject: RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED] Andrew MacIntyre wrote: From the SWIG examples I inferred the following type references for the pointers in question: CELL* = SWIGTYPE_p_CELL FCELL* = SWIGTYPE_p_FCELL DCELL* = SWIGTYPE_p_DCELL provided that CELL*/FCELL*/DCELL* are typedef'ed. So I expected that in pyobj_to_ptr() you could do something like But I still don't see what happens when a function returns a CELL* which is subsequently passed to a function accepting a void*, or vice-versa. This is common enough that if SWIG can't handle it, we need to use something other than SWIG. That seems to be what the SWIG Pointer Library function ptrcast() is for, according to http://www.swig.org/Doc1.1/HTML/Typemaps.html, so something like (in Python): import GrassMod pCELL = GrassMod.Returns_CELLptr() pvoid = GrassMod.castptr(pCELL, void *) result = GrassMod.Takes_voidptr(pvoid) (substitute the correct name of the GRASS SWIG module for GrassMod) If you decide to investigate using ctypes, the following links might be of interest: - http://starship.python.net/crew/theller/ctypes/old/codegen.html AFAICT, this has been superseded by ctypeslib: http://pypi.python.org/pypi/ctypeslib/ Thanks, the old page was something I had bookmarked. - http://www.language-binding.net/pygccxml/pygccxml.html I wasn't aware of this one. I also found ctypesgen: http://code.google.com/p/ctypesgen/ And that one's new to me, for which thanks. Cheers, Andrew. - These thoughts are mine alone! - Andrew MacIntyre Operations Branch tel: +61 2 6219 5356 Communications Infrastructure Division fax: +61 2 6253 3277 Australian Communications Media Authority email: andrew.macint...@acma.gov.auhttp://www.acma.gov.au/ If you have received this email in error, please notify the sender immediately and erase all copies of the email and any attachments to it. The information contained in this email and any attachments may be private, confidential and legally privileged or the subject of copyright. If you are not the addressee it may be illegal to review, disclose, use, forward, or distribute this email and/or its contents. Unless otherwise specified, the information in the email and any attachments is intended as a guide only and should not be relied upon as legal or technical advice or regarded as a substitute for legal or technical advice in individual cases. Opinions contained in this email or any of its attachments do not necessarily reflect the opinions of ACMA. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
Bah! typo... But I still don't see what happens when a function returns a CELL* which is subsequently passed to a function accepting a void*, or vice-versa. This is common enough that if SWIG can't handle it, we need to use something other than SWIG. That seems to be what the SWIG Pointer Library function ptrcast() is for, according to http://www.swig.org/Doc1.1/HTML/Typemaps.html, so something like (in Python): import GrassMod pCELL = GrassMod.Returns_CELLptr() pvoid = GrassMod.castptr(pCELL, void *) above line should read: pvoid = GrassMod.ptrcast(pCELL, void *) result = GrassMod.Takes_voidptr(pvoid) (substitute the correct name of the GRASS SWIG module for GrassMod) Cheers, Andrew. - These thoughts are mine alone! - Andrew MacIntyre Operations Branch tel: +61 2 6219 5356 Communications Infrastructure Division fax: +61 2 6253 3277 Australian Communications Media Authority email: andrew.macint...@acma.gov.auhttp://www.acma.gov.au/ If you have received this email in error, please notify the sender immediately and erase all copies of the email and any attachments to it. The information contained in this email and any attachments may be private, confidential and legally privileged or the subject of copyright. If you are not the addressee it may be illegal to review, disclose, use, forward, or distribute this email and/or its contents. Unless otherwise specified, the information in the email and any attachments is intended as a guide only and should not be relied upon as legal or technical advice or regarded as a substitute for legal or technical advice in individual cases. Opinions contained in this email or any of its attachments do not necessarily reflect the opinions of ACMA. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
That seems to be what the SWIG Pointer Library function ptrcast() is for, according to http://www.swig.org/Doc1.1/HTML/Typemaps.html, so something like (in Python): import GrassMod pCELL = GrassMod.Returns_CELLptr() pvoid = GrassMod.castptr(pCELL, void *) above line should read: pvoid = GrassMod.ptrcast(pCELL, void *) result = GrassMod.Takes_voidptr(pvoid) (substitute the correct name of the GRASS SWIG module for GrassMod) Just noticed that the docco I referenced was for SWIG 1.1, so the above is probably moot. However, the 2nd para of s5.3.2 of the SWIG 1.3 documentation at http://www.swig.org/Doc1.3/SWIG.html#SWIG_nn17 states Like C, void * matches any kind of pointer. which suggests to me that the casts from CELL*/FCELL*/DCELL to void* would be implicitly performed anyway - be worth testing. Cheers, Andrew. - These thoughts are mine alone! - Andrew MacIntyre Operations Branch tel: +61 2 6219 5356 Communications Infrastructure Division fax: +61 2 6253 3277 Australian Communications Media Authority email: andrew.macint...@acma.gov.auhttp://www.acma.gov.au/ If you have received this email in error, please notify the sender immediately and erase all copies of the email and any attachments to it. The information contained in this email and any attachments may be private, confidential and legally privileged or the subject of copyright. If you are not the addressee it may be illegal to review, disclose, use, forward, or distribute this email and/or its contents. Unless otherwise specified, the information in the email and any attachments is intended as a guide only and should not be relied upon as legal or technical advice or regarded as a substitute for legal or technical advice in individual cases. Opinions contained in this email or any of its attachments do not necessarily reflect the opinions of ACMA. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
Andrew MacIntyre wrote: In the case where SWIG is using its own objects to contain returned pointers, then I expect pyobj_to_ptr() needs to be changed to use SWIG_ConvertPtr() rather than PyCObject_AsVoidPtr() to retrieve the actual pointer, or the output typemaps for returned pointers need to be changed to use Python's CObjects rather than native SWIG wrapper objects. So, what do you suggest? Remove the PyCObject_Check() call, explicitly check for string, read buffer, write buffer, and sequence, then call SWIG_ConvertPtr() on anything which is left over? That's what I would try. Except ... SWIG_ConvertPtr() seems to require that you have access to the type descriptor. The documentation seems to suggest (although, as usual, it's unclear) that you need to know the actual type. I don't think you need to get rid of the typemap - I suspect you might have to expand it to distinguish between the different types of pointers though (see below)... I get the impression from the above linked documentation that, for example, there should be an explicit typemap reference for CELL pointers (as returned by Rast_allocate_c_buf()) and that the typemap for CELL pointers as an input argument should use SWIG_ConvertPtr(), which will set an exception if the inbound object is not a suitable input GRASS uses void * to refer to any of CELL*/FCELL*/DCELL* quite extensively. I think that I'm not going to put any more effort into the SWIG wrapper unless usable documentation materialises. In the meantime, I'm going to look for alternatives. -- Glynn Clements gl...@gclements.plus.com ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
From: Glynn Clements
Andrew MacIntyre wrote:
In the case where SWIG is using its own objects to contain
returned
pointers, then I expect pyobj_to_ptr() needs to be changed to
use
SWIG_ConvertPtr() rather than PyCObject_AsVoidPtr() to retrieve
the
actual pointer, or the output typemaps for returned pointers
need to be
changed to use Python's CObjects rather than native SWIG wrapper
objects.
So, what do you suggest? Remove the PyCObject_Check() call,
explicitly
check for string, read buffer, write buffer, and sequence, then
call
SWIG_ConvertPtr() on anything which is left over?
That's what I would try.
Except ... SWIG_ConvertPtr() seems to require that you have access to
the type descriptor. The documentation seems to suggest (although, as
usual, it's unclear) that you need to know the actual type.
From the SWIG examples I inferred the following type references for
the pointers in question:
CELL* = SWIGTYPE_p_CELL
FCELL* = SWIGTYPE_p_FCELL
DCELL* = SWIGTYPE_p_DCELL
provided that CELL*/FCELL*/DCELL* are typedef'ed.
So I expected that in pyobj_to_ptr() you could do something like
int rc;
void *ptr;
...
switch (type_parm) {
case CELL_PTR:
rc = SWIG_ConvertPtr(PyObj, ptr,
SWIGTYPE_p_CELL, SWIG_POINTER_EXCEPTION);
break;
case FCELL_PTR:
rc = SWIG_ConvertPtr(PyObj, ptr,
SWIGTYPE_p_FCELL, SWIG_POINTER_EXCEPTION);
break;
case DCELL_PTR:
rc = SWIG_ConvertPtr(PyObj, ptr,
SWIGTYPE_p_DCELL, SWIG_POINTER_EXCEPTION);
break;
}
if (rc == -1)
return NULL;
...
type_parm is the type code parameter passed in to pyobj_to_ptr() - I
don't
have a link to the swig file to check the actual parameter name.
I don't think you need to get rid of the typemap - I suspect you
might
have to expand it to distinguish between the different types of
pointers
though (see below)...
I get the impression from the above linked documentation that, for
example, there should be an explicit typemap reference for CELL
pointers
(as returned by Rast_allocate_c_buf()) and that the typemap for CELL
pointers as an input argument should use SWIG_ConvertPtr(), which
will
set an exception if the inbound object is not a suitable input
GRASS uses void * to refer to any of CELL*/FCELL*/DCELL* quite
extensively.
I think that I'm not going to put any more effort into the SWIG
wrapper unless usable documentation materialises. In the meantime, I'm
going to look for alternatives.
If you decide to investigate using ctypes, the following links might be
of interest:
- http://starship.python.net/crew/theller/ctypes/old/codegen.html
- http://www.language-binding.net/pygccxml/pygccxml.html
The latter seems more focussed on C++. I haven't used either so can't
give
any guidance.
Cheers,
Andrew.
- These thoughts are mine alone! -
Andrew MacIntyre Operations Branch
tel: +61 2 6219 5356 Communications Infrastructure Division
fax: +61 2 6253 3277 Australian Communications Media Authority
email: andrew.macint...@acma.gov.auhttp://www.acma.gov.au/
If you have received this email in error, please notify the sender immediately
and erase all copies of the email and any attachments to it. The information
contained in this email and any attachments may be private, confidential and
legally privileged or the subject of copyright. If you are not the addressee it
may be illegal to review, disclose, use, forward, or distribute this email
and/or its contents.
Unless otherwise specified, the information in the email and any attachments is
intended as a guide only and should not be relied upon as legal or technical
advice or regarded as a substitute for legal or technical advice in individual
cases. Opinions contained in this email or any of its attachments do not
necessarily reflect the opinions of ACMA.
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
From: Glynn Clements Andrew MacIntyre wrote: In the case where SWIG is using its own objects to contain returned pointers, then I expect pyobj_to_ptr() needs to be changed to use SWIG_ConvertPtr() rather than PyCObject_AsVoidPtr() to retrieve the actual pointer, or the output typemaps for returned pointers need to be changed to use Python's CObjects rather than native SWIG wrapper objects. So, what do you suggest? Remove the PyCObject_Check() call, explicitly check for string, read buffer, write buffer, and sequence, then call SWIG_ConvertPtr() on anything which is left over? That's what I would try. Cheers, Andrew. - These thoughts are mine alone! - Andrew MacIntyre Operations Branch tel: +61 2 6219 5356 Communications Infrastructure Division fax: +61 2 6253 3277 Australian Communications Media Authority email: andrew.macint...@acma.gov.auhttp://www.acma.gov.au/ If you have received this email in error, please notify the sender immediately and erase all copies of the email and any attachments to it. The information contained in this email and any attachments may be private, confidential and legally privileged or the subject of copyright. If you are not the addressee it may be illegal to review, disclose, use, forward, or distribute this email and/or its contents. Unless otherwise specified, the information in the email and any attachments is intended as a guide only and should not be relied upon as legal or technical advice or regarded as a substitute for legal or technical advice in individual cases. Opinions contained in this email or any of its attachments do not necessarily reflect the opinions of ACMA. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
From: Glynn Clements [mailto:gl...@gclements.plus.com] Sent: Saturday, 24 April 2010 2:34 AM To: Andrew MacIntyre Cc: GRASS user list Subject: RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED] Andrew MacIntyre wrote: That's what I suspect, namely that the typemap for pointers is getting in the way. Removing the typemap would mean that you couldn't pass a Python string, array, buffer, sequence, etc directly to a GRASS function, but would have to explicitly allocate and populate a block of memory. I don't think you need to get rid of the typemap - I suspect you might have to expand it to distinguish between the different types of pointers though (see below)... Yes, but I know how to get rid of it; I don't know how to expand it. In the SWIG wrapper? I see a note about SWIG_ConvertPtr() in http://www.swig.org/Doc1.3/Python.html#Python_nn64 ? That might help, although I still don't know how to test whether a given PyObject * is a SWIG-wrapped pointer. I get the impression from the above linked documentation that, for example, there should be an explicit typemap reference for CELL pointers (as returned by Rast_allocate_c_buf()) and that the typemap for CELL pointers as an input argument should use SWIG_ConvertPtr(), which will set an exception if the inbound object is not a suitable input (if the exception flag is used; returning NULL will then propagate the exception). As far as SWIGged pointers go, this seems to be the closest you can get to Python's *Check* APIs. But does that convert arrays, buffers, etc? FWIW, the typemaps are in swig/include/python/my_typemaps.i. I finally had the chance to look at this last night. I think the issue is that pyobj_to_ptr() in my_typemaps.i is expecting the SWIGged pointer as a Python CObject (though it does seem odd that a pointer object, previously returned from a SWIGged call, being passed in is at least accepted without a ValueError - which suggests that these objects are at least look like CObjects to Python). I would suggest that this isn't correct unless you can determine that SWIG is definitely generating CObjects to contain returned pointers (I couldn't find anything to suggest this). In the case where SWIG is using its own objects to contain returned pointers, then I expect pyobj_to_ptr() needs to be changed to use SWIG_ConvertPtr() rather than PyCObject_AsVoidPtr() to retrieve the actual pointer, or the output typemaps for returned pointers need to be changed to use Python's CObjects rather than native SWIG wrapper objects. Several other approaches occur to me: - use simple SWIG typemaps for the various pointer types and create helper functions which translate from Python objects to GRASS data structures and vice versa. It might be possible to use ctypes objects too, though that could be quite tricky as I couldn't find any direct API support. - go to the trouble of actually creating full Python object wrappers for the various data structures and use them exclusively instead of using thinly wrapped pointers (this has the potential advantage that memory management becomes a lot more automatic for the user). - ditch SWIG completely for Python and use ctypes exclusively. The ctypes approach is the one I myself would take, but I acknowledge there are downsides - the potential for sub-optimal performance in certain cases and the cost of initially generating the wrapper definitions being the most obvious. I haven't tried the gccxml based code generator that ctypes author Thomas Heller created, which might make the job a lot easier if it works on the GRASS headers. I suspect that the output typemaps change might be easiest to implement. Cheers, Andrew. - These thoughts are mine alone! - Andrew MacIntyre Operations Branch tel: +61 2 6219 5356 Communications Infrastructure Division fax: +61 2 6253 3277 Australian Communications Media Authority email: andrew.macint...@acma.gov.auhttp://www.acma.gov.au/ If you have received this email in error, please notify the sender immediately and erase all copies of the email and any attachments to it. The information contained in this email and any attachments may be private, confidential and legally privileged or the subject of copyright. If you are not the addressee it may be illegal to review, disclose, use, forward, or distribute this email and/or its contents. Unless otherwise specified, the information in the email and any attachments is intended as a guide only and should not be relied upon as legal or technical advice or regarded as a substitute for legal or technical advice in individual cases. Opinions contained in this email or any of its attachments do not necessarily reflect the opinions of ACMA. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
Andrew MacIntyre wrote: I think the issue is that pyobj_to_ptr() in my_typemaps.i is expecting the SWIGged pointer as a Python CObject (though it does seem odd that a pointer object, previously returned from a SWIGged call, being passed in is at least accepted without a ValueError - which suggests that these objects are at least look like CObjects to Python). I would suggest that this isn't correct unless you can determine that SWIG is definitely generating CObjects to contain returned pointers (I couldn't find anything to suggest this). The support for CObjects isn't critical; it was really just a case of what Python types can reasonably be converted to a pointer?. In the case where SWIG is using its own objects to contain returned pointers, then I expect pyobj_to_ptr() needs to be changed to use SWIG_ConvertPtr() rather than PyCObject_AsVoidPtr() to retrieve the actual pointer, or the output typemaps for returned pointers need to be changed to use Python's CObjects rather than native SWIG wrapper objects. So, what do you suggest? Remove the PyCObject_Check() call, explicitly check for string, read buffer, write buffer, and sequence, then call SWIG_ConvertPtr() on anything which is left over? Several other approaches occur to me: - use simple SWIG typemaps for the various pointer types and create helper functions which translate from Python objects to GRASS data structures and vice versa. It might be possible to use ctypes objects too, though that could be quite tricky as I couldn't find any direct API support. IOW, strings, lists, arrays, etc would need to be converted explicitly? That's inconvenient, but I personally consider the issue of what's possible to be more important than what's simple. If the typemaps make certain things impossible, they should go, even if that makes other things less convenient. - go to the trouble of actually creating full Python object wrappers for the various data structures and use them exclusively instead of using thinly wrapped pointers (this has the potential advantage that memory management becomes a lot more automatic for the user). - ditch SWIG completely for Python and use ctypes exclusively. The ctypes approach is the one I myself would take, but I acknowledge there are downsides - the potential for sub-optimal performance in certain cases and the cost of initially generating the wrapper definitions being the most obvious. The problem isn't initial generation, but maintenance. If someone changes a structure, enum, prototype, etc in the headers, it's unlikely that they'll remember to change any ctypes wrapper to match. I haven't tried the gccxml based code generator that ctypes author Thomas Heller created, which might make the job a lot easier if it works on the GRASS headers. If ctypes wrappers can be generated automatically from the headers, that solves the problem. Manually generated ctypes wrappers have to be kept in sync, while SWIG wrappers tend to be hard to use (mostly due to inadequate documentation). -- Glynn Clements gl...@gclements.plus.com ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
Andrew MacIntyre wrote: That's what I suspect, namely that the typemap for pointers is getting in the way. Removing the typemap would mean that you couldn't pass a Python string, array, buffer, sequence, etc directly to a GRASS function, but would have to explicitly allocate and populate a block of memory. I don't think you need to get rid of the typemap - I suspect you might have to expand it to distinguish between the different types of pointers though (see below)... Yes, but I know how to get rid of it; I don't know how to expand it. In the SWIG wrapper? I see a note about SWIG_ConvertPtr() in http://www.swig.org/Doc1.3/Python.html#Python_nn64 ? That might help, although I still don't know how to test whether a given PyObject * is a SWIG-wrapped pointer. I get the impression from the above linked documentation that, for example, there should be an explicit typemap reference for CELL pointers (as returned by Rast_allocate_c_buf()) and that the typemap for CELL pointers as an input argument should use SWIG_ConvertPtr(), which will set an exception if the inbound object is not a suitable input (if the exception flag is used; returning NULL will then propagate the exception). As far as SWIGged pointers go, this seems to be the closest you can get to Python's *Check* APIs. But does that convert arrays, buffers, etc? FWIW, the typemaps are in swig/include/python/my_typemaps.i. -- Glynn Clements gl...@gclements.plus.com ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
From: Glynn Clements Andrew MacIntyre wrote: That was what I tried first, but it results in Rast_get_c_row() segfaulting. Then it seems to me that there's some problem with the SWIG wrapper for Rast_get_c_row(), if it won't accept a valid SWIG pointer object returned from another wrapped function. That's what I suspect, namely that the typemap for pointers is getting in the way. Removing the typemap would mean that you couldn't pass a Python string, array, buffer, sequence, etc directly to a GRASS function, but would have to explicitly allocate and populate a block of memory. I don't think you need to get rid of the typemap - I suspect you might have to expand it to distinguish between the different types of pointers though (see below)... There is a typemap for pointers, which converts a CObject, string, buffer or sequence. I have no idea how to test for or convert a SWIG-wrapped pointer. In the SWIG wrapper? I see a note about SWIG_ConvertPtr() in http://www.swig.org/Doc1.3/Python.html#Python_nn64 ? That might help, although I still don't know how to test whether a given PyObject * is a SWIG-wrapped pointer. I get the impression from the above linked documentation that, for example, there should be an explicit typemap reference for CELL pointers (as returned by Rast_allocate_c_buf()) and that the typemap for CELL pointers as an input argument should use SWIG_ConvertPtr(), which will set an exception if the inbound object is not a suitable input (if the exception flag is used; returning NULL will then propagate the exception). As far as SWIGged pointers go, this seems to be the closest you can get to Python's *Check* APIs. Cheers, Andrew. - These thoughts are mine alone! - Andrew MacIntyre Operations Branch tel: +61 2 6219 5356 Communications Infrastructure Division fax: +61 2 6253 3277 Australian Communications Media Authority email: andrew.macint...@acma.gov.auhttp://www.acma.gov.au/ If you have received this email in error, please notify the sender immediately and erase all copies of the email and any attachments to it. The information contained in this email and any attachments may be private, confidential and legally privileged or the subject of copyright. If you are not the addressee it may be illegal to review, disclose, use, forward, or distribute this email and/or its contents. Unless otherwise specified, the information in the email and any attachments is intended as a guide only and should not be relied upon as legal or technical advice or regarded as a substitute for legal or technical advice in individual cases. Opinions contained in this email or any of its attachments do not necessarily reflect the opinions of ACMA. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
Andrew MacIntyre wrote: After some googling, it seems that the pointer address should be obtainable by using int() on the pointer object returned by the SWIG wrapper (i.e. int(p) in the above example), with the resulting value usable in ctypes' from_address() method. I might be wrong, but I think that e.g. ctypes.c_void_p(int(p)) might also work to create a ctypes void pointer instance. That doesn't seem to work either: import grass.lib.grass as gis import grass.lib.raster as rast import ctypes gis.G_gisinit() p = rast.Rast_allocate_c_buf() a = ctypes.cast(int(p), ctypes.POINTER(ctypes.c_int)) infd = rast.Rast_open_old(elevation.dem, ) rast.Rast_get_c_row(infd, a, 10) I think the above line should read: rast. Rast_get_c_row(infd, p, 10) That was what I tried first, but it results in Rast_get_c_row() segfaulting. There is a typemap for pointers, which converts a CObject, string, buffer or sequence. I have no idea how to test for or convert a SWIG-wrapped pointer. The fact that passing a doesn't segfault appears to indicate that it was successfully converted to a pointer. The fact that accessing a[0] afterwards leads me to suspect that it may be getting converted to a pointer to the container rather than to the array proper. -- Glynn Clements gl...@gclements.plus.com ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
From: Glynn Clements Andrew MacIntyre wrote: After some googling, it seems that the pointer address should be obtainable by using int() on the pointer object returned by the SWIG wrapper (i.e. int(p) in the above example), with the resulting value usable in ctypes' from_address() method. I might be wrong, but I think that e.g. ctypes.c_void_p(int(p)) might also work to create a ctypes void pointer instance. That doesn't seem to work either: import grass.lib.grass as gis import grass.lib.raster as rast import ctypes gis.G_gisinit() p = rast.Rast_allocate_c_buf() a = ctypes.cast(int(p), ctypes.POINTER(ctypes.c_int)) infd = rast.Rast_open_old(elevation.dem, ) rast.Rast_get_c_row(infd, a, 10) I think the above line should read: rast. Rast_get_c_row(infd, p, 10) That was what I tried first, but it results in Rast_get_c_row() segfaulting. Then it seems to me that there's some problem with the SWIG wrapper for Rast_get_c_row(), if it won't accept a valid SWIG pointer object returned from another wrapped function. There is a typemap for pointers, which converts a CObject, string, buffer or sequence. I have no idea how to test for or convert a SWIG-wrapped pointer. In the SWIG wrapper? I see a note about SWIG_ConvertPtr() in http://www.swig.org/Doc1.3/Python.html#Python_nn64 ? The fact that passing a doesn't segfault appears to indicate that it was successfully converted to a pointer. The fact that accessing a[0] afterwards leads me to suspect that it may be getting converted to a pointer to the container rather than to the array proper. Well, a is a ctypes object, so I wouldn't have expected a useful coercion. Sadly, I doubt I can contribute much more to this matter :( Andrew. - These thoughts are mine alone! - Andrew MacIntyre Operations Branch tel: +61 2 6219 5356 Communications Infrastructure Division fax: +61 2 6253 3277 Australian Communications Media Authority email: andrew.macint...@acma.gov.auhttp://www.acma.gov.au/ If you have received this email in error, please notify the sender immediately and erase all copies of the email and any attachments to it. The information contained in this email and any attachments may be private, confidential and legally privileged or the subject of copyright. If you are not the addressee it may be illegal to review, disclose, use, forward, or distribute this email and/or its contents. Unless otherwise specified, the information in the email and any attachments is intended as a guide only and should not be relied upon as legal or technical advice or regarded as a substitute for legal or technical advice in individual cases. Opinions contained in this email or any of its attachments do not necessarily reflect the opinions of ACMA. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
Andrew MacIntyre wrote: After some googling, it seems that the pointer address should be obtainable by using int() on the pointer object returned by the SWIG wrapper (i.e. int(p) in the above example), with the resulting value usable in ctypes' from_address() method. I might be wrong, but I think that e.g. ctypes.c_void_p(int(p)) might also work to create a ctypes void pointer instance. That doesn't seem to work either: import grass.lib.grass as gis import grass.lib.raster as rast import ctypes gis.G_gisinit() p = rast.Rast_allocate_c_buf() a = ctypes.cast(int(p), ctypes.POINTER(ctypes.c_int)) infd = rast.Rast_open_old(elevation.dem, ) rast.Rast_get_c_row(infd, a, 10) a[0] Segmentation fault (core dumped) 14:52:56 It's possible that the typemap is getting in the way here, but I have no idea what should be done instead. -- Glynn Clements gl...@gclements.plus.com ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
[Oops - original inadvertently sent only to Glynn, again :-(] From: Glynn Clements Andrew MacIntyre wrote: After some googling, it seems that the pointer address should be obtainable by using int() on the pointer object returned by the SWIG wrapper (i.e. int(p) in the above example), with the resulting value usable in ctypes' from_address() method. I might be wrong, but I think that e.g. ctypes.c_void_p(int(p)) might also work to create a ctypes void pointer instance. That doesn't seem to work either: import grass.lib.grass as gis import grass.lib.raster as rast import ctypes gis.G_gisinit() p = rast.Rast_allocate_c_buf() a = ctypes.cast(int(p), ctypes.POINTER(ctypes.c_int)) infd = rast.Rast_open_old(elevation.dem, ) rast.Rast_get_c_row(infd, a, 10) I think the above line should read: rast. Rast_get_c_row(infd, p, 10) a[0] Then this might work. - These thoughts are mine alone! - Andrew MacIntyre Operations Branch tel: +61 2 6219 5356 Communications Infrastructure Division fax: +61 2 6253 3277 Australian Communications Media Authority email: andrew.macint...@acma.gov.au http://www.acma.gov.au/ If you have received this email in error, please notify the sender immediately and erase all copies of the email and any attachments to it. The information contained in this email and any attachments may be private, confidential and legally privileged or the subject of copyright. If you are not the addressee it may be illegal to review, disclose, use, forward, or distribute this email and/or its contents. Unless otherwise specified, the information in the email and any attachments is intended as a guide only and should not be relied upon as legal or technical advice or regarded as a substitute for legal or technical advice in individual cases. Opinions contained in this email or any of its attachments do not necessarily reflect the opinions of ACMA. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
Andrew MacIntyre wrote:
Glynn Clements wrote:
{...}
Nor can I figure out how to make use of any pointers returned from
functions. Typemaps exist to allow strings, arrays, etc to be
converted to pointers when passing them into functions, but it isn't
possible to do the reverse (converting to a pointer discards the size
and type; converting from a pointer would require some way to obtain
this information). The functions generated by cpointer.i and carrays.i
don't seem to work as expected (I get a segfault).
Doesn't the ctypes cast() function work for you in making use of
returned pointers?
In case it wasn't clear, I was talking about SWIG-wrapped pointers,
e.g.:
import grass.lib.raster as rast
import grass.lib.grass as gis
import ctypes
gis.G_gisinit()
p = rast.Rast_allocate_c_buf()
p
Swig Object of type 'int *' at 0x912e5b8
q = ctypes.cast(p, ctypes.c_void_p)
Traceback (most recent call last):
File stdin, line 1, in module
File /usr/lib/python2.6/ctypes/__init__.py, line 489, in cast
return _cast(obj, obj, typ)
ctypes.ArgumentError: argument 1: type 'exceptions.TypeError': wrong type
--
Glynn Clements gl...@gclements.plus.com
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
[Oops - original inadvertently sent only to Glynn]
From: Glynn Clements [mailto:gl...@gclements.plus.com]
Andrew MacIntyre wrote:
Glynn Clements wrote:
{...}
Nor can I figure out how to make use of any pointers returned from
functions. Typemaps exist to allow strings, arrays, etc to be
converted to pointers when passing them into functions, but it
isn't possible to do the reverse (converting to a pointer discards
the size and type; converting from a pointer would require some
way to obtain this information). The functions generated by
cpointer.i and carrays.i don't seem to work as expected (I get a
segfault).
Doesn't the ctypes cast() function work for you in making use of
returned pointers?
In case it wasn't clear, I was talking about SWIG-wrapped pointers,
e.g.:
import grass.lib.raster as rast
import grass.lib.grass as gis
import ctypes
gis.G_gisinit()
p = rast.Rast_allocate_c_buf()
p
Swig Object of type 'int *' at 0x912e5b8
q = ctypes.cast(p, ctypes.c_void_p)
Traceback (most recent call last):
File stdin, line 1, in module
File /usr/lib/python2.6/ctypes/__init__.py, line 489, in cast
return _cast(obj, obj, typ)
ctypes.ArgumentError: argument 1: type 'exceptions.TypeError': wrong
type
Ahh, sorry for not paying close enough attention.
I've never used SWIG and don't profess to understand exactly what its
returning in the code for _wrap_Rast_allocate_c_buf in raster_wrap.c
(~line 12530).
If the pointer object being made available in Python can be made to
publish the pointer address, then the from_address() method of the
target ctypes type (such as a structure) can be used.
After some googling, it seems that the pointer address should be
obtainable by using int() on the pointer object returned by the SWIG
wrapper (i.e. int(p) in the above example), with the resulting value
usable in ctypes' from_address() method. I might be wrong, but I think
that e.g. ctypes.c_void_p(int(p)) might also work to create a ctypes
void pointer instance.
- These thoughts are mine alone! -
Andrew MacIntyre Operations Branch
tel: +61 2 6219 5356 Communications Infrastructure Division
fax: +61 2 6253 3277 Australian Communications Media Authority
email: andrew.macint...@acma.gov.auhttp://www.acma.gov.au/
If you have received this email in error, please notify the sender immediately
and erase all copies of the email and any attachments to it. The information
contained in this email and any attachments may be private, confidential and
legally privileged or the subject of copyright. If you are not the addressee it
may be illegal to review, disclose, use, forward, or distribute this email
and/or its contents.
Unless otherwise specified, the information in the email and any attachments is
intended as a guide only and should not be relied upon as legal or technical
advice or regarded as a substitute for legal or technical advice in individual
cases. Opinions contained in this email or any of its attachments do not
necessarily reflect the opinions of ACMA.
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
RE: [GRASS-user] Question with Python-SWIG example [SEC=UNCLASSIFIED]
Glynn Clements wrote:
{...}
Nor can I figure out how to make use of any pointers returned from
functions. Typemaps exist to allow strings, arrays, etc to be
converted to pointers when passing them into functions, but it isn't
possible to do the reverse (converting to a pointer discards the size
and type; converting from a pointer would require some way to obtain
this information). The functions generated by cpointer.i and carrays.i
don't seem to work as expected (I get a segfault).
Doesn't the ctypes cast() function work for you in making use of
returned pointers?
- These thoughts are mine alone! -
Andrew MacIntyre Operations Branch
tel: +61 2 6219 5356 Communications Infrastructure Division
fax: +61 2 6253 3277 Australian Communications Media Authority
email: andrew.macint...@acma.gov.auhttp://www.acma.gov.au/
If you have received this email in error, please notify the sender immediately
and erase all copies of the email and any attachments to it. The information
contained in this email and any attachments may be private, confidential and
legally privileged or the subject of copyright. If you are not the addressee it
may be illegal to review, disclose, use, forward, or distribute this email
and/or its contents.
Unless otherwise specified, the information in the email and any attachments is
intended as a guide only and should not be relied upon as legal or technical
advice or regarded as a substitute for legal or technical advice in individual
cases. Opinions contained in this email or any of its attachments do not
necessarily reflect the opinions of ACMA.
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
