Re: [Numpy-discussion] Help using numpy.distutils.fcompiler for my GSoC project
Kurt Smith wrote: > On Wed, Jun 24, 2009 at 9:05 AM, David > Cournapeau wrote: > >> Kurt Smith wrote: >> >>> On Tue, Jun 23, 2009 at 10:17 PM, David Cournapeau >>> wrote: >>> >>> >>> If possible, you should not build executables, it is not portable. Compiling and linking is Ok, running is not. For a tool which is aimed a general use, I think this is important. Knowing the exact tests needed by the OP would help me to give more detailed advices. >>> Hmmm. Thanks for the input. >>> >>> Ironically, the reason we're building the executable is for >>> portability of the interoperable types. By running the genconfig >>> program it guarantees that we get the correct C type <-> Fortran type >>> correspondence set up. This is especially desirable given that >>> compiler flags can change the size of some datatypes, which would be >>> captured correctly by the genconfig program's output -- if everything >>> goes as planned ;-) We'd like to make it so that any fortran >>> procedure can be wrapped without having to modify the kind type >>> parameters of the arguments. >>> >>> >> Can't you do this without running executables ? What is not portable is >> to run executables (because they cannot always run - for example cross >> compilation on windows). Windows causes quite a headache with recent >> version of pythons w.r.t running executables if you need to link against >> the C runtime. >> > > What we're attempting to do is similar to Cython. Cython is run with > a .pyx file as argument, and outputs a standalone C file that can be > shipped and later compiled without any Cython dependence from then > onwards. > > We'd like to have f2cy do the same: it is run with fortran source > files as arguments, generates wrappers & various build scripts for > convenience (a Makefile, a setup.py, SConstruct file, etc) that are > shipped. Later these build files/scripts are used to create the > extension module, without any dependence on f2cy. So these wrappers & > build scripts must have a way to portably generate the Fortran type > <-> C type mappings, portable between compiler and platform. > > A simple example for illustration: > > a fortran subroutine to be wrapped: > > subroutine foo(a) > integer(kind=selected_int_kind(10)), intent(inout) :: a > ... > end subroutine foo > > The 'selected_int_kind(10)' call might correspond to an actual kind > type parameter of 1,2,4 or 8 depending on the fortran compiler -- some > compilers have the ktp correspond to the byte size, others just label > them sequentially, so no consistent meaning is assignable to the value > of the selected_int_kind(10) call. It is impossible beforehand to > know if that kind-type-parameter corresponds to a c_int, a c_long, > etc, since the correspondence can vary platform to platform and can be > changed with compiler flags. Some compilers will raise an error if an > incorrect assumption is made. > > We have to have the genconfig executable run beforehand to ensure that > the right correspondence is set-up, adapted to that platform and that > compiler. Again, since f2cy isn't around to make sure things are > set-up correctly, then we're stuck with running the executable to > handle all possible points in the [compiler] x [platform] space. > > There may be a via media: we could have the user supply the > type-mapping information to f2cy, which would generate the wrappers > with these mappings assumed. It wouldn't be as portable or general, > but that would be the tradeoff. It would avoid running the executable > on those platforms that won't allow it. > > >>> This would need to be run before compiling the extension module. Is >>> it possible to make this portable? >>> >>> >> No. But most of the time, you can test things without running anything. >> For example, all the type sizeofs are detected by compilation-only with >> numpy, using some C hackery. What are the exact tests you need to do ? >> > > I assume the type sizeofs are available for the C types The configure checks are specific to C, but maybe they can be adapted for fortran (I don't know enough fortran to be sure, though). The code can be found here: http://projects.scipy.org/numpy/browser/trunk/numpy/distutils/command/config.py (line 169) The idea is simple: you use sizeof(type) + a shift as an index of an array, and rely on the compiler to fail if the index is negative. cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Help using numpy.distutils.fcompiler for my GSoC project
On Wed, Jun 24, 2009 at 9:05 AM, David Cournapeau wrote: > Kurt Smith wrote: >> On Tue, Jun 23, 2009 at 10:17 PM, David Cournapeau wrote: >> >> >>> If possible, you should not build executables, it is not portable. >>> Compiling and linking is Ok, running is not. For a tool which is aimed >>> a general use, I think this is important. Knowing the exact tests >>> needed by the OP would help me to give more detailed advices. >>> >> >> Hmmm. Thanks for the input. >> >> Ironically, the reason we're building the executable is for >> portability of the interoperable types. By running the genconfig >> program it guarantees that we get the correct C type <-> Fortran type >> correspondence set up. This is especially desirable given that >> compiler flags can change the size of some datatypes, which would be >> captured correctly by the genconfig program's output -- if everything >> goes as planned ;-) We'd like to make it so that any fortran >> procedure can be wrapped without having to modify the kind type >> parameters of the arguments. >> > > Can't you do this without running executables ? What is not portable is > to run executables (because they cannot always run - for example cross > compilation on windows). Windows causes quite a headache with recent > version of pythons w.r.t running executables if you need to link against > the C runtime. What we're attempting to do is similar to Cython. Cython is run with a .pyx file as argument, and outputs a standalone C file that can be shipped and later compiled without any Cython dependence from then onwards. We'd like to have f2cy do the same: it is run with fortran source files as arguments, generates wrappers & various build scripts for convenience (a Makefile, a setup.py, SConstruct file, etc) that are shipped. Later these build files/scripts are used to create the extension module, without any dependence on f2cy. So these wrappers & build scripts must have a way to portably generate the Fortran type <-> C type mappings, portable between compiler and platform. A simple example for illustration: a fortran subroutine to be wrapped: subroutine foo(a) integer(kind=selected_int_kind(10)), intent(inout) :: a ... end subroutine foo The 'selected_int_kind(10)' call might correspond to an actual kind type parameter of 1,2,4 or 8 depending on the fortran compiler -- some compilers have the ktp correspond to the byte size, others just label them sequentially, so no consistent meaning is assignable to the value of the selected_int_kind(10) call. It is impossible beforehand to know if that kind-type-parameter corresponds to a c_int, a c_long, etc, since the correspondence can vary platform to platform and can be changed with compiler flags. Some compilers will raise an error if an incorrect assumption is made. We have to have the genconfig executable run beforehand to ensure that the right correspondence is set-up, adapted to that platform and that compiler. Again, since f2cy isn't around to make sure things are set-up correctly, then we're stuck with running the executable to handle all possible points in the [compiler] x [platform] space. There may be a via media: we could have the user supply the type-mapping information to f2cy, which would generate the wrappers with these mappings assumed. It wouldn't be as portable or general, but that would be the tradeoff. It would avoid running the executable on those platforms that won't allow it. > >> This would need to be run before compiling the extension module. Is >> it possible to make this portable? >> > > No. But most of the time, you can test things without running anything. > For example, all the type sizeofs are detected by compilation-only with > numpy, using some C hackery. What are the exact tests you need to do ? I assume the type sizeofs are available for the C types, but not the Fortran types, right? Its the fortran type sizeofs that we would need, which I doubt could be determined in a portable way (see above). Thanks again, Kurt ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Help using numpy.distutils.fcompiler for my GSoC project
Kurt Smith wrote: > On Tue, Jun 23, 2009 at 10:17 PM, David Cournapeau wrote: > > >> If possible, you should not build executables, it is not portable. >> Compiling and linking is Ok, running is not. For a tool which is aimed >> a general use, I think this is important. Knowing the exact tests >> needed by the OP would help me to give more detailed advices. >> > > Hmmm. Thanks for the input. > > Ironically, the reason we're building the executable is for > portability of the interoperable types. By running the genconfig > program it guarantees that we get the correct C type <-> Fortran type > correspondence set up. This is especially desirable given that > compiler flags can change the size of some datatypes, which would be > captured correctly by the genconfig program's output -- if everything > goes as planned ;-) We'd like to make it so that any fortran > procedure can be wrapped without having to modify the kind type > parameters of the arguments. > Can't you do this without running executables ? What is not portable is to run executables (because they cannot always run - for example cross compilation on windows). Windows causes quite a headache with recent version of pythons w.r.t running executables if you need to link against the C runtime. > This would need to be run before compiling the extension module. Is > it possible to make this portable? > No. But most of the time, you can test things without running anything. For example, all the type sizeofs are detected by compilation-only with numpy, using some C hackery. What are the exact tests you need to do ? cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Help using numpy.distutils.fcompiler for my GSoC project
On Tue, Jun 23, 2009 at 10:17 PM, David Cournapeau wrote: > If possible, you should not build executables, it is not portable. > Compiling and linking is Ok, running is not. For a tool which is aimed > a general use, I think this is important. Knowing the exact tests > needed by the OP would help me to give more detailed advices. Hmmm. Thanks for the input. Ironically, the reason we're building the executable is for portability of the interoperable types. By running the genconfig program it guarantees that we get the correct C type <-> Fortran type correspondence set up. This is especially desirable given that compiler flags can change the size of some datatypes, which would be captured correctly by the genconfig program's output -- if everything goes as planned ;-) We'd like to make it so that any fortran procedure can be wrapped without having to modify the kind type parameters of the arguments. For clarity: is it the actual steps to run the executable that isn't portable (spawning, etc)? Or is the problem in compiling the object file to an executable? Would it be possible to compile the executable, have it run on those systems that 'work' -- which would hopefully be any Unix-flavor system, and have the user do it manually otherwise? That's suboptimal to say the least, but possibly worth it for the benefits of complete type interoperability. FYI, the genconfig.f95 file is completely self-contained, i.e. we just need to do, essentially, $ fort-compiler genconfig.f95 -o genconfig && ./genconfig (adapted to the platform, of course). This would need to be run before compiling the extension module. Is it possible to make this portable? Thanks again, Kurt ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Help using numpy.distutils.fcompiler for my GSoC project
On Wed, Jun 24, 2009 at 3:41 AM, Lisandro Dalcin wrote:
> On Mon, Jun 22, 2009 at 11:53 PM, Kurt Smith wrote:
>> Hello,
>>
>> Is there a way for numpy.distutils to compile a fortran source file
>> into an executable?
>
> If the whole point of building the executable is to run it in order to
> parse the output, then you can start with this:
>
> $ cat setup.py
> from numpy.distutils.core import setup
> from numpy.distutils.command.config import config as config_orig
>
> helloworld = """
> program hello
> write(*,*) "Hello, World!"
> end program hello
> """
>
> class config(config_orig):
> def run(self):
> self.try_run(helloworld, lang='f90')
>
> setup(name="ConfigF90",
> cmdclass={'config' : config})
>
>
> $ python setup.py config
> <... lots of ouput ...>
> gfortran:f90: _configtest.f90
> /usr/bin/gfortran -Wall -Wall _configtest.o -lgfortran -o _configtest
> _configtest
> Hello, World!
> success!
> removing: _configtest.f90 _configtest.o _configtest
>
>
> In order to actually capture the ouput, you will have to implement
> method spawn() in class config(), likely using subprocess module (or
> older os.* APIS's for max backward compatibility)
If possible, you should not build executables, it is not portable.
Compiling and linking is Ok, running is not. For a tool which is aimed
a general use, I think this is important. Knowing the exact tests
needed by the OP would help me to give more detailed advices.
cheers,
David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Help using numpy.distutils.fcompiler for my GSoC project
On Tue, Jun 23, 2009 at 1:41 PM, Lisandro Dalcin wrote:
> On Mon, Jun 22, 2009 at 11:53 PM, Kurt Smith wrote:
>> Hello,
>>
>> Is there a way for numpy.distutils to compile a fortran source file
>> into an executable?
>
> If the whole point of building the executable is to run it in order to
> parse the output, then you can start with this:
>
> $ cat setup.py
> from numpy.distutils.core import setup
> from numpy.distutils.command.config import config as config_orig
>
> helloworld = """
> program hello
> write(*,*) "Hello, World!"
> end program hello
> """
>
> class config(config_orig):
> def run(self):
> self.try_run(helloworld, lang='f90')
>
> setup(name="ConfigF90",
> cmdclass={'config' : config})
>
>
> $ python setup.py config
> <... lots of ouput ...>
> gfortran:f90: _configtest.f90
> /usr/bin/gfortran -Wall -Wall _configtest.o -lgfortran -o _configtest
> _configtest
> Hello, World!
> success!
> removing: _configtest.f90 _configtest.o _configtest
Thank you!
The comments at the top of numpy.distutils.command.config.py make me
think that try_run won't work for all fortran compilers on all
platforms. But this certainly helps and I've almost got it solved.
What I've discovered is something more low-level, like the following:
$ cat batch.py
from numpy.distutils.fcompiler import new_fcompiler
fcomp = new_fcompiler() # will grab the default fcompiler, overridable
with keyword args.
fcomp.customize()
objs = fcomp.compile(['./genconfig.f95']) # compiles into an object.
fcomp.link_executable(objs, 'genconfig') # makes the executable 'genconfig'
# spawn the executable here to generate config files.
The above works for simple cases (but for the spawning step, but I'll
have that working soon) -- I'll probably end up wrapping it in a run
method inside a config subclass, like you have above.
Thanks again,
Kurt
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Help using numpy.distutils.fcompiler for my GSoC project
On Mon, Jun 22, 2009 at 11:53 PM, Kurt Smith wrote:
> Hello,
>
> Is there a way for numpy.distutils to compile a fortran source file
> into an executable?
If the whole point of building the executable is to run it in order to
parse the output, then you can start with this:
$ cat setup.py
from numpy.distutils.core import setup
from numpy.distutils.command.config import config as config_orig
helloworld = """
program hello
write(*,*) "Hello, World!"
end program hello
"""
class config(config_orig):
def run(self):
self.try_run(helloworld, lang='f90')
setup(name="ConfigF90",
cmdclass={'config' : config})
$ python setup.py config
<... lots of ouput ...>
gfortran:f90: _configtest.f90
/usr/bin/gfortran -Wall -Wall _configtest.o -lgfortran -o _configtest
_configtest
Hello, World!
success!
removing: _configtest.f90 _configtest.o _configtest
In order to actually capture the ouput, you will have to implement
method spawn() in class config(), likely using subprocess module (or
older os.* APIS's for max backward compatibility)
Hope this helps,
> I'm not well-versed in the black arts of
> distutils or numpy.distutils. I know the basics and am quickly
> reading up on it -- any pointers would be very appreciated.
>
> Thanks,
>
> Kurt Smith
> ___
> Numpy-discussion mailing list
> Numpy-discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
--
Lisandro Dalcín
---
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
