Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Charles R Harris]

PR #6429 is a preliminary cut at removing single file build support. A bit
of cleanup remains, mostly rearranging some defines for style.



Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Mon, Oct 5, 2015 at 11:26 PM, Nathaniel Smith  wrote:

> Hi all,
>
> For a long time, NumPy has supported two different ways of being compiled:
>
> "Separate compilation" mode: like most C projects, each .c file gets
> compiled to a .o file, and then the .o files get linked together to
> make a shared library. (This has been the default since 1.8.0.)
>
> "One file" mode: first concatenate all the .c files together to make
> one monster .c file, and then compile that .c file to make a shared
> library. (This was the default before 1.8.0.)
>
> Supporting these two different build modes creates a drag on
> development progress; in particular Stefan recently ran into this in
> this experiments with porting parts of the NumPy internals to Cython:
>   https://github.com/numpy/numpy/pull/6408
> (I suspect the particular problem he's running into can be fixed b/c
> so far he only has one .pyx file, but I also suspect that it will be
> impossible to support "one file" mode once we have multiple .pyx
> files.)
>
> There are some rumors that "one file" mode might be needed on some
> obscure platform somewhere, or that it might be necessary for
> statically linking numpy into the CPython executable, but we can't
> continue supporting things forever based only on rumors. If all we can
> get are rumors, then eventually we have to risk breaking things just
> to force anyone who cares to actually show up and explain what they
> need so we can support it properly :-).
>

Assuming one of the rumour is related to some comments I made some time
(years ?) earlier, the context was the ability to hide exported symbols. As
you know, the issue is not to build extensions w/ multiple compilation
units, but sharing functionalities between them without sharing them
outside the extension. I am just reiterating that point so that we all
discuss under the right context :)

I also agree the current situation is not sustainable -- as we discussed
privately before, cythonizing numpy.core is made quite more complicated by
this. I have myself quite a few issues w/ cythonizing the other parts of
umath. I would also like to support the static link better than we do now
(do we know some static link users we can contact to validate our approach
?)

Currently, what we have in numpy core is the following:

numpy.core.multiarray -> compilation units in numpy/core/src/multiarray/ +
statically link npymath
numpy.core.umath -> compilation units in numpy/core/src/umath + statically
link npymath/npysort + some shenanigans to use things in
numpy.core.multiarray

I would suggest to have a more layered approach, to enable both 'normal'
build and static build, without polluting the public namespace too much.
This is an approach followed by most large libraries (e.g. MKL), and is
fairly flexible.

Concretely, we could start by putting more common functionalities (aka the
'core' library) into its own static library. The API would be considered
private to numpy (no stability guaranteed outside numpy), and every
exported symbol from that library would be decorated appropriately to avoid
potential clashes (e.g. '_npy_internal_').

FWIW, that has always been my intention to go toward this when I split up
multiarray/umath into multiple .c files and extracted out npymath.

cheers,
David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Antoine Pitrou]

On Tue, 6 Oct 2015 11:00:30 +0100
David Cournapeau  wrote:
> 
> Assuming one of the rumour is related to some comments I made some time
> (years ?) earlier, the context was the ability to hide exported symbols. As
> you know, the issue is not to build extensions w/ multiple compilation
> units, but sharing functionalities between them without sharing them
> outside the extension.

Can't you use the visibility attribute with gcc for this?
Other Unix compilers probably provide something similar. The issue
doesn't exist on Windows by construction.

https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/Function-Attributes.html#Function-Attributes

By the way, external packages may reuse the npy_* functions, so I would
like them not the be hidden :-)

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 12:07 PM, Antoine Pitrou  wrote:

> On Tue, 6 Oct 2015 11:00:30 +0100
> David Cournapeau  wrote:
> >
> > Assuming one of the rumour is related to some comments I made some time
> > (years ?) earlier, the context was the ability to hide exported symbols.
> As
> > you know, the issue is not to build extensions w/ multiple compilation
> > units, but sharing functionalities between them without sharing them
> > outside the extension.
>
> Can't you use the visibility attribute with gcc for this?
>

We do that already for gcc, I think the question was whether every platform
supported this or not (and whether we should care).


> Other Unix compilers probably provide something similar. The issue
> doesn't exist on Windows by construction.
>
>
> https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/Function-Attributes.html#Function-Attributes
>
> By the way, external packages may reuse the npy_* functions, so I would
> like them not the be hidden :-)
>

The npy_ functions in npymath were designed to be exported. Those would
stay that way.

David

>
> Regards
>
> Antoine.
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Antoine Pitrou]

On Tue, 6 Oct 2015 09:40:43 -0700
Nathaniel Smith  wrote:
> 
> If you need some npy_* function it'd be much better to let us know
> what it is and let us export it in an intentional way, instead of just
> relying on whatever stuff we accidentally exposed?

Ok, we seem to be using only the complex math functions (npy_cpow and
friends, I could make a complete list if required).

And, of course, we would also benefit from the CBLAS functions (or any
kind of C wrappers around them) :-)
https://github.com/numpy/numpy/issues/6324

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Nathaniel Smith]

On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou  wrote:
> On Tue, 6 Oct 2015 09:40:43 -0700
> Nathaniel Smith  wrote:
>>
>> If you need some npy_* function it'd be much better to let us know
>> what it is and let us export it in an intentional way, instead of just
>> relying on whatever stuff we accidentally exposed?
>
> Ok, we seem to be using only the complex math functions (npy_cpow and
> friends, I could make a complete list if required).

And how are you getting at them? Are you just relying the way that on
ELF systems, if two libraries are loaded into the same address space
then they automatically get access to each other's symbols, even if
they aren't linked to each other? What do you do on Windows?

> And, of course, we would also benefit from the CBLAS functions (or any
> kind of C wrappers around them) :-)
> https://github.com/numpy/numpy/issues/6324

This is difficult to do from NumPy itself -- we don't necessarily have
access to a full BLAS or LAPACK API -- in some configurations we fall
back on our minimal internal implementations that just have what we
need.

There was an interesting idea that came up in some discussions here a
few weeks ago -- we already know that we want to package up BLAS
inside a Python package that (numpy / scipy / scikit-learn / ...) can
depend on and assume is there to link against.

Maybe this new package would also be a good place for exposing these wrappers?

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Nathaniel Smith]

On Tue, Oct 6, 2015 at 9:51 AM, David Cournapeau  wrote:
>
> On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith  wrote:
>>
>> On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau 
>> wrote:
>> > The npy_ functions in npymath were designed to be exported. Those would
>> > stay
>> > that way.
>>
>> If we want to export these then I vote that we either:
>> - use the usual API export mechanism, or else
>> - provide a static library for people to link to, instead of trying to
>> do runtime binding. (I.e. drop it in some known place, and then
>> provide some functions for extension modules to find it at build time
>> -- similar to how np.get_include() works.)
>
> Unless something changed, that's more or less how it works already (npymath
> is used in scipy, for example, which was one of the rationale for writing it
> in the first place !).

Okay... in fact multiarray.so right now *does* export tons and tons of
random junk into the global symbol namespace (on systems like Linux
that do have a global symbol namespace), so it isn't obvious whether
people are asking for that to continue :-). I'm just specifically
saying that we should try to get this back down to the 1 exported
symbol.

(Try:
   objdump -T $(python -c 'import numpy; print(numpy.core.multiarray.__file__)')
This *should* print 1 line... I currently get ~700. numpy.core.umath
is similar.)

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 6:07 PM, Nathaniel Smith  wrote:

> On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou 
> wrote:
> > On Tue, 6 Oct 2015 09:40:43 -0700
> > Nathaniel Smith  wrote:
> >>
> >> If you need some npy_* function it'd be much better to let us know
> >> what it is and let us export it in an intentional way, instead of just
> >> relying on whatever stuff we accidentally exposed?
> >
> > Ok, we seem to be using only the complex math functions (npy_cpow and
> > friends, I could make a complete list if required).
>
> And how are you getting at them? Are you just relying the way that on
> ELF systems, if two libraries are loaded into the same address space
> then they automatically get access to each other's symbols, even if
> they aren't linked to each other? What do you do on Windows?
>

It is possible (and documented) to use any of the npy_ symbols from npymath
from outside numpy:
http://docs.scipy.org/doc/numpy-dev/reference/c-api.coremath.html#linking-against-the-core-math-library-in-an-extension

The design is not perfect (I was young and foolish :) ), but it has worked
fairly well and has been used in at least scipy since the 1.4/1.5 days IIRC
(including windows).

David


>
> > And, of course, we would also benefit from the CBLAS functions (or any
> > kind of C wrappers around them) :-)
> > https://github.com/numpy/numpy/issues/6324
>
> This is difficult to do from NumPy itself -- we don't necessarily have
> access to a full BLAS or LAPACK API -- in some configurations we fall
> back on our minimal internal implementations that just have what we
> need.
>
> There was an interesting idea that came up in some discussions here a
> few weeks ago -- we already know that we want to package up BLAS
> inside a Python package that (numpy / scipy / scikit-learn / ...) can
> depend on and assume is there to link against.
>
> Maybe this new package would also be a good place for exposing these
> wrappers?
>
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 6:14 PM, Nathaniel Smith  wrote:

> On Tue, Oct 6, 2015 at 10:10 AM, David Cournapeau 
> wrote:
> >
> >
> > On Tue, Oct 6, 2015 at 6:07 PM, Nathaniel Smith  wrote:
> >>
> >> On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou 
> >> wrote:
> >> > On Tue, 6 Oct 2015 09:40:43 -0700
> >> > Nathaniel Smith  wrote:
> >> >>
> >> >> If you need some npy_* function it'd be much better to let us know
> >> >> what it is and let us export it in an intentional way, instead of
> just
> >> >> relying on whatever stuff we accidentally exposed?
> >> >
> >> > Ok, we seem to be using only the complex math functions (npy_cpow and
> >> > friends, I could make a complete list if required).
> >>
> >> And how are you getting at them? Are you just relying the way that on
> >> ELF systems, if two libraries are loaded into the same address space
> >> then they automatically get access to each other's symbols, even if
> >> they aren't linked to each other? What do you do on Windows?
> >
> >
> > It is possible (and documented) to use any of the npy_ symbols from
> npymath
> > from outside numpy:
> >
> http://docs.scipy.org/doc/numpy-dev/reference/c-api.coremath.html#linking-against-the-core-math-library-in-an-extension
> >
> > The design is not perfect (I was young and foolish :) ), but it has
> worked
> > fairly well and has been used in at least scipy since the 1.4/1.5 days
> IIRC
> > (including windows).
>
> Okay, so just to confirm, it looks like this does indeed implement the
> static linking thing I just suggested (so perhaps I am also young and
> foolish ;-)) -- from looking at the output of get_info("npymath"), it
> seems to add -I.../numpy/core/include to the compiler flags, add
> -lnpymath -L.../numpy/core/lib to the linker flags, and then
> .../numpy/core/lib contains only libnpymath.a, so it's static linking.
>

Yes, I was not trying to argue otherwise. If you thought I was, blame it on
my poor English (which sadly does not get better as I get less young...).

My proposal is to extend this technique for *internal* API, but with the
following differences:
 * the declarations are not put in any public header
 * we don't offer any way to link to this library, and name it something
scary enough that people would have to be foolish (young or not) to use it.

David


> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Antoine Pitrou]

On Tue, 6 Oct 2015 10:07:13 -0700
Nathaniel Smith  wrote:
> 
> And how are you getting at them? Are you just relying the way that on
> ELF systems, if two libraries are loaded into the same address space
> then they automatically get access to each other's symbols, even if
> they aren't linked to each other? What do you do on Windows?

Well it seems to work on Windows too, thanks to
numpy.distutils.misc_util.get_info('npymath').

Under Windows, I seem to have a
"\site-packages\numpy\core\lib\npymath.lib" static library,
and there's also a "npy-pkg-config" subdirectory there with some INI
files in it.  Hopefully you know better than me what this all is :-)

> > And, of course, we would also benefit from the CBLAS functions (or any
> > kind of C wrappers around them) :-)
> > https://github.com/numpy/numpy/issues/6324
> 
> This is difficult to do from NumPy itself -- we don't necessarily have
> access to a full BLAS or LAPACK API -- in some configurations we fall
> back on our minimal internal implementations that just have what we
> need.

I'm thinking about the functions exposed in
"numpy/core/src/private/npy_cblas.h".
My knowledge of the Numpy build system doesn't allow me to tell if it's
always available or not :-)

> There was an interesting idea that came up in some discussions here a
> few weeks ago -- we already know that we want to package up BLAS
> inside a Python package that (numpy / scipy / scikit-learn / ...) can
> depend on and assume is there to link against.
> 
> Maybe this new package would also be a good place for exposing these wrappers?

Yeah, why not - as long as there's something well-known and
well-supported to depend on.  But given Numpy is a hard dependency for
all the other packages you mentioned, it may make sense (and simplify
dependency management) to bundle it with Numpy.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith  wrote:

> On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau 
> wrote:
> > The npy_ functions in npymath were designed to be exported. Those would
> stay
> > that way.
>
> If we want to export these then I vote that we either:
> - use the usual API export mechanism, or else
> - provide a static library for people to link to, instead of trying to
> do runtime binding. (I.e. drop it in some known place, and then
> provide some functions for extension modules to find it at build time
> -- similar to how np.get_include() works.)
>

Unless something changed, that's more or less how it works already (npymath
is used in scipy, for example, which was one of the rationale for writing
it in the first place !).

You access the compilation/linking issues through the numpy distutils
get_info function.

David


> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Nathaniel Smith]

On Tue, Oct 6, 2015 at 9:51 AM, David Cournapeau  wrote:
>
> On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith  wrote:
>>
>> On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau 
>> wrote:
>> > The npy_ functions in npymath were designed to be exported. Those would
>> > stay
>> > that way.
>>
>> If we want to export these then I vote that we either:
>> - use the usual API export mechanism, or else
>> - provide a static library for people to link to, instead of trying to
>> do runtime binding. (I.e. drop it in some known place, and then
>> provide some functions for extension modules to find it at build time
>> -- similar to how np.get_include() works.)
>
> Unless something changed, that's more or less how it works already (npymath
> is used in scipy, for example, which was one of the rationale for writing it
> in the first place !).

Okay... in fact multiarray.so right now *does* export tons and tons of
random junk into the global symbol namespace (on systems like Linux
that do have a global symbol namespace), so it isn't obvious whether
people are asking for that to continue :-). I'm just specifically
saying that we should try to get this back down to the 1 exported
symbol.

(Try:
   objdump -T $(python -c 'import numpy; print(numpy.core.multiarray.__file__)')
This *should* print 1 line... I currently get ~700. numpy.core.umath
is similar.)

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 5:51 PM, David Cournapeau  wrote:

>
>
> On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith  wrote:
>
>> On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau 
>> wrote:
>> > The npy_ functions in npymath were designed to be exported. Those would
>> stay
>> > that way.
>>
>> If we want to export these then I vote that we either:
>> - use the usual API export mechanism, or else
>> - provide a static library for people to link to, instead of trying to
>> do runtime binding. (I.e. drop it in some known place, and then
>> provide some functions for extension modules to find it at build time
>> -- similar to how np.get_include() works.)
>>
>
> Unless something changed, that's more or less how it works already
> (npymath is used in scipy, for example, which was one of the rationale for
> writing it in the first place !).
>
> You access the compilation/linking issues through the numpy distutils
> get_info function.
>

And my suggestion is to use a similar mechanism for multiarray and umath,
so that in the end the exported Python C API is just a thin layer on top of
the underlying static library. That would make cython and I suspect static
linking quite a bit easier. The API between the low layer and python C API
of multiarray/umath would be considered private and outside any API/ABI
stability.

IOW, it would be an internal change, and should not cause visible changes
to the users, except that some _npy_private_ symbols would be exported (but
you would be crazy to use them, and the prototype declarations would not be
available when you install numpy anyway). Think of those as the internal
driver API/ABI of Linux or similar.

David

>
> David
>
>
>> -n
>>
>> --
>> Nathaniel J. Smith -- http://vorpus.org
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Nathaniel Smith]

On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau  wrote:
> The npy_ functions in npymath were designed to be exported. Those would stay
> that way.

If we want to export these then I vote that we either:
- use the usual API export mechanism, or else
- provide a static library for people to link to, instead of trying to
do runtime binding. (I.e. drop it in some known place, and then
provide some functions for extension modules to find it at build time
-- similar to how np.get_include() works.)

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 6:18 PM, David Cournapeau  wrote:

>
>
> On Tue, Oct 6, 2015 at 6:14 PM, Nathaniel Smith  wrote:
>
>> On Tue, Oct 6, 2015 at 10:10 AM, David Cournapeau 
>> wrote:
>> >
>> >
>> > On Tue, Oct 6, 2015 at 6:07 PM, Nathaniel Smith  wrote:
>> >>
>> >> On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou 
>> >> wrote:
>> >> > On Tue, 6 Oct 2015 09:40:43 -0700
>> >> > Nathaniel Smith  wrote:
>> >> >>
>> >> >> If you need some npy_* function it'd be much better to let us know
>> >> >> what it is and let us export it in an intentional way, instead of
>> just
>> >> >> relying on whatever stuff we accidentally exposed?
>> >> >
>> >> > Ok, we seem to be using only the complex math functions (npy_cpow and
>> >> > friends, I could make a complete list if required).
>> >>
>> >> And how are you getting at them? Are you just relying the way that on
>> >> ELF systems, if two libraries are loaded into the same address space
>> >> then they automatically get access to each other's symbols, even if
>> >> they aren't linked to each other? What do you do on Windows?
>> >
>> >
>> > It is possible (and documented) to use any of the npy_ symbols from
>> npymath
>> > from outside numpy:
>> >
>> http://docs.scipy.org/doc/numpy-dev/reference/c-api.coremath.html#linking-against-the-core-math-library-in-an-extension
>> >
>> > The design is not perfect (I was young and foolish :) ), but it has
>> worked
>> > fairly well and has been used in at least scipy since the 1.4/1.5 days
>> IIRC
>> > (including windows).
>>
>> Okay, so just to confirm, it looks like this does indeed implement the
>> static linking thing I just suggested (so perhaps I am also young and
>> foolish ;-)) -- from looking at the output of get_info("npymath"), it
>> seems to add -I.../numpy/core/include to the compiler flags, add
>> -lnpymath -L.../numpy/core/lib to the linker flags, and then
>> .../numpy/core/lib contains only libnpymath.a, so it's static linking.
>>
>
> Yes, I was not trying to argue otherwise. If you thought I was, blame it
> on my poor English (which sadly does not get better as I get less young...).
>
> My proposal is to extend this technique for *internal* API, but with the
> following differences:
>  * the declarations are not put in any public header
>  * we don't offer any way to link to this library, and name it something
> scary enough that people would have to be foolish (young or not) to use it.
>

I am stupid: we of course do not even ship that internal library, it would
just be linked into multiarray/umath and never installed or part of binary
packages.

David

>
> David
>
>
>> -n
>>
>> --
>> Nathaniel J. Smith -- http://vorpus.org
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 5:58 PM, Nathaniel Smith  wrote:

> On Tue, Oct 6, 2015 at 9:51 AM, David Cournapeau 
> wrote:
> >
> > On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith  wrote:
> >>
> >> On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau 
> >> wrote:
> >> > The npy_ functions in npymath were designed to be exported. Those
> would
> >> > stay
> >> > that way.
> >>
> >> If we want to export these then I vote that we either:
> >> - use the usual API export mechanism, or else
> >> - provide a static library for people to link to, instead of trying to
> >> do runtime binding. (I.e. drop it in some known place, and then
> >> provide some functions for extension modules to find it at build time
> >> -- similar to how np.get_include() works.)
> >
> > Unless something changed, that's more or less how it works already
> (npymath
> > is used in scipy, for example, which was one of the rationale for
> writing it
> > in the first place !).
>
> Okay... in fact multiarray.so right now *does* export tons and tons of
> random junk into the global symbol namespace (on systems like Linux
> that do have a global symbol namespace), so it isn't obvious whether
> people are asking for that to continue :-). I'm just specifically
> saying that we should try to get this back down to the 1 exported
> symbol.
>
> (Try:
>objdump -T $(python -c 'import numpy;
> print(numpy.core.multiarray.__file__)')
> This *should* print 1 line... I currently get ~700. numpy.core.umath
> is similar.)
>
>
I think this overestimates the amount by quite a bit, since you see GLIBC
symbols, etc... I am using nm -Dg --defined-only $(python -c 'import numpy;
print(numpy.core.multiarray.__file__)')instead.

I see around 290 symboles: the npy_ from npymath don't bother me, but the
ones from npysort do. We should at least prefix those with npy_ (I don't
think npysort API has ever been publisher in our header like npymath was ?)

David

-n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Nathaniel Smith]

On Tue, Oct 6, 2015 at 10:10 AM, David Cournapeau  wrote:
>
>
> On Tue, Oct 6, 2015 at 6:07 PM, Nathaniel Smith  wrote:
>>
>> On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou 
>> wrote:
>> > On Tue, 6 Oct 2015 09:40:43 -0700
>> > Nathaniel Smith  wrote:
>> >>
>> >> If you need some npy_* function it'd be much better to let us know
>> >> what it is and let us export it in an intentional way, instead of just
>> >> relying on whatever stuff we accidentally exposed?
>> >
>> > Ok, we seem to be using only the complex math functions (npy_cpow and
>> > friends, I could make a complete list if required).
>>
>> And how are you getting at them? Are you just relying the way that on
>> ELF systems, if two libraries are loaded into the same address space
>> then they automatically get access to each other's symbols, even if
>> they aren't linked to each other? What do you do on Windows?
>
>
> It is possible (and documented) to use any of the npy_ symbols from npymath
> from outside numpy:
> http://docs.scipy.org/doc/numpy-dev/reference/c-api.coremath.html#linking-against-the-core-math-library-in-an-extension
>
> The design is not perfect (I was young and foolish :) ), but it has worked
> fairly well and has been used in at least scipy since the 1.4/1.5 days IIRC
> (including windows).

Okay, so just to confirm, it looks like this does indeed implement the
static linking thing I just suggested (so perhaps I am also young and
foolish ;-)) -- from looking at the output of get_info("npymath"), it
seems to add -I.../numpy/core/include to the compiler flags, add
-lnpymath -L.../numpy/core/lib to the linker flags, and then
.../numpy/core/lib contains only libnpymath.a, so it's static linking.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Antoine Pitrou]

On Mon, 5 Oct 2015 15:26:17 -0700
Nathaniel Smith  wrote:
> Hi all,
> 
> For a long time, NumPy has supported two different ways of being compiled:
> 
> "Separate compilation" mode: like most C projects, each .c file gets
> compiled to a .o file, and then the .o files get linked together to
> make a shared library. (This has been the default since 1.8.0.)
> 
> "One file" mode: first concatenate all the .c files together to make
> one monster .c file, and then compile that .c file to make a shared
> library. (This was the default before 1.8.0.)
> 
[...]
> 
> There are some rumors that "one file" mode might be needed on some
> obscure platform somewhere, or that it might be necessary for
> statically linking numpy into the CPython executable, but we can't
> continue supporting things forever based only on rumors.

If those rumors were true, CPython would not even be able to build
(the _io module in 3.x is linked from several C object files, for
example).

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion