Re: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way

[Nathaniel Smith]

On Mon, Sep 21, 2015 at 7:29 AM, Jaime Fernández del Río
 wrote:
> We have the PyArrayObject vs PyArrayObject_fields definition in
> ndarraytypes.h that is used to enforce access to the members through inline
> functions rather than directly,  which seems to me like the right way to go:
> don't leave stones unturned, hide everything and provide PyUFunc_NIN,
> PyUFunc_NOUT and friends to handle those too.

The PyArrayObject vs PyArrayObject_fields distinction is only enabled
if a downstream library explicitly requests it with #define
NPY_NO_DEPRECATED_API, though -- the idea is that the changes in this
NEP would be enabled unconditionally, even for old code. So the reason
nin/nout/nargs are left exposed in this proposal is that there's some
existing code out there that would break (until updated) if we hid
them, and not much benefit to breaking it.

If we're fine with breaking that code then we could just hide them
unconditionally too. The only code I found in the wild that would be
affected is the "rational" user-defined dtype, which would be
trivially fixable since the only thing it does with ufunc->nargs is a
quick consistency check:

  
https://github.com/numpy/numpy-dtypes/blob/c0175a6b1c5aa89b4520b29487f06d0e200e2a03/npytypes/rational/rational.c#L1140-L1151

Also it's not 100% clear right now whether we even want to keep
supporting the old user-defined dtype API that this particular code is
based around. But if this code uses ufunc->nargs then perhaps other
code does too? I'm open to opinions -- I doubt it matters that much
either way. I just want to make sure that we can hide the other stuff
:-).

When it comes to evolving these APIs in general: one unfortunate thing
about the PyArrayObject changes in 1.7 is that because they were
implemented using *inline* functions (/macros) they haven't affected
the a*B*i exposure at all, even in code that has upgraded to the new
calling conventions. While user code no longer *names* the internal
fields directly, we still have to implement exactly the same fields
and put them in exactly the same place in memory or else break ABI.
And the other unfortunate thing is that we don't really have a
mechanism for saying "okay, we're dropping support for the old way of
doing things in 1.xx" -- in particular the current
NPY_NO_DEPRECATED_API mechanism doesn't give us any way to detect and
error out if someone tries to use an old version of the APIs, so ABI
breaks still mean segfaults. I'm thinking that if/when we figure out
how to implement the "sliding window" API/ABI idea that we talked
about at SciPy, then that will give us a strategy for cleanly
transitioning to a world with a maintainable API+ABI and it becomes
worth sitting down and making up a set of setters/getters for the
attributes that we want to make public in a maintainable way. But
until then our only real options are either hard breaks or nothing, so
unless we want to do a hard break there's not much point talking about
it.

-n

> On Sun, Sep 20, 2015 at 9:13 PM, Nathaniel Smith  wrote:
>>
>> Hi all,
>>
>> Here's a first draft NEP for comments.
>>
>> --
>>
>> Synopsis
>> 
>>
>> Improving numpy's dtype system requires that ufunc loops start having
>> access to details of the specific dtype instance they are acting on:
>> e.g. an implementation of np.equal for strings needs access to the
>> dtype object in order to know what "n" to pass to strncmp. Similar
>> issues arise with variable length strings, missing values, categorical
>> data, unit support, datetime with timezone support, etc. -- this is a
>> major blocker for improving numpy.
>>
>> Unfortunately, the current ufunc inner loop function signature makes
>> it very difficult to provide this information. We might be able to
>> wedge it in there, but it'd be ugly.
>>
>> The other option would be to change the signature. What would happen
>> if we did this? For most common uses of the C API/ABI, we could do
>> this easily while maintaining backwards compatibility. But there are
>> also some rarely-used parts  of the API/ABI that would be
>> prohibitively difficult to preserve.
>>
>> In addition, there are other potential changes to ufuncs on the
>> horizon (e.g. extensions of gufuncs to allow them to be used more
>> generally), and the current API exposure is so massive that any such
>> changes will be difficult to make in a fully compatible way. This NEP
>> thus considers the possibility of closing down the ufunc API to a
>> minimal, maintainable subset of the current API.
>>
>> To better understand the consequences of this potential change, I
>> performed an exhaustive analysis of all the code on Github, Bitbucket,
>> and Fedora, among others. The results make me highly confident that of
>> all the publically available projects in the world, the only ones
>> which touch the problematic parts of the ufunc API are: Numba,
>> dynd-python, and `gulinalg `_
>> (with the 

Re: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way

[Nathaniel Smith]

Hi Antoine,

On Mon, Sep 21, 2015 at 2:44 AM, Antoine Pitrou  wrote:
>
> Hi Nathaniel,
>
> On Sun, 20 Sep 2015 21:13:30 -0700
> Nathaniel Smith  wrote:
>> Given this, I propose that for 1.11 we:
>> 1) go ahead and hide/disable the problematic parts of the ABI/API,
>> 2) coordinate with the known affected projects to minimize disruption
>> to their users (which is made easier since they are all projects that
>> are almost exclusively distributed via conda, which enforces strict
>> NumPy ABI versioning),
>> 3) publicize these changes widely so as to give any private code that
>> might be affected a chance to speak up or adapt, and
>> 4) leave the "ABI version tag" as it is, so as not to force rebuilds
>> of the vast majority of projects that will be unaffected by these
>> changes.
>
> Thanks for a detailed and clear explanation of the proposed changes.
> As far as Numba is concerned, making changes is ok for us provided
> Numpy provides APIs to do what we want.

Good to hear, thanks!

Any interest in designing those new APIs that will do what you want?
:-) A no-brainer is that PyUFuncObject should just take responsibility
for managing the memory of its own internal arrays instead of assuming
that they'll always be statically allocated and forcing elaborate
workarounds when they're not, but there is a lot of complicated stuff
going on in numba/npyufunc/_internal.c... I am even wondering whether
we should go ahead and reify a first-class "ufunc loop" object, so it
can have its own refcounting.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Governance model request

[Stefan van der Walt]

On 2015-09-20 11:20:28, Travis Oliphant  wrote:
> I would recommend three possible adjustments to the steering council
> concept.
>
> 1 - define a BDFL for the council.  I would nominate chuck Harris
>
> 2 - limit the council to 3 people.  I would nominate chuck, nathaniel, and
> pauli.
>
> 3 - add me as a permanent member of the steering council.

I would split the above into two parts: a suggestion on how to change
the governance model (first half of 1 and 2) and then some thoughts on
what to do once those changes have been made (latter half of 1 and 2, as
well as 3).

For now, since those changes are not in place yet, it's probably best
to focus on the governance model.

I would agree that one person (or a very small group) is best suited to
"getting things unstuck".  And, personally, I believe it best for that
person/persons to be elected by the community (whatever we define "the
community" to be)---which is what I presume you suggested when you
mentioned nominating candidates.

Since Matthew mentioned the governance proposal we're working on, here
is a very early draft:

https://github.com/stefanv/skimage-org/blob/governance_proposal/governance.md

As I said, this is still a work-in-progress--comments are welcome.
E.g., the weighting element in the voting has to be fine tuned (but was
put in place to prevent rapid take-overs).

Essentially, we need:

- a way for community members to express disagreement without being
  ousted,
- protection against individuals who want to exert disproportional
  influence,
- protection against those in leadership roles who cause the project
  long-term harm,
- and a way for the community to change the direction of the project if
  they so wished.

Stéfan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way

[Bryan Van de Ven]

> 
> until then our only real options are either hard breaks or nothing, so
> unless we want to do a hard break there's not much point talking about
> it.

I think this is the most important sentence from this thread. Thank you 
Nathaniel for you extremely thorough analysis of the impact on real-world 
projects. 

Bryan 


> On Sep 22, 2015, at 12:23 AM, Nathaniel Smith  wrote:
> 
> On Mon, Sep 21, 2015 at 7:29 AM, Jaime Fernández del Río
>  wrote:
>> We have the PyArrayObject vs PyArrayObject_fields definition in
>> ndarraytypes.h that is used to enforce access to the members through inline
>> functions rather than directly,  which seems to me like the right way to go:
>> don't leave stones unturned, hide everything and provide PyUFunc_NIN,
>> PyUFunc_NOUT and friends to handle those too.
> 
> The PyArrayObject vs PyArrayObject_fields distinction is only enabled
> if a downstream library explicitly requests it with #define
> NPY_NO_DEPRECATED_API, though -- the idea is that the changes in this
> NEP would be enabled unconditionally, even for old code. So the reason
> nin/nout/nargs are left exposed in this proposal is that there's some
> existing code out there that would break (until updated) if we hid
> them, and not much benefit to breaking it.
> 
> If we're fine with breaking that code then we could just hide them
> unconditionally too. The only code I found in the wild that would be
> affected is the "rational" user-defined dtype, which would be
> trivially fixable since the only thing it does with ufunc->nargs is a
> quick consistency check:
> 
>  
> https://github.com/numpy/numpy-dtypes/blob/c0175a6b1c5aa89b4520b29487f06d0e200e2a03/npytypes/rational/rational.c#L1140-L1151
> 
> Also it's not 100% clear right now whether we even want to keep
> supporting the old user-defined dtype API that this particular code is
> based around. But if this code uses ufunc->nargs then perhaps other
> code does too? I'm open to opinions -- I doubt it matters that much
> either way. I just want to make sure that we can hide the other stuff
> :-).
> 
> When it comes to evolving these APIs in general: one unfortunate thing
> about the PyArrayObject changes in 1.7 is that because they were
> implemented using *inline* functions (/macros) they haven't affected
> the a*B*i exposure at all, even in code that has upgraded to the new
> calling conventions. While user code no longer *names* the internal
> fields directly, we still have to implement exactly the same fields
> and put them in exactly the same place in memory or else break ABI.
> And the other unfortunate thing is that we don't really have a
> mechanism for saying "okay, we're dropping support for the old way of
> doing things in 1.xx" -- in particular the current
> NPY_NO_DEPRECATED_API mechanism doesn't give us any way to detect and
> error out if someone tries to use an old version of the APIs, so ABI
> breaks still mean segfaults. I'm thinking that if/when we figure out
> how to implement the "sliding window" API/ABI idea that we talked
> about at SciPy, then that will give us a strategy for cleanly
> transitioning to a world with a maintainable API+ABI and it becomes
> worth sitting down and making up a set of setters/getters for the
> attributes that we want to make public in a maintainable way. But
> until then our only real options are either hard breaks or nothing, so
> unless we want to do a hard break there's not much point talking about
> it.
> 
> -n
> 
>> On Sun, Sep 20, 2015 at 9:13 PM, Nathaniel Smith  wrote:
>>> 
>>> Hi all,
>>> 
>>> Here's a first draft NEP for comments.
>>> 
>>> --
>>> 
>>> Synopsis
>>> 
>>> 
>>> Improving numpy's dtype system requires that ufunc loops start having
>>> access to details of the specific dtype instance they are acting on:
>>> e.g. an implementation of np.equal for strings needs access to the
>>> dtype object in order to know what "n" to pass to strncmp. Similar
>>> issues arise with variable length strings, missing values, categorical
>>> data, unit support, datetime with timezone support, etc. -- this is a
>>> major blocker for improving numpy.
>>> 
>>> Unfortunately, the current ufunc inner loop function signature makes
>>> it very difficult to provide this information. We might be able to
>>> wedge it in there, but it'd be ugly.
>>> 
>>> The other option would be to change the signature. What would happen
>>> if we did this? For most common uses of the C API/ABI, we could do
>>> this easily while maintaining backwards compatibility. But there are
>>> also some rarely-used parts  of the API/ABI that would be
>>> prohibitively difficult to preserve.
>>> 
>>> In addition, there are other potential changes to ufuncs on the
>>> horizon (e.g. extensions of gufuncs to allow them to be used more
>>> generally), and the current API exposure is so massive that any such
>>> changes will be difficult to make in a fully compatible way. This NEP
>>> thus considers 

Re: [Numpy-discussion] Governance model request

[Travis Oliphant]

On Mon, Sep 21, 2015 at 8:24 PM, Matthew Brett 
wrote:

> Hi Travis, and all,
>
> You might have seen I was advocating for having someone who takes
> final responsibility for the project, partly to get discussions
> unstuck, as you said.
>
> I agree with Chris, that at this stage, there is no-one who could be
> Benevolent Dictator for the project.  It seems to me that (nearly)
> everyone would agree that, if there is a leader, we would not want a
> person who dictates, but someone who is good at putting in the hard
> work and time it takes to hear all the arguments, and guide people to
> agreement, where that is possible.  If there is a leader, I think we
> should select them for those skills.
>

I don't really agree that someone couldn't be found.   I think the key is
that 1) the person would be able to make a decision about what needs to be
done if the community is stuck and 2) their is not really a "for life"
clause in that their would be a way to call for a replacement.


> For the proposal that you join the steering committee, I see two problems.
>
> The first is, that the members of the steering committee have to be
> the people who are keeping in touch with the day to day work of the
> project.   I am sure you would agree that, in the past, you have not
> had enough time for that.  Your recent emails about the new work you
> want to do, also imply that you may be too busy to get involved in the
> detailed discussions needed for getting the code merged.  In any case,
> I think you'd also agree that in the past you have hoped that you
> would have more time for numpy than you did.
>

I don't think this is true.   The steering committee would only be called
upon to unstick things and make decisions.At those times, it would not
take long to come up to speed on the issues and make a decision.


> The second problem is that you have a potential conflict of interest,
> in that it is possible for the needs of Continuum to conflict with the
> needs of numpy.


I think this is a red-herring and should not be an issue --- except for
obvious situations where I would have to not participate in a "vote" (such
as whether or not Continuum would be an institutional sponsor or something
like that).

Everyone has associations and affiliations and goals and plans that could
lead to conflicts of interest.This kind of requirement un-necessarily
limits the governance to only certain kinds of people who have only
"volunteer" time.   This is actually quite damaging as it limits the
ability for people to get paid to work on NumPy.This feels like a
world-view debate that is actually best left to a different mailing list.


>  I believe, from previous emails on this list, that
> you don't think that is very important, but I continue to disagree
> about that.

For example, see this interview with Linus Torvalds,
> where he talks about going out of his way to make sure that people can
> trust his decisions, despite the fact he is paid to work on Linux [1].
>

But this is in a BDFL role and not a steering committee role with 9 other
members. I don't think the situation compares or applies at all.   The
unwarranted fear this kind of reasoning can create in the mind of otherwise
reasonable people is unfortunate.


>
> In practice, the most obvious step that I can think of, is to defer
> the decision as to whether you are on the steering committee for six
> months.  I guess over that time you will be working with the other
> numpy developers more closely.
>

Fundamentally the "steering committee" is too big. I think the
committee should be much smaller (no more than 5 and really three is the
right size for what it is supposed to actually do).   I've had experience
with small committees and large committees.   Large councils make it very
difficult to move things forward.

If it is to remain that big, I think it needs people on it like me, Robert
Kern, or David Cournapeau who have a longer history with the project and
understand why some of the things are the way they are.   I still hear far
too many conversations that start from a lack of understanding of the early
conversations that led to why NumPy is the way it is.   This lack of
context is not helpful and potentially dangerous.   The advanced indexing
discussions happening right now and the renewed array-scalar discussions
for example are both examples of features that have had previous
discussions and have a long history of back and forth between various
people.


>
> I should say that Stefan and I are working on a governance proposal,
> in this case for scikit-image, where we split the governance into 1)
> the developers doing the work and making the day to day decisions and
> 2) the trustees, usually from other relevant projects, or
> no-longer-active developers, who make sure the project does not go off
> the rails.   I think you'd be an excellent and obvious trustee, in
> that model.
>

I like the trustee model too and think such 

Re: [Numpy-discussion] Governance model request

[Bryan Van de Ven]

> On Sep 21, 2015, at 9:42 PM, David Cournapeau  wrote:
> 
> 
> 
> On Mon, Sep 21, 2015 at 6:33 PM, Bryan Van de Ven  wrote:
> 
> > On Sep 21, 2015, at 9:24 PM, Matthew Brett  wrote:
> >
> > The second problem is that you have a potential conflict of interest,
> > in that it is possible for the needs of Continuum to conflict with the
> > needs of numpy.   I believe, from previous emails on this list, that
> > you don't think that is very important, but I continue to disagree
> > about that.   For example, see this interview with Linus Torvalds,
> 
> Can you actually spell out some particular, articulable, concrete points of 
> concern? Otherwise this just seems like idle, non-constructive speculation 
> (at best).
> 
> There is ample history of such things happening in OSS history, so I think 
> that's a fair concern, even if that has not happened for numpy yet.

Specific examples to support that claim would be appreciated. In particular, 
examples where an OSS project was corrupted (is that the word?) by a company 
specifically at the hand of the project's original creator would be especially 
relevant. 

Beyond that, what (even in a broad sense) is an example of a goal that 
"Continuum might need" that would conceivably do detriment to the NumPy 
community? That it be faster? Simpler to maintain? Easier to extend? Integrate 
better with more OS projects? Attract new active developers? Receive more 
financial support? Grow its user base even more?

And then, even if there is some sliver of daylight to be uncovered between 
Travis and the community, what is the current organizational mechanism by which 
a problematic contribution is forced into NumPy against the will of all other 
core developers?

Finally, is there any previous instance whatsoever that can be pointed to of 
Travis forcing some change into NumPy contrary to the interest of the 
community? If not, what actual basis is there to exclude him from a steering 
committee?

I obviously have a point of view; but my opinion here is my own, which is: this 
sort of pre-emptive impugning of someone's integrity is speculation upon 
speculation upon speculation 

Bryan 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Governance model request

[Bryan Van de Ven]

> On Sep 21, 2015, at 9:24 PM, Matthew Brett  wrote:
> 
> The second problem is that you have a potential conflict of interest,
> in that it is possible for the needs of Continuum to conflict with the
> needs of numpy.   I believe, from previous emails on this list, that
> you don't think that is very important, but I continue to disagree
> about that.   For example, see this interview with Linus Torvalds,

Can you actually spell out some particular, articulable, concrete points of 
concern? Otherwise this just seems like idle, non-constructive speculation (at 
best). 

Bryan 
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] ANN: Numpy 1.9.3 bugfix release

[Matthew Brett]

Hi,

We have just released numpy 1.9.3, a small bugfix release to fix a bug
on Python 3.5, as well as some build issues.

You likely only need to upgrade from 1.9.2 if you are on Python 3.5.

There are source and OSX wheels up on pypi.  We currently have no
plans to add 1.9.3 to the sourceforge site, because we cannot yet
build a Python 3.5 installer with the standard mingw compiler.

Issues fixed


* `#5866 `__: fix error finding
  Python headers when ``build_ext`` ``--include-dirs`` is set;
* `#6016 `__: fix ``np.loadtxt``
  error on Python 3.5 when reading from gzip files;
* `# `__: Replace deprecated
  options for ifort;
* `#6096 `__: remove /GL for VS2015
  in check_long_double_representation;
* `#6141 `__: enable Visual Studio
  2015 C99 features;
* `#6171 `__: revert C99 complex for
  MSVC14.

Cheers,

Matthew Brett - for the numpy developers.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Governance model request

[Matthew Brett]

Hi Travis, and all,

You might have seen I was advocating for having someone who takes
final responsibility for the project, partly to get discussions
unstuck, as you said.

I agree with Chris, that at this stage, there is no-one who could be
Benevolent Dictator for the project.  It seems to me that (nearly)
everyone would agree that, if there is a leader, we would not want a
person who dictates, but someone who is good at putting in the hard
work and time it takes to hear all the arguments, and guide people to
agreement, where that is possible.  If there is a leader, I think we
should select them for those skills.

For the proposal that you join the steering committee, I see two problems.

The first is, that the members of the steering committee have to be
the people who are keeping in touch with the day to day work of the
project.   I am sure you would agree that, in the past, you have not
had enough time for that.  Your recent emails about the new work you
want to do, also imply that you may be too busy to get involved in the
detailed discussions needed for getting the code merged.  In any case,
I think you'd also agree that in the past you have hoped that you
would have more time for numpy than you did.

The second problem is that you have a potential conflict of interest,
in that it is possible for the needs of Continuum to conflict with the
needs of numpy.   I believe, from previous emails on this list, that
you don't think that is very important, but I continue to disagree
about that.   For example, see this interview with Linus Torvalds,
where he talks about going out of his way to make sure that people can
trust his decisions, despite the fact he is paid to work on Linux [1].

In practice, the most obvious step that I can think of, is to defer
the decision as to whether you are on the steering committee for six
months.  I guess over that time you will be working with the other
numpy developers more closely.

I should say that Stefan and I are working on a governance proposal,
in this case for scikit-image, where we split the governance into 1)
the developers doing the work and making the day to day decisions and
2) the trustees, usually from other relevant projects, or
no-longer-active developers, who make sure the project does not go off
the rails.   I think you'd be an excellent and obvious trustee, in
that model.

Cheers,

Matthew


[1] http://www.bbc.com/news/technology-18419231
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] 1.10.0rc1 coming tomorrow, 22 Sept.

[Charles R Harris]

Hi All,

Just a heads up. The lack of reported problems in 1.10.0b1 has been
stunning.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way

[Jaime Fernández del Río]

We have the PyArrayObject vs PyArrayObject_fields definition in
ndarraytypes.h that is used to enforce access to the members through inline
functions rather than directly,  which seems to me like the right way to
go: don't leave stones unturned, hide everything and provide PyUFunc_NIN,
PyUFunc_NOUT and friends to handle those too.

On Sun, Sep 20, 2015 at 9:13 PM, Nathaniel Smith  wrote:

> Hi all,
>
> Here's a first draft NEP for comments.
>
> --
>
> Synopsis
> 
>
> Improving numpy's dtype system requires that ufunc loops start having
> access to details of the specific dtype instance they are acting on:
> e.g. an implementation of np.equal for strings needs access to the
> dtype object in order to know what "n" to pass to strncmp. Similar
> issues arise with variable length strings, missing values, categorical
> data, unit support, datetime with timezone support, etc. -- this is a
> major blocker for improving numpy.
>
> Unfortunately, the current ufunc inner loop function signature makes
> it very difficult to provide this information. We might be able to
> wedge it in there, but it'd be ugly.
>
> The other option would be to change the signature. What would happen
> if we did this? For most common uses of the C API/ABI, we could do
> this easily while maintaining backwards compatibility. But there are
> also some rarely-used parts  of the API/ABI that would be
> prohibitively difficult to preserve.
>
> In addition, there are other potential changes to ufuncs on the
> horizon (e.g. extensions of gufuncs to allow them to be used more
> generally), and the current API exposure is so massive that any such
> changes will be difficult to make in a fully compatible way. This NEP
> thus considers the possibility of closing down the ufunc API to a
> minimal, maintainable subset of the current API.
>
> To better understand the consequences of this potential change, I
> performed an exhaustive analysis of all the code on Github, Bitbucket,
> and Fedora, among others. The results make me highly confident that of
> all the publically available projects in the world, the only ones
> which touch the problematic parts of the ufunc API are: Numba,
> dynd-python, and `gulinalg `_
> (with the latter's exposure being trivial).
>
> Given this, I propose that for 1.11 we:
> 1) go ahead and hide/disable the problematic parts of the ABI/API,
> 2) coordinate with the known affected projects to minimize disruption
> to their users (which is made easier since they are all projects that
> are almost exclusively distributed via conda, which enforces strict
> NumPy ABI versioning),
> 3) publicize these changes widely so as to give any private code that
> might be affected a chance to speak up or adapt, and
> 4) leave the "ABI version tag" as it is, so as not to force rebuilds
> of the vast majority of projects that will be unaffected by these
> changes.
>
> This NEP defers the question of exactly what the improved API should
> be, since there's no point in trying to nail down the details until
> we've decided whether it's even possible to change.
>
>
> Details
> ===
>
> The problem
> ---
>
> Currently, a ufunc inner loop implementation is called via the
> following function prototype::
>
> typedef void (*PyUFuncGenericFunction)
> (char **args,
>  npy_intp *dimensions,
>  npy_intp *strides,
>  void *innerloopdata);
>
> Here ``args`` is an array of pointers to 1-d buffers of input/output
> data, ``dimensions`` is a pointer to the number of entries in these
> buffers, ``strides`` is an array of integers giving the strides for
> each input/output array, and ``innerloopdata`` is an arbitrary void*
> supplied by whoever registered the ufunc loop. (For gufuncs, extra
> shape and stride information about the core dimensions also gets
> packed into the ends of these arrays in a somewhat complicated way.)
>
> There are 4 key items that define a NumPy array: data, shape, strides,
> dtype. Notice that this function only gets access to 3 of them. Our
> goal is to fix that. For example, a better signature would be::
>
> typedef void (*PyUFuncGenericFunction_NEW)
> (char **data,
>  npy_intp *shapes,
>  npy_intp *strides,
>  PyArray_Descr *dtypes,   /* NEW */
>  void *innerloopdata);
>
> (In practice I suspect we might want to make some more changes as
> well, like upgrading gufunc core shape/strides to proper arguments
> instead of tacking it onto the existing arrays, and adding an "escape
> valve" void* reserved for future extensions. But working out such
> details is outside the scope of this NEP; the above will do for
> illustration.)
>
> The goal of this NEP is to clear the ground so that we can start
> supporting ufunc inner loops that take dtype arguments, and make other
> enhancements to ufunc functionality going 

Re: [Numpy-discussion] Governance model request

[Travis Oliphant]

I wrote my recommendations quickly before heading on a plane.I hope the
spirit of them was caught correctly.I also want to re-emphasize that I
completely understand that the Steering Council is not to be making
decisions that often and almost all activity will be similar to it is now
--- discussion, debate, proposals, and pull-requests --- that is a good
thing.

However, there is a need for leadership to help unstick things and move the
project forward from time to time because quite often doing *something* can
be better than trying to please everyone with a voice.   My concerns about
how to do this judgment have 2 major components:

1) The need for long-term consistency --- a one-year horizon on defining
this group is too short in my mind for a decades-old project like NumPy.
2) The group that helps unstick things needs to be small (1, 3, or 5 at the
most)

We could call this group the "adjudication group" rather than the "Steering
Council" as well.   I could see that having a formal method of changing
that "adjudication group" would be a good idea as well (and perhaps that
formal vote could be made by a vote of a group of active contributors.   In
that case, I would define active as having a time-window of 5 years instead
of just 1).

Thanks,

-Travis




On Mon, Sep 21, 2015 at 2:39 AM, Sebastian Berg 
wrote:

> On Mo, 2015-09-21 at 11:32 +0200, Sebastian Berg wrote:
> > On So, 2015-09-20 at 11:20 -0700, Travis Oliphant wrote:
> > > After long conversations at BIDS this weekend and after reading the
> > > entire governance document,  I realized that the steering council is
> > > very large and I don't agree with the mechanism by which it is
> > > chosen.
> > >
> >
> > Hmmm, well I never had the impression that the steering council would be
> > huge. But maybe you are right, and if it is, I could imagine something
> > like option 2, but vote based (could possibly dual use those in charge
> > of NumFOCUS relations, we had even discussed this possibility) which
> > would have final say if necessary (could mean that the contributers
> > definition could be broadened a bit).
> > However, I am not sure this is what you suggested, because for me it
> > should be a regular vote (if just because I am scared of having to make
> > the right pick). And while I will not block this if others agree, I am
> > currently not comfortable with either picking a BDFL (sorry guys :P) or
> > very fond of an oligarchy for live.
> >
> > Anyway, I still don't claim to have a good grasp on these things, but
> > without a vote, it seems a bit what Matthew warned about.
> >
> > One thing I could imagine is something like an "Advisory Board", without
> > (much) formal power. If we had a voted Steering Council, it could be the
> > former members + old time contributers which we would choose now. These
> > could be invited to meetings at the very least.
> >
> > Just my current, probably not well thought out thoughts on the matter.
> > But neither of your three options feel very obvious to me unfortunately.
> >
> > - Sebastian
> >
> >
> > > A one year time frame is pretty short on the context of a two decades
> > > old project and I believe the current council has too few people who
> > > have been around the community long enough to help unstuck difficult
> > > situations if that were necessary.
> > >
> > > I would recommend three possible adjustments to the steering council
> > > concept.
> > >
> > > 1 - define a BDFL for the council.  I would nominate chuck Harris
> > >
> > > 2 - limit the council to 3 people.  I would nominate chuck, nathaniel,
> > > and pauli.
> > >
> > > 3 - add me as a permanent member of the steering council.
> > >
>
> Though, maybe you should be in the steering council in any case even by
> the current rules. Maybe you were not too active for a while, but I
> doubt you will quite stop doing stuff on numpy soon
>
>
> > > Writing NumPy was a significant amount of work.  I have been working
> > > indirectly or directly in support of NumPy continously since I wrote
> > > it.  While I don't actively participate all the time, I still have a
> > > lot of knowledge, context, and experience in how NumPy is used, why it
> > > is the way it is, and how things could be better.  I also work with
> > > people directly who have and will contribute regularly.
> > >
> > > I am formally requesting that the steering council concept be adjusted
> > > in one of these three ways.
> > >
> > > Thanks,
> > >
> > > Travis
> > >
> > > ___
> > > NumPy-Discussion mailing list
> > > NumPy-Discussion@scipy.org
> > > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> ___
> NumPy-Discussion mailing list
> 

Re: [Numpy-discussion] Governance model request

[Sebastian Berg]

On Mo, 2015-09-21 at 11:32 +0200, Sebastian Berg wrote:
> On So, 2015-09-20 at 11:20 -0700, Travis Oliphant wrote:
> > After long conversations at BIDS this weekend and after reading the
> > entire governance document,  I realized that the steering council is
> > very large and I don't agree with the mechanism by which it is
> > chosen. 
> > 
> 
> Hmmm, well I never had the impression that the steering council would be
> huge. But maybe you are right, and if it is, I could imagine something
> like option 2, but vote based (could possibly dual use those in charge
> of NumFOCUS relations, we had even discussed this possibility) which
> would have final say if necessary (could mean that the contributers
> definition could be broadened a bit).
> However, I am not sure this is what you suggested, because for me it
> should be a regular vote (if just because I am scared of having to make
> the right pick). And while I will not block this if others agree, I am
> currently not comfortable with either picking a BDFL (sorry guys :P) or
> very fond of an oligarchy for live.
> 
> Anyway, I still don't claim to have a good grasp on these things, but
> without a vote, it seems a bit what Matthew warned about.
> 
> One thing I could imagine is something like an "Advisory Board", without
> (much) formal power. If we had a voted Steering Council, it could be the
> former members + old time contributers which we would choose now. These
> could be invited to meetings at the very least.
> 
> Just my current, probably not well thought out thoughts on the matter.
> But neither of your three options feel very obvious to me unfortunately.
> 
> - Sebastian
> 
> 
> > A one year time frame is pretty short on the context of a two decades
> > old project and I believe the current council has too few people who
> > have been around the community long enough to help unstuck difficult
> > situations if that were necessary. 
> > 
> > I would recommend three possible adjustments to the steering council
> > concept.  
> > 
> > 1 - define a BDFL for the council.  I would nominate chuck Harris
> > 
> > 2 - limit the council to 3 people.  I would nominate chuck, nathaniel,
> > and pauli.
> > 
> > 3 - add me as a permanent member of the steering council. 
> > 

Though, maybe you should be in the steering council in any case even by
the current rules. Maybe you were not too active for a while, but I
doubt you will quite stop doing stuff on numpy soon


> > Writing NumPy was a significant amount of work.  I have been working
> > indirectly or directly in support of NumPy continously since I wrote
> > it.  While I don't actively participate all the time, I still have a
> > lot of knowledge, context, and experience in how NumPy is used, why it
> > is the way it is, and how things could be better.  I also work with
> > people directly who have and will contribute regularly.
> > 
> > I am formally requesting that the steering council concept be adjusted
> > in one of these three ways.
> > 
> > Thanks,
> > 
> > Travis
> > 
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion



signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way

[Antoine Pitrou]

Hi Nathaniel,

On Sun, 20 Sep 2015 21:13:30 -0700
Nathaniel Smith  wrote:
> Given this, I propose that for 1.11 we:
> 1) go ahead and hide/disable the problematic parts of the ABI/API,
> 2) coordinate with the known affected projects to minimize disruption
> to their users (which is made easier since they are all projects that
> are almost exclusively distributed via conda, which enforces strict
> NumPy ABI versioning),
> 3) publicize these changes widely so as to give any private code that
> might be affected a chance to speak up or adapt, and
> 4) leave the "ABI version tag" as it is, so as not to force rebuilds
> of the vast majority of projects that will be unaffected by these
> changes.

Thanks for a detailed and clear explanation of the proposed changes.
As far as Numba is concerned, making changes is ok for us provided
Numpy provides APIs to do what we want.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Governance model request

[Chris Barker]

On Sun, Sep 20, 2015 at 11:20 AM, Travis Oliphant 
wrote:

> After long conversations at BIDS this weekend and after reading the entire
> governance document,  I realized that the steering council is very large
>
How large are we talking? I think there were 8 people named -- and I'm not
sure all 8 want to do it. But 5 or seems like a fine number -- I wonder if
defining the number is a good idea.

> A one year time frame is pretty short on the context of a two decades old
> project
>
I see:

"""
who have produced contributions that are substantial in
quality and quantity, and sustained over at least one year
"""

that could be longer, but it looks like it DOESN'T mean "have contributed
within the last year" -- so we could certainly, at this point, have a lot
of folks that have been around a long time that could qualify. A certain
guy named Travis come to mind...

and I believe the current council has too few people who have been around
> the community long enough to help unstuck difficult situations if that were
> necessary.
>
That's actually a bad sign right there -- the suggested group are the folks
that have actually been contributing lately -- too bad that apparently
isn't anyone that has been around a long time :-(

 also see:
"""
 This will
include but is not limited to code, code review, infrastructure work,
mailing list and chat participation, community help/building,
education and outreach, design work, etc.
"""

so it's not only people doing the actual coding ( A really  small number
:-( )

But I note that the the suggestion to "seed the  council" with "everyone
who has reviewed/merged a pull request since Jan 1, 2014" - I'm not sure
that gives us as diverse a council as we'd like -- but it is only the
seed...
>
> 1 - define a BDFL for the council.
>
Well, that would make a number of things easier -- but I think there was a
consensus that there simply was no obvious candidate -- and if the
candidate is not obvious, then they can't really be a BDFL.

> 2 - limit the council to 3 people.  I would nominate chuck, nathaniel, and
> pauli.
>
Good group -- but maybe 3 a bit too small.

> 3 - add me as a permanent member of the steering council.
>
I'd love to see you on the council, and there is no question of your
qualifications -- but I don't think anyone should be anymore permanent than
any other. According to the docs so far, the only thing anyone needs to do
to remain on the council is stay active -- and not piss everyone off so
much as to get a consensus from everyone else that you need to be kicked
off.

And the end of this -- I think the big question still at hand is how big
the council should be.

My experience with consensus suggests that it not be very big :-)

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Governance model request

[Sebastian Berg]

On So, 2015-09-20 at 11:20 -0700, Travis Oliphant wrote:
> After long conversations at BIDS this weekend and after reading the
> entire governance document,  I realized that the steering council is
> very large and I don't agree with the mechanism by which it is
> chosen. 
> 

Hmmm, well I never had the impression that the steering council would be
huge. But maybe you are right, and if it is, I could imagine something
like option 2, but vote based (could possibly dual use those in charge
of NumFOCUS relations, we had even discussed this possibility) which
would have final say if necessary (could mean that the contributers
definition could be broadened a bit).
However, I am not sure this is what you suggested, because for me it
should be a regular vote (if just because I am scared of having to make
the right pick). And while I will not block this if others agree, I am
currently not comfortable with either picking a BDFL (sorry guys :P) or
very fond of an oligarchy for live.

Anyway, I still don't claim to have a good grasp on these things, but
without a vote, it seems a bit what Matthew warned about.

One thing I could imagine is something like an "Advisory Board", without
(much) formal power. If we had a voted Steering Council, it could be the
former members + old time contributers which we would choose now. These
could be invited to meetings at the very least.

Just my current, probably not well thought out thoughts on the matter.
But neither of your three options feel very obvious to me unfortunately.

- Sebastian


> A one year time frame is pretty short on the context of a two decades
> old project and I believe the current council has too few people who
> have been around the community long enough to help unstuck difficult
> situations if that were necessary. 
> 
> I would recommend three possible adjustments to the steering council
> concept.  
> 
> 1 - define a BDFL for the council.  I would nominate chuck Harris
> 
> 2 - limit the council to 3 people.  I would nominate chuck, nathaniel,
> and pauli.
> 
> 3 - add me as a permanent member of the steering council. 
> 
> Writing NumPy was a significant amount of work.  I have been working
> indirectly or directly in support of NumPy continously since I wrote
> it.  While I don't actively participate all the time, I still have a
> lot of knowledge, context, and experience in how NumPy is used, why it
> is the way it is, and how things could be better.  I also work with
> people directly who have and will contribute regularly.
> 
> I am formally requesting that the steering council concept be adjusted
> in one of these three ways.
> 
> Thanks,
> 
> Travis
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion



signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion