Re: [Numpy-discussion] 1.10.0rc1
Ok, I tested matplotlib master against numpy master, and there were no errors. I did get a bunch of new deprecation warnings though such as: /nas/home/broot/centos6/lib/python2.7/site-packages/matplotlib-1.5.dev1-py2.7-linux-x86_64.egg/matplotlib/colorbar.py:539: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 5 but corresponding boolean dimension is 3 colors = np.asarray(colors)[igood] The message isn't exactly clear. I suspect the problem is a shape mismatch, like colors is 5x3, and igood is just 3 for some reason. Could somebody shine some light on this, please? Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1.10.0rc1
On Thu, Aug 27, 2015 at 7:52 AM, Benjamin Root ben.v.r...@gmail.com wrote: Ok, I tested matplotlib master against numpy master, and there were no errors. I did get a bunch of new deprecation warnings though such as: /nas/home/broot/centos6/lib/python2.7/site-packages/matplotlib-1.5.dev1-py2.7-linux-x86_64.egg/matplotlib/colorbar.py:539: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 5 but corresponding boolean dimension is 3 colors = np.asarray(colors)[igood] The message isn't exactly clear. I suspect the problem is a shape mismatch, like colors is 5x3, and igood is just 3 for some reason. Could somebody shine some light on this, please? IIRC, Boolean indexing would fill out the dimension, i.e., len 3 would be expanded to len 5 in this case. That behavior is deprecated. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Thu, Aug 27, 2015 at 8:57 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Aug 27, 2015 at 12:11 PM, Sebastian Berg sebast...@sipsolutions.net wrote: On Do, 2015-08-27 at 10:45 +0100, Matthew Brett wrote: Hi, On Thu, Aug 27, 2015 at 10:35 AM, Bryan Van de Ven bry...@continuum.io wrote: On Aug 27, 2015, at 10:22 AM, Matthew Brett matthew.br...@gmail.com wrote: In the case of the 'core' model, we have some compelling testimony from someone with a great deal of experience: Much of this early structure (CVS, web site, cabal [core group], etc.) was copied verbatim by other open source (this term not being in wide use yet) projects -- even the form of the project name and the term core. This later became a kind of standard template for starting up an open source project. [...] I'm sorry to say that I helped create this problem, and that most of the projects which modeled themselves after NetBSD (probably due to its high popularity in 1993 and 1994) have suffered similar problems. FreeBSD and XFree86, for example, have both forked successor projects (Dragonfly and X.org) for very similar reasons. Who goes on to propose: 7) The core group must be replaced with people who are actually competent and dedicated enough to review proposals, accept feedback, and make good decisions. More to the point, though, the core group must only act when *needed* -- most technical decisions should be left to the community to hash out; it must not preempt the community from developing better solutions. (This is how the core group worked during most of the project's growth period.) Sure. I think it's reasonable to give high weight to Hannum's assessment of the failure of the core group, but less weight to his proposal for a replacement, because at the time, I don't believe he was in a good position to assess whether his (apparent) alternative would run into the same trouble. It's always tempting to blame the people rather than the system, but in this case, I strongly suspect that it was the system that was fundamentally flawed, therefore either promoting the wrong people or putting otherwise competent people into situations where they are no longer getting useful feedback. Maybe so. I do not know much at all about these models, but I am not sure how much applies here to numpy. Isn't at least FreeBSD a magnitude larger then numpy? It seems to me that numpy suffers from the same risks of poor accountability, stagnation and conservatism that larger projects do. Is there a reason that would not be the case? We do need to have some formality about how to give out commit rights, and do final decision when all else fails. Yes, sure, something formal is probably but not certainly better than nothing, depending on what the 'something formal' is. One thing I do not know is how a community vote could work at all, considering I do not even know how to count its members. Votes and presidents make sense to me for large projects with hundrets of developers on different corners (think of the gnome foundation, debian probably) [1]. The 'president' idea is to get at the problem of lack of accountability, along with selection for leadership skill rather than coding ability. It's trying to get at the advantages of the BDFL model in our situation where there is no obvious BDFL.For the me the problem is that, at the moment, if the formal or informal governing body makes a bad decision, then no member will feel responsible for that decision or its consequences. That tends to lead to an atmosphere of - oh well, what could we do, X wouldn't agree to A and Y wouldn't agree to B so we're stuck. It seems to me we need a system such that whoever is in charge feels so strongly that it is their job to make numpy as good as possible, that they will take whatever difficult or sensitive decisions are necessary to make that happen. On the other hand the 'core' system seems to function on a model of mutual deference and personal loyalty that I believe is destructive of good management. I don't really see a problem with codifying the status quo. It might become necessary to have something like an administrative director if numpy becomes a more formal organization with funding, but for the development of the project I don't see any need for a president. If there is no obvious BDFL, then I guess there is also no obvious president. (I would vote for Ralf as president of everything, but I don't think he's available.) As the current debate shows it's possible to have a public discussion about the direction of the project without having to delegate providing a vision to a president. Given the current pattern all critical issues end up in a public debate on the mailing list. What numpy (and scipy) need is to have someone as a tie breaker to make any
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Aug 27, 2015, at 1:57 PM, Matthew Brett matthew.br...@gmail.com wrote: The 'president' idea ...seems to be predicated on a steady stream of people who: actually want job, don't mind campaigning, are willing to accept any and all blame, and have the technical experience to make final decisions. As others have pointed out the active developer community for NumPy is not measured in the hundreds (or even the tens, really). So: what is your proposed recourse if you hold an election and no-one shows up to run? Bryan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
Hi, On Thu, Aug 27, 2015 at 12:11 PM, Sebastian Berg sebast...@sipsolutions.net wrote: On Do, 2015-08-27 at 10:45 +0100, Matthew Brett wrote: Hi, On Thu, Aug 27, 2015 at 10:35 AM, Bryan Van de Ven bry...@continuum.io wrote: On Aug 27, 2015, at 10:22 AM, Matthew Brett matthew.br...@gmail.com wrote: In the case of the 'core' model, we have some compelling testimony from someone with a great deal of experience: Much of this early structure (CVS, web site, cabal [core group], etc.) was copied verbatim by other open source (this term not being in wide use yet) projects -- even the form of the project name and the term core. This later became a kind of standard template for starting up an open source project. [...] I'm sorry to say that I helped create this problem, and that most of the projects which modeled themselves after NetBSD (probably due to its high popularity in 1993 and 1994) have suffered similar problems. FreeBSD and XFree86, for example, have both forked successor projects (Dragonfly and X.org) for very similar reasons. Who goes on to propose: 7) The core group must be replaced with people who are actually competent and dedicated enough to review proposals, accept feedback, and make good decisions. More to the point, though, the core group must only act when *needed* -- most technical decisions should be left to the community to hash out; it must not preempt the community from developing better solutions. (This is how the core group worked during most of the project's growth period.) Sure. I think it's reasonable to give high weight to Hannum's assessment of the failure of the core group, but less weight to his proposal for a replacement, because at the time, I don't believe he was in a good position to assess whether his (apparent) alternative would run into the same trouble. It's always tempting to blame the people rather than the system, but in this case, I strongly suspect that it was the system that was fundamentally flawed, therefore either promoting the wrong people or putting otherwise competent people into situations where they are no longer getting useful feedback. Maybe so. I do not know much at all about these models, but I am not sure how much applies here to numpy. Isn't at least FreeBSD a magnitude larger then numpy? It seems to me that numpy suffers from the same risks of poor accountability, stagnation and conservatism that larger projects do. Is there a reason that would not be the case? We do need to have some formality about how to give out commit rights, and do final decision when all else fails. Yes, sure, something formal is probably but not certainly better than nothing, depending on what the 'something formal' is. One thing I do not know is how a community vote could work at all, considering I do not even know how to count its members. Votes and presidents make sense to me for large projects with hundrets of developers on different corners (think of the gnome foundation, debian probably) [1]. The 'president' idea is to get at the problem of lack of accountability, along with selection for leadership skill rather than coding ability. It's trying to get at the advantages of the BDFL model in our situation where there is no obvious BDFL.For the me the problem is that, at the moment, if the formal or informal governing body makes a bad decision, then no member will feel responsible for that decision or its consequences. That tends to lead to an atmosphere of - oh well, what could we do, X wouldn't agree to A and Y wouldn't agree to B so we're stuck. It seems to me we need a system such that whoever is in charge feels so strongly that it is their job to make numpy as good as possible, that they will take whatever difficult or sensitive decisions are necessary to make that happen. On the other hand the 'core' system seems to function on a model of mutual deference and personal loyalty that I believe is destructive of good management. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Do, 2015-08-27 at 17:22 +0100, Matthew Brett wrote: Hi On Thu, Aug 27, 2015 at 5:11 PM, josef.p...@gmail.com wrote: On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Aug 27, 2015 at 3:34 PM, josef.p...@gmail.com wrote: [snip] I don't really see a problem with codifying the status quo. That's an excellent point.If we believe that the current situation is the best possible, both now and in the future, then codifying the status quo is an excellent idea. So, we should probably first start by asking ourselves: * what numpy is doing well; * what numpy could do better; and then ask, is there some way we could make it more likely we will improve over time. [snip] As the current debate shows it's possible to have a public discussion about the direction of the project without having to delegate providing a vision to a president. The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for the performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1] Still doesn't sound like the need for a president to me the person who holds themselves responsible for the performance of the project sounds more like the role of the core group (adding plural to persons) to me, and cannot be pushed of to an official president. Except that, in the past, having multiple people taking decisions has led to the situation where no-one feels themselves accountable for the result, hence this situation tends to lead to stagnation. Frankly, I am failing to see the direction of these arguments. One thing to remember, that a core group is much like a BDFL/president with multiple personalities ;), and a core group is not a fixed Oligarchy. Anyone able and willing should be in it and the governance document is clear about that I think (of course nothing is perfect, but we can try). There is the thing of how. I simply fail to see how the president can even be defined considering the size of the numpy development team (say 10, most of whom are busy with other things most of the time). Also, I fail to see how the president would be any more useful then the agreement of some tasks being handled by some people who are enthusiastic about them (note those do not even have to be in the core group for starters, though they should become part of it quickly). This is a community effort and I am starting to feel that the ideas you are giving are from a different management/company context. The goal of the governance is to show how and hopefully make it easy for *anyone* to provide vision. We do not need a manager who decides how to focus allocate resources, instead we must tell everyone that we are happy about any help we can get, and that anyone can pick up a topic they are enthusiastic about and drive numpy ahead. And considering accountability, that help may well amount in saying: Do NOT do this. A president willing to run for such an election, should have a specific vision? Why should they be special to implement it? Note this is also the case in BDFL organizations. If you have a vision to improve python, it does not really matter if you happen to be Guido. You write a PEP and, if people like it, it will be implemented. At the same time we *must* have a well defined form of governance also for organizational things. Right now we cannot even decide on putting someone in charge of overseeing our NumFOCUS donations. NumPy could not even spend its own money! Sorry, getting way too long :( - Sebastian Nathaniel to push and organize the discussion, Chuck for continuity, and several core developers for detailed ideas and implementation, and a large number of contributors. (stylized roles) and noisy mailing list for feedback and discussion. Given the size of the numpy development group, numpy needs individuals for the vision and to push things not a president, vice-presidents and assistant vice-presidents, IMO. Yes, if the roles were honorary and administrative, they would not be useful. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi On Thu, Aug 27, 2015 at 5:11 PM, josef.p...@gmail.com wrote: On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Aug 27, 2015 at 3:34 PM, josef.p...@gmail.com wrote: [snip] I don't really see a problem with codifying the status quo. That's an excellent point.If we believe that the current situation is the best possible, both now and in the future, then codifying the status quo is an excellent idea. So, we should probably first start by asking ourselves: * what numpy is doing well; * what numpy could do better; and then ask, is there some way we could make it more likely we will improve over time. [snip] As the current debate shows it's possible to have a public discussion about the direction of the project without having to delegate providing a vision to a president. The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for the performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1] Still doesn't sound like the need for a president to me the person who holds themselves responsible for the performance of the project sounds more like the role of the core group (adding plural to persons) to me, and cannot be pushed of to an official president. Except that, in the past, having multiple people taking decisions has led to the situation where no-one feels themselves accountable for the result, hence this situation tends to lead to stagnation. Is there any evidence for this? First, it's several individuals taking joint decisions, jointly agree or not object (LGTM) to merging PRs, it's still a joint decision and accountability is not exclusive. The PR review process makes decisions much more into a joint decision process than it was with isolated SVN commits. (*) Second, if there are separated decisions, then it could also lead to excess change. All these enthusiastic new developers bringing in whatever they (and the local chief) like, and nobody to stop them. In either case, the developer, or local chief, has to deal with the consequences. You merged this PR, now fix it. or Why are you holding up this PR? I can merge it. (*) Even though I'm not a scipy developer anymore, I still feel partially responsible for it, I'm still reviewing some PRs and comment on them, sometimes as cheerleader in favor of something or sometimes pointing out problems, or just checking that it makes sense, and always with an eye on what downstream impact it might have. Another thought: Having an accountable president might actually reduce the feeling of accountability and responsibility of individual developers, so the neteffect is negative. The president is responsible for this (even though he doesn't have enough time), so I can skip part of this review. Josef Nathaniel to push and organize the discussion, Chuck for continuity, and several core developers for detailed ideas and implementation, and a large number of contributors. (stylized roles) and noisy mailing list for feedback and discussion. Given the size of the numpy development group, numpy needs individuals for the vision and to push things not a president, vice-presidents and assistant vice-presidents, IMO. Yes, if the roles were honorary and administrative, they would not be useful. I'm not sure what you mean here. Given that it's all volunteer work, any president wouldn't have any hard tools. Josef Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1.10.0rc1
Ok, I just wanted to make sure I understood the issue before going bug hunting. Chances are, it has been a bug on our end for a while now. Just to make sure, is the following valid? arr = np.zeros((5, 3)) ind = np.array([True, True, True, False, True]) arr[ind] # gives a 4x3 result Running that at the REPL doesn't produce a warning, so i am guessing that it is valid. Ben Root On Thu, Aug 27, 2015 at 10:44 AM, Sebastian Berg sebast...@sipsolutions.net wrote: On Do, 2015-08-27 at 08:04 -0600, Charles R Harris wrote: On Thu, Aug 27, 2015 at 7:52 AM, Benjamin Root ben.v.r...@gmail.com wrote: Ok, I tested matplotlib master against numpy master, and there were no errors. I did get a bunch of new deprecation warnings though such as: /nas/home/broot/centos6/lib/python2.7/site-packages/matplotlib-1.5.dev1-py2.7-linux-x86_64.egg/matplotlib/colorbar.py:539: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 5 but corresponding boolean dimension is 3 colors = np.asarray(colors)[igood] The message isn't exactly clear. I suspect the problem is a shape mismatch, like colors is 5x3, and igood is just 3 for some reason. Could somebody shine some light on this, please? IIRC, Boolean indexing would fill out the dimension, i.e., len 3 would be expanded to len 5 in this case. That behavior is deprecated. Yes, this is exactly the case, you have something like: arr = np.zeros((5, 3)) ind = np.array([True, False, False]) arr[ind, :] and numpy nowadays thinks that such code is likely a bug (when the ind is shorter than arr it is somewhat OK, the other way around gets more creepy). If you have an idea of how to make the error message clearer, or objections to the change, I am happy to hear it! - Sebastian Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1.10.0rc1
On Do, 2015-08-27 at 11:15 -0400, Benjamin Root wrote: Ok, I just wanted to make sure I understood the issue before going bug hunting. Chances are, it has been a bug on our end for a while now. Just to make sure, is the following valid? arr = np.zeros((5, 3)) ind = np.array([True, True, True, False, True]) arr[ind] # gives a 4x3 result Running that at the REPL doesn't produce a warning, so i am guessing that it is valid. Sure, that is perfect (you can add the slice and write `arr[ind, :]` to make it a bit more clear if you like I guess). - Sebastian Ben Root On Thu, Aug 27, 2015 at 10:44 AM, Sebastian Berg sebast...@sipsolutions.net wrote: On Do, 2015-08-27 at 08:04 -0600, Charles R Harris wrote: On Thu, Aug 27, 2015 at 7:52 AM, Benjamin Root ben.v.r...@gmail.com wrote: Ok, I tested matplotlib master against numpy master, and there were no errors. I did get a bunch of new deprecation warnings though such as: /nas/home/broot/centos6/lib/python2.7/site-packages/matplotlib-1.5.dev1-py2.7-linux-x86_64.egg/matplotlib/colorbar.py:539: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 5 but corresponding boolean dimension is 3 colors = np.asarray(colors)[igood] The message isn't exactly clear. I suspect the problem is a shape mismatch, like colors is 5x3, and igood is just 3 for some reason. Could somebody shine some light on this, please? IIRC, Boolean indexing would fill out the dimension, i.e., len 3 would be expanded to len 5 in this case. That behavior is deprecated. Yes, this is exactly the case, you have something like: arr = np.zeros((5, 3)) ind = np.array([True, False, False]) arr[ind, :] and numpy nowadays thinks that such code is likely a bug (when the ind is shorter than arr it is somewhat OK, the other way around gets more creepy). If you have an idea of how to make the error message clearer, or objections to the change, I am happy to hear it! - Sebastian Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1.10.0rc1
The reason why we don't have that extra slice is because we may not know ahead of time that we are dealing with a 2D array. It could be a 1D array. I guess we could use ellipses, but I wanted to make sure that the numpy devs consider the above to be perfectly valid semantics because it is entrenched in our codebase. Ben Root On Thu, Aug 27, 2015 at 11:33 AM, Sebastian Berg sebast...@sipsolutions.net wrote: On Do, 2015-08-27 at 11:15 -0400, Benjamin Root wrote: Ok, I just wanted to make sure I understood the issue before going bug hunting. Chances are, it has been a bug on our end for a while now. Just to make sure, is the following valid? arr = np.zeros((5, 3)) ind = np.array([True, True, True, False, True]) arr[ind] # gives a 4x3 result Running that at the REPL doesn't produce a warning, so i am guessing that it is valid. Sure, that is perfect (you can add the slice and write `arr[ind, :]` to make it a bit more clear if you like I guess). - Sebastian Ben Root On Thu, Aug 27, 2015 at 10:44 AM, Sebastian Berg sebast...@sipsolutions.net wrote: On Do, 2015-08-27 at 08:04 -0600, Charles R Harris wrote: On Thu, Aug 27, 2015 at 7:52 AM, Benjamin Root ben.v.r...@gmail.com wrote: Ok, I tested matplotlib master against numpy master, and there were no errors. I did get a bunch of new deprecation warnings though such as: /nas/home/broot/centos6/lib/python2.7/site-packages/matplotlib-1.5.dev1-py2.7-linux-x86_64.egg/matplotlib/colorbar.py:539: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 5 but corresponding boolean dimension is 3 colors = np.asarray(colors)[igood] The message isn't exactly clear. I suspect the problem is a shape mismatch, like colors is 5x3, and igood is just 3 for some reason. Could somebody shine some light on this, please? IIRC, Boolean indexing would fill out the dimension, i.e., len 3 would be expanded to len 5 in this case. That behavior is deprecated. Yes, this is exactly the case, you have something like: arr = np.zeros((5, 3)) ind = np.array([True, False, False]) arr[ind, :] and numpy nowadays thinks that such code is likely a bug (when the ind is shorter than arr it is somewhat OK, the other way around gets more creepy). If you have an idea of how to make the error message clearer, or objections to the change, I am happy to hear it! - Sebastian Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
Hi On Thu, Aug 27, 2015 at 5:11 PM, josef.p...@gmail.com wrote: On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Aug 27, 2015 at 3:34 PM, josef.p...@gmail.com wrote: [snip] I don't really see a problem with codifying the status quo. That's an excellent point.If we believe that the current situation is the best possible, both now and in the future, then codifying the status quo is an excellent idea. So, we should probably first start by asking ourselves: * what numpy is doing well; * what numpy could do better; and then ask, is there some way we could make it more likely we will improve over time. [snip] As the current debate shows it's possible to have a public discussion about the direction of the project without having to delegate providing a vision to a president. The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for the performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1] Still doesn't sound like the need for a president to me the person who holds themselves responsible for the performance of the project sounds more like the role of the core group (adding plural to persons) to me, and cannot be pushed of to an official president. Except that, in the past, having multiple people taking decisions has led to the situation where no-one feels themselves accountable for the result, hence this situation tends to lead to stagnation. Nathaniel to push and organize the discussion, Chuck for continuity, and several core developers for detailed ideas and implementation, and a large number of contributors. (stylized roles) and noisy mailing list for feedback and discussion. Given the size of the numpy development group, numpy needs individuals for the vision and to push things not a president, vice-presidents and assistant vice-presidents, IMO. Yes, if the roles were honorary and administrative, they would not be useful. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015
On Wed, Aug 26, 2015 at 10:06 AM, Travis Oliphant tra...@continuum.io wrote: On Wed, Aug 26, 2015 at 1:41 AM, Nathaniel Smith n...@pobox.com wrote: Hi Travis, Thanks for taking the time to write up your thoughts! I have many thoughts in return, but I will try to restrict myself to two main ones :-). 1) On the question of whether work should be directed towards improving NumPy-as-it-is or instead towards a compatibility-breaking replacement: There's plenty of room for debate about whether it's better engineering practice to try and evolve an existing system in place versus starting over, and I guess we have some fundamental disagreements there, but I actually think this debate is a distraction -- we can agree to disagree, because in fact we have to try both. Yes, on this we agree. I think NumPy can improve *and* we can have new innovative array objects. I don't disagree about that. At a practical level: NumPy *is* going to continue to evolve, because it has users and people interested in evolving it; similarly, dynd and other alternatives libraries will also continue to evolve, because they also have people interested in doing it. And at a normative level, this is a good thing! If NumPy and dynd both get better, than that's awesome: the worst case is that NumPy adds the new features that we talked about at the meeting, and dynd simultaneously becomes so awesome that everyone wants to switch to it, and the result of this would be... that those NumPy features are exactly the ones that will make the transition to dynd easier. Or if some part of that plan goes wrong, then well, NumPy will still be there as a fallback, and in the mean time we've actually fixed the major pain points our users are begging us to fix. You seem to be urging us all to make a double-or-nothing wager that your extremely ambitious plans will all work out, with the entire numerical Python ecosystem as the stakes. I think this ambition is awesome, but maybe it'd be wise to hedge our bets a bit? You are mis-characterizing my view. I think NumPy can evolve (though I would personally rather see a bigger change to the underlying system like I outlined before).But, I don't believe it can even evolve easily in the direction needed without breaking ABI and that insisting on not breaking it or even putting too much effort into not breaking it will continue to create less-optimal solutions that are harder to maintain and do not take advantage of knowledge this community now has. I'm also very concerned that 'evolving' NumPy will create a situation where there are regular semantic and subtle API changes that will cause NumPy to be less stable for it's user-base.I've watched this happen. This at a time that people are already looking around for new and different approaches anyway. 2) You really emphasize this idea of an ABI-breaking (but not API-breaking) release, and I think this must indicate some basic gap in how we're looking at things. Where I'm getting stuck here is that... I actually can't think of anything important that we can't do now, but could if we were allowed to break ABI compatibility. The kinds of things that break ABI but keep API are like... rearranging what order the fields in a struct fall in, or changing the numeric value of opaque constants like NPY_ARRAY_WRITEABLE. The biggest win I can think of is that we could save a few bytes per array by arranging the fields inside the ndarray struct more optimally, but that's hardly a feature to hang a 2.0 on. You seem to have a vision of this ABI-breaking release as being something very different from that, and I'm not clear on what this vision is. We already broke the ABI with date-time changes --- it's still broken for a certain percentage of users last I checked.So, part of my disagreement is that we've tried this and it didn't work --- even though smart people thought it would.I've had to deal with this personally and I'm not enthusiastic about having to deal with this for the next 5 years because of even more attempts to make changes while not breaking the ABI. I think the group is more careful now --- but I still think the API is broad enough and uses of NumPy deep enough that the effort involved in trying not to break the ABI is just not worth the effort (because it's a non-feature today).Adding new dtypes without breaking the ABI is tricky (and to do it without breaking the ABI is ugly). I also continue to believe that putting out a new ABI-breaking NumPy will allow re-compiling *once* (with some porting changes needed) and not subtle breakages requiring code-changes every time a release is made.If subtle changes aren't made, then the new features won't come. Right now, I'd rather have stability from NumPy than new features. New features can come from other libraries. One specific change that could easily be made in NumPy 2.0 (the current code
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Aug 27, 2015 at 3:34 PM, josef.p...@gmail.com wrote: [snip] I don't really see a problem with codifying the status quo. That's an excellent point.If we believe that the current situation is the best possible, both now and in the future, then codifying the status quo is an excellent idea. So, we should probably first start by asking ourselves: * what numpy is doing well; * what numpy could do better; and then ask, is there some way we could make it more likely we will improve over time. [snip] As the current debate shows it's possible to have a public discussion about the direction of the project without having to delegate providing a vision to a president. The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for the performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1] Still doesn't sound like the need for a president to me the person who holds themselves responsible for the performance of the project sounds more like the role of the core group (adding plural to persons) to me, and cannot be pushed of to an official president. Nathaniel to push and organize the discussion, Chuck for continuity, and several core developers for detailed ideas and implementation, and a large number of contributors. (stylized roles) and noisy mailing list for feedback and discussion. Given the size of the numpy development group, numpy needs individuals for the vision and to push things not a president, vice-presidents and assistant vice-presidents, IMO. (Given the importance of numpy itself, there should be enough remedies if the core group ever gets `out of touch` with the very large user base.) Josef Given the current pattern all critical issues end up in a public debate on the mailing list. What numpy (and scipy) need is to have someone as a tie breaker to make any final decisions if there is no clear consensus, if there is no BDFL for it, then having the core group making those decisions looks appropriate to me. On the other hand the 'core' system seems to function on a model of mutual deference and personal loyalty that I believe is destructive of good management. That sounds actually like a good basis for team work to me. Also that has mutual in it instead of just deferring and being loyal to a president. Since I know scipy development much better: scipy has made a huge progress in the last 5 or 6 years since I've been following it, both in terms of code, in terms of development workflow, and in the number of developers. (When I started, I was essentially alone in scipy.stats, now there are 3 to 5 core developers that at least partially work on it, everything goes through PRs with public discussion and with critical issues additionally raised on the mailing list.) Ralf and Pauli are the defacto BDFLs for scipy overall, but all decisions in recent years have been without a fight, but not without lots of arguments, and, given the size and breadth of scipy, there are field experts (although not enough of those) to help in the discussion. I agree entirely, I think scipy is a good example where stability and clarity of leadership has made a huge difference to the health of the project. Cheers, Matthew [1] https://hbr.org/2005/07/level-5-leadership-the-triumph-of-humility-and-fierce-resolve ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
Hi, On Thu, Aug 27, 2015 at 3:34 PM, josef.p...@gmail.com wrote: [snip] I don't really see a problem with codifying the status quo. That's an excellent point.If we believe that the current situation is the best possible, both now and in the future, then codifying the status quo is an excellent idea. So, we should probably first start by asking ourselves: * what numpy is doing well; * what numpy could do better; and then ask, is there some way we could make it more likely we will improve over time. [snip] As the current debate shows it's possible to have a public discussion about the direction of the project without having to delegate providing a vision to a president. The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for the performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1] Given the current pattern all critical issues end up in a public debate on the mailing list. What numpy (and scipy) need is to have someone as a tie breaker to make any final decisions if there is no clear consensus, if there is no BDFL for it, then having the core group making those decisions looks appropriate to me. On the other hand the 'core' system seems to function on a model of mutual deference and personal loyalty that I believe is destructive of good management. That sounds actually like a good basis for team work to me. Also that has mutual in it instead of just deferring and being loyal to a president. Since I know scipy development much better: scipy has made a huge progress in the last 5 or 6 years since I've been following it, both in terms of code, in terms of development workflow, and in the number of developers. (When I started, I was essentially alone in scipy.stats, now there are 3 to 5 core developers that at least partially work on it, everything goes through PRs with public discussion and with critical issues additionally raised on the mailing list.) Ralf and Pauli are the defacto BDFLs for scipy overall, but all decisions in recent years have been without a fight, but not without lots of arguments, and, given the size and breadth of scipy, there are field experts (although not enough of those) to help in the discussion. I agree entirely, I think scipy is a good example where stability and clarity of leadership has made a huge difference to the health of the project. Cheers, Matthew [1] https://hbr.org/2005/07/level-5-leadership-the-triumph-of-humility-and-fierce-resolve ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
Hi, On Thu, Aug 27, 2015 at 3:43 PM, Bryan Van de Ven bry...@continuum.io wrote: On Aug 27, 2015, at 1:57 PM, Matthew Brett matthew.br...@gmail.com wrote: The 'president' idea ...seems to be predicated on a steady stream of people who: actually want job, don't mind campaigning, are willing to accept any and all blame, and have the technical experience to make final decisions. As others have pointed out the active developer community for NumPy is not measured in the hundreds (or even the tens, really). So: what is your proposed recourse if you hold an election and no-one shows up to run? That seems to me a soluble problem, if there's agreement that the president idea is a sensible one. One very simple idea would be to revert to a 'core' system for the term for which there were no candidates. On the other hand, I suspect that there are people who care enough about numpy that they are prepared to step up and take the blame if things go wrong. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Thu, Aug 27, 2015 at 2:06 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Aug 27, 2015 at 6:23 PM, josef.p...@gmail.com wrote: On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi On Thu, Aug 27, 2015 at 5:11 PM, josef.p...@gmail.com wrote: On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Aug 27, 2015 at 3:34 PM, josef.p...@gmail.com wrote: [snip] I don't really see a problem with codifying the status quo. That's an excellent point.If we believe that the current situation is the best possible, both now and in the future, then codifying the status quo is an excellent idea. So, we should probably first start by asking ourselves: * what numpy is doing well; * what numpy could do better; and then ask, is there some way we could make it more likely we will improve over time. [snip] As the current debate shows it's possible to have a public discussion about the direction of the project without having to delegate providing a vision to a president. The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for the performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1] Still doesn't sound like the need for a president to me the person who holds themselves responsible for the performance of the project sounds more like the role of the core group (adding plural to persons) to me, and cannot be pushed of to an official president. Except that, in the past, having multiple people taking decisions has led to the situation where no-one feels themselves accountable for the result, hence this situation tends to lead to stagnation. Is there any evidence for this? Oh - dear - that's the key point, but I'm obviously not making it clearly enough. Yes there is, and that was the evidence I was pointing to before. If you mean the XFree and NetBSD cases, then I don't see any similarity to the numpy or scipy development pattern. If I would draw any conclusion, then maybe that NetBSD hat too much of formal governance structures and not enough informal governance. It would be difficult to take over the government if there is no government. just one aside *No desire to recruit new users * We are on a mission to take over the world (*). And forks of numpy like pandas turn out to be mostly complementary and increase the userbase of numpy. (R and Python are in friendly, or sometimes unfriendly, competition, but, AFAICS, we are both gaining users because of the others' presence. It's not a zero sum game in this case.) (*) But that's not in the mission statement. But anyway - Sebastian is right - this discussion isn't going anywhere useful. So - let's step back. In thinking about governance, we first need to ask what we want to achieve. This includes considering the risks ahead for the project. So, in the spirit of fruitful discussion, can I ask what y'all consider to be the current problems with working on numpy (other than the technical ones). What is numpy doing well, and what is it doing badly? What risks do we have to plan for in the future? I thought that was implicit or explicit in the other thread. Josef Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On 2015-08-27 11:06:10, Matthew Brett matthew.br...@gmail.com wrote: So, in the spirit of fruitful discussion, can I ask what y'all consider to be the current problems with working on numpy (other than the technical ones). What is numpy doing well, and what is it doing badly? What risks do we have to plan for in the future? It looks to me as though the team is doing an excellent job of maintaining NumPy. The growth of the project has stagnated somewhat for numerous reasons---and a lack of ideas on the table is not one of them, rather whether / how to take them forward. The question, and I think what you also highlighted in the earlier part of this discussion, is: how to decide on which vision to adopt, and who takes responsibility for making that happen? Are the two models proposed thus far so different, or can they be merged in a way that makes sense? E.g., can we work as a community to rally behind a vision as set out by one person, and then repeat that process to focus on another a year later? Think of it as the iterative development equivalent of governance. This may just be another way of phrasing a precedency, but with a strong emphasis on its temporary nature, as well as a focus on a group-decided outcome. Alternatively, see it as a community governance model with a strong emphasis on responsibility. Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Do, 2015-08-27 at 12:34 -0700, Stefan van der Walt wrote: On 2015-08-27 11:06:10, Matthew Brett matthew.br...@gmail.com wrote: So, in the spirit of fruitful discussion, can I ask what y'all consider to be the current problems with working on numpy (other than the technical ones). What is numpy doing well, and what is it doing badly? What risks do we have to plan for in the future? It looks to me as though the team is doing an excellent job of maintaining NumPy. The growth of the project has stagnated somewhat for numerous reasons---and a lack of ideas on the table is not one of them, rather whether / how to take them forward. The question, and I think what you also highlighted in the earlier part of this discussion, is: how to decide on which vision to adopt, and who takes responsibility for making that happen? Are the two models proposed thus far so different, or can they be merged in a way that makes sense? E.g., can we work as a community to rally behind a vision as set out by one person, and then repeat that process to focus on another a year later? Think of it as the iterative development equivalent of governance. This may just be another way of phrasing a precedency, but with a strong emphasis on its temporary nature, as well as a focus on a group-decided outcome. Alternatively, see it as a community governance model with a strong emphasis on responsibility. Agreed. Are not PEP's/NEP's just that (and could possibly be formalized more, not sure how much they are in the current proposal) in some sense? Since they have a sponsor/author who can be said to be assigned to it/responsible once accepted. I will add one more thing which I think is important: The governance has to be create as little hassle as possible and it should be simple/short enough to quickly understand. - Sebastian Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Thu, Aug 27, 2015 at 11:06 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Aug 27, 2015 at 6:23 PM, josef.p...@gmail.com wrote: On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi On Thu, Aug 27, 2015 at 5:11 PM, josef.p...@gmail.com wrote: On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Aug 27, 2015 at 3:34 PM, josef.p...@gmail.com wrote: [snip] I don't really see a problem with codifying the status quo. That's an excellent point.If we believe that the current situation is the best possible, both now and in the future, then codifying the status quo is an excellent idea. So, we should probably first start by asking ourselves: * what numpy is doing well; * what numpy could do better; and then ask, is there some way we could make it more likely we will improve over time. [snip] As the current debate shows it's possible to have a public discussion about the direction of the project without having to delegate providing a vision to a president. The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for the performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1] Still doesn't sound like the need for a president to me the person who holds themselves responsible for the performance of the project sounds more like the role of the core group (adding plural to persons) to me, and cannot be pushed of to an official president. Except that, in the past, having multiple people taking decisions has led to the situation where no-one feels themselves accountable for the result, hence this situation tends to lead to stagnation. Is there any evidence for this? Oh - dear - that's the key point, but I'm obviously not making it clearly enough. Yes there is, and that was the evidence I was pointing to before. But anyway - Sebastian is right - this discussion isn't going anywhere useful. So - let's step back. In thinking about governance, we first need to ask what we want to achieve. This includes considering the risks ahead for the project. So, in the spirit of fruitful discussion, can I ask what y'all consider to be the current problems with working on numpy (other than the technical ones). What is numpy doing well, and what is it doing badly? What risks do we have to plan for in the future? joke Are you trying to prove the point that consensus doesn't work by making it impossible to reach a consensus on this? ;-) /joke One thing we are doing very badly is leveraging resources outside of contributions of work and time from individuals. Getting sponsors to finance work on what is the cornerstone of just about any Python package that has to add two numbers together shouldn't be too hard, especially seeing success stories like Jupyter's, who I believe has several paid developers working full time. That requires formalizing governance, because apparently sponsors are a little wary of giving money to people on the internet. ;-) Fernando Pérez was extremely emphatic about the size of the opportunity NumPy was letting slip by not formalizing *any* governance model. And it is a necessary first step so that e.g. we have the money to, say a year from now, get the right people together for a couple of days to figure out a better governance model. I'd argue that money would be better spent financing a talented developer to advance e.g. Nathaniel's new dtype system to end all dtype systems, but that's a different story. Largely because of the above, even if Nathaniel's document involved tossing a coin to resolve disputes, I'd rather have that now than something much better never. Because there really is no alternative to Nathaniel's write-up of the status quo, other than the status quo without a write-up: it has taken him two months to put this draft together, **after** we agreed over several hours of face to face discussion on what the model should be. And I'm sure he has hated every minute he has had to put into it. So if we keep going around this in circles, after a few days we will all grow tired and go back to fighting over whether indexing should transpose subspaces or not, and all that other cool stuff we really enjoy. And a year from now we will be in the same place we are now, only a year older and deeper in (technical) debt. Jaime -- (\__/) ( O.o) ( ) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
Hi, On Thu, Aug 27, 2015 at 10:35 AM, Bryan Van de Ven bry...@continuum.io wrote: On Aug 27, 2015, at 10:22 AM, Matthew Brett matthew.br...@gmail.com wrote: In the case of the 'core' model, we have some compelling testimony from someone with a great deal of experience: Much of this early structure (CVS, web site, cabal [core group], etc.) was copied verbatim by other open source (this term not being in wide use yet) projects -- even the form of the project name and the term core. This later became a kind of standard template for starting up an open source project. [...] I'm sorry to say that I helped create this problem, and that most of the projects which modeled themselves after NetBSD (probably due to its high popularity in 1993 and 1994) have suffered similar problems. FreeBSD and XFree86, for example, have both forked successor projects (Dragonfly and X.org) for very similar reasons. Who goes on to propose: 7) The core group must be replaced with people who are actually competent and dedicated enough to review proposals, accept feedback, and make good decisions. More to the point, though, the core group must only act when *needed* -- most technical decisions should be left to the community to hash out; it must not preempt the community from developing better solutions. (This is how the core group worked during most of the project's growth period.) Sure. I think it's reasonable to give high weight to Hannum's assessment of the failure of the core group, but less weight to his proposal for a replacement, because at the time, I don't believe he was in a good position to assess whether his (apparent) alternative would run into the same trouble. It's always tempting to blame the people rather than the system, but in this case, I strongly suspect that it was the system that was fundamentally flawed, therefore either promoting the wrong people or putting otherwise competent people into situations where they are no longer getting useful feedback. It would be great, and very convenient, if the only management we needed was getting out of the way, but I doubt very much that that is the case. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Thu, Aug 27, 2015 at 2:16 AM, Matthew Brett matthew.br...@gmail.com wrote: So, I speculate, that a good governance model would have: * one 'president' who has to take final responsibility for all decisions; * this president might well have a fixed term, maybe with limits on the number of terms they can serve. * the president would be chosen by community vote and explicitly on the basis that they were good managers as well as coders; * for the presidential election, the candidates should set out what their vision for the project is, and how they plan to achieve that vision; We actually discussed some variants on this kind of idea at the meeting, and I think the general sense of those present was we didn't want to go there (for whatever that's worth). At least personally, I have to admit that the idea of a governance model involving elections fills me with creeping horror. The reason is that whole point of having a governance model (IMHO) is to (a) minimize the rise of interpersonal drama, (b) when some amount of interpersonal drama does inevitably arise anyway, provide some regulated channel for it, hopefully one that leads to a drama sink. But elections are a huge massive drama source. No-one wants to spend time campaigning or wondering how some technical proposal will effect their re-election chances, we want to get this sorted out so that we can stop thinking about it and go back to solving actually interesting problems... As for evidence... there are obviously projects that have had serious problems with some variant of core team model, but there are also many many successful projects that are also using variants of this model, and the document I sent around attempts to incorporate the lessons that have been learned in the process. OTOH after wracking my brain I think the only project I'm familiar with that has elections at all like this is Fedora, which elects... a core team (FESCo). Given that we don't have the problem of trying to manage thousands of contributors, I'm not sure their experience is really relevant. Or I guess Debian's use of General Resolutions as a decision-making procedure of last resort is kinda relevant, but... pretty different. (They also elect the project leader, which is more similar to what you're describing, but the project leader has no technical authority; in Debian the final authority short of a GR is the CTTE, which is explicitly designed as a classic beholden-to-nobody institution -- and even overriding the CTTE requires a supermajority.) I kinda feel like... as a rule of thumb, if your description of your governance model starts with the words I speculate that..., then NumPy is probably not a good project to use for your experiment? -n -- Nathaniel J. Smith -- http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [DRAFT] numpy governance document
On Thu, Aug 27, 2015 at 1:44 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Aug 27, 2015 at 7:33 AM, Nathaniel Smith n...@pobox.com wrote: Hi all, Here's a first draft of a governance document for NumPy. Thanks for this. I wasn't sure from your email whether you were asking for feedback as to whether this was the right governance model? I mean that - for code - I think the usual procedure would be to discuss various potential solutions on the mailing list, and then follow up with something like a NEP that lays out the various alternatives with their pros and cons. But I have the impression here that you consider the general form to be set, and that you are asking for comments on the detail. Is that right? I believe that the draft I sent around does reflect the consensus of those who were present at the dev meeting, but (as the document itself emphasizes!) of course it's helpful to hear critiques and concerns and ideas for how to do better... -- Nathaniel J. Smith -- http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [DRAFT] numpy governance document
Hi, On Thu, Aug 27, 2015 at 11:10 AM, Nathaniel Smith n...@pobox.com wrote: On Thu, Aug 27, 2015 at 1:44 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Aug 27, 2015 at 7:33 AM, Nathaniel Smith n...@pobox.com wrote: Hi all, Here's a first draft of a governance document for NumPy. Thanks for this. I wasn't sure from your email whether you were asking for feedback as to whether this was the right governance model? I mean that - for code - I think the usual procedure would be to discuss various potential solutions on the mailing list, and then follow up with something like a NEP that lays out the various alternatives with their pros and cons. But I have the impression here that you consider the general form to be set, and that you are asking for comments on the detail. Is that right? I believe that the draft I sent around does reflect the consensus of those who were present at the dev meeting, but (as the document itself emphasizes!) of course it's helpful to hear critiques and concerns and ideas for how to do better... I imagine this document was designed to be uncontroversial, in the sense that it more or less formalizes the status quo? I think it would be useful to set out alternatives that were or could be considered. It would be a shame to drift into the wrong governance model for lack of considering others. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] [DRAFT] numpy governance document
Hi all, Here's a first draft of a governance document for NumPy. A few people have seen sneak peeks and have suggested possibly reorganizing it either by taking some of the how consensus works stuff out into a separate document, or alternatively keeping that stuff in the foreground and moving some of the more legalistic bits into footnotes or something. I think there's some value in having a single document that sorta lays out everything a new contributor needs to know, both how things work normally and how they work when we need to fall back on formal processes, and have tried to make the different aspects fit together better in this draft, but I'm happy to reorganize however if that's what people want -- I just figured that I should at least get this out so the content is available for discussion, even if the form isn't perfect yet :-). In case it's useful for context, large chunks of text are taken from the Jupyter/IPython project's IPEP 29: https://github.com/ipython/ipython/wiki/IPEP-29:-Project-Governance There are some random tweaks throughout, but the parts that are mostly new are the Summary, Consensus-based decision making by the community, and Council decision making sections. -n - The purpose of this document is to formalize the governance process used by the NumPy project in both ordinary and extraordinary situations, and to clarify how decisions are made and how the various elements of our community interact, including the relationship between open source collaborative development and work that may be funded by for-profit or non-profit entities. Summary === NumPy is a community-owned and community-run project. To the maximum extent possible, decisions about project direction are made by community consensus (but note that consensus here has a somewhat technical meaning that might not match everyone's expectations -- see below). Some members of the community additionally contribute by serving on the NumPy steering council, where they are responsible for facilitating the establishment of community consensus, for stewarding project resources, and -- in extreme cases -- for making project decisions if the normal community-based process breaks down. The Project === The NumPy Project (The Project) is an open source software project affiliated with the 501(c)3 NumFocus Foundation. The goal of The Project is to develop open source software for array-based computing in Python, and in particular the `numpy` package, along with related software such as `f2py` and the NumPy Sphinx extensions. The Software developed by The Project is released under the BSD (or similar) open source license, developed openly and hosted on public GitHub repositories under the `numpy` GitHub organization. The Project is developed by a team of distributed developers, called Contributors. Contributors are individuals who have contributed code, documentation, designs or other work to the Project. Anyone can be a Contributor. Contributors can be affiliated with any legal entity or none. Contributors participate in the project by submitting, reviewing and discussing GitHub Pull Requests and Issues and participating in open and public Project discussions on GitHub, mailing lists, and other channels. The foundation of Project participation is openness and transparency. Here is a list of the current Contributors to the main NumPy repository: [https://github.com/numpy/numpy/graphs/contributors](https://github.com/numpy/numpy/graphs/contributors) The Project Community consists of all Contributors and Users of the Project. Contributors work on behalf of and are responsible to the larger Project Community and we strive to keep the barrier between Contributors and Users as low as possible. The Project is formally affiliated with the 501(c)3 NumFOCUS Foundation ([http://numfocus.org](http://numfocus.org)), which serves as its fiscal sponsor, may hold project trademarks and other intellectual property, helps manage project donations and acts as a parent legal entity. NumFOCUS is the only legal entity that has a formal relationship with the project (see Institutional Partners section below). Governance == This section describes the governance and leadership model of The Project. The foundations of Project governance are: - Openness Transparency - Active Contribution - Institutional Neutrality Consensus-based decision making by the community Normally, all project decisions will be made by consensus of all interested Contributors. The primary goal of this approach is to ensure that the people who are most affected by and involved in any given change can contribute their knowledge in the confidence that their voices will be heard, because thoughtful review from a broad community is the best mechanism we know of for creating high-quality software. The mechanism we use to accomplish this goal may be unfamiliar for those who
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
Hi, On Thu, Aug 27, 2015 at 11:05 AM, Nathaniel Smith n...@pobox.com wrote: On Thu, Aug 27, 2015 at 2:16 AM, Matthew Brett matthew.br...@gmail.com wrote: So, I speculate, that a good governance model would have: * one 'president' who has to take final responsibility for all decisions; * this president might well have a fixed term, maybe with limits on the number of terms they can serve. * the president would be chosen by community vote and explicitly on the basis that they were good managers as well as coders; * for the presidential election, the candidates should set out what their vision for the project is, and how they plan to achieve that vision; We actually discussed some variants on this kind of idea at the meeting, and I think the general sense of those present was we didn't want to go there (for whatever that's worth). At least personally, I have to admit that the idea of a governance model involving elections fills me with creeping horror. The reason is that whole point of having a governance model (IMHO) is to (a) minimize the rise of interpersonal drama Right - I think this is key to the problem in this model. It is designed not to cause any trouble, and to keep things running as they are without controversy. It works OK on average as long as 'no change' is the desired outcome. In general the core group know each other fairly well, and feel a sense of shared loyalty to the group. This loyalty is exercised when some outside or inside force challenges the direction of the project. This was what made it so hard for the XFree86 core group to pull back from the course they had set. The question is - is avoiding the potential controversy important enough to force us into a model that has (in my opinion) a high risk of tending to conservatism and stagnation? As for evidence... there are obviously projects that have had serious problems with some variant of core team model, but there are also many many successful projects that are also using variants of this model, and the document I sent around attempts to incorporate the lessons that have been learned in the process. OTOH after wracking my brain I think the only project I'm familiar with that has elections at all like this is Fedora, which elects... a core team (FESCo). Given that we don't have the problem of trying to manage thousands of contributors, I'm not sure their experience is really relevant. Or I guess Debian's use of General Resolutions as a decision-making procedure of last resort is kinda relevant, but... pretty different. (They also elect the project leader, which is more similar to what you're describing, but the project leader has no technical authority; in Debian the final authority short of a GR is the CTTE, which is explicitly designed as a classic beholden-to-nobody institution -- and even overriding the CTTE requires a supermajority.) I kinda feel like... as a rule of thumb, if your description of your governance model starts with the words I speculate that..., then NumPy is probably not a good project to use for your experiment? So my argument would be that our current data on success (lots of projects use this model and many of them are OK) is much less useful than the data from successful forks. I suppose my question ends up being - do you agree that the core model does have these risks? Do they worry you? What do you think we can do to guard against them? Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Do, 2015-08-27 at 10:45 +0100, Matthew Brett wrote: Hi, On Thu, Aug 27, 2015 at 10:35 AM, Bryan Van de Ven bry...@continuum.io wrote: On Aug 27, 2015, at 10:22 AM, Matthew Brett matthew.br...@gmail.com wrote: In the case of the 'core' model, we have some compelling testimony from someone with a great deal of experience: Much of this early structure (CVS, web site, cabal [core group], etc.) was copied verbatim by other open source (this term not being in wide use yet) projects -- even the form of the project name and the term core. This later became a kind of standard template for starting up an open source project. [...] I'm sorry to say that I helped create this problem, and that most of the projects which modeled themselves after NetBSD (probably due to its high popularity in 1993 and 1994) have suffered similar problems. FreeBSD and XFree86, for example, have both forked successor projects (Dragonfly and X.org) for very similar reasons. Who goes on to propose: 7) The core group must be replaced with people who are actually competent and dedicated enough to review proposals, accept feedback, and make good decisions. More to the point, though, the core group must only act when *needed* -- most technical decisions should be left to the community to hash out; it must not preempt the community from developing better solutions. (This is how the core group worked during most of the project's growth period.) Sure. I think it's reasonable to give high weight to Hannum's assessment of the failure of the core group, but less weight to his proposal for a replacement, because at the time, I don't believe he was in a good position to assess whether his (apparent) alternative would run into the same trouble. It's always tempting to blame the people rather than the system, but in this case, I strongly suspect that it was the system that was fundamentally flawed, therefore either promoting the wrong people or putting otherwise competent people into situations where they are no longer getting useful feedback. Maybe so. I do not know much at all about these models, but I am not sure how much applies here to numpy. Isn't at least FreeBSD a magnitude larger then numpy? We do need to have some formality about how to give out commit rights, and do final decision when all else fails. One thing I do not know is how a community vote could work at all, considering I do not even know how to count its members. Votes and presidents make sense to me for large projects with hundrets of developers on different corners (think of the gnome foundation, debian probably) [1]. One thing I could imagine adding is that the community should be encouraged to ask for/propose new members for the core team. Nobody is particularly in love with this model, but maybe out of our own ignorance, we do not see many alternatives after ruling out a BDFL. Yes, it is a lot fixing a status quo, but we have to fixate something. Any alternative suggestions are welcome and would be even after deciding on this. Though maybe that takes away some momentum. - Sebastian [1] If we were to have a central governance for the SciPy stack the story would seem very different to me. It would be great, and very convenient, if the only management we needed was getting out of the way, but I doubt very much that that is the case. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1.10.0rc1
Hi again,
The change seems to have possibly unforeseen consequences because some
ufuncs don't declare all possible types, e.g.:
a = np.arange(10, dtype=np.int32)
out = np.zeros_like(a)
np.fabs(a, out=out)
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: ufunc 'fabs' output (typecode 'd') could not be coerced to provided
output parameter (typecode 'i') according to the casting rule ''same_kind''
np.fabs.types
['e-e', 'f-f', 'd-d', 'g-g', 'O-O']
(while fabs() wouldn't necessarily make sense on complex numbers, it
does make sense on integers... and, ah, I've just noticed that np.abs()
also exists with more input types, which is confusing...)
Regards
Antoine.
On Wed, 26 Aug 2015 07:52:09 -0600
Charles R Harris charlesr.har...@gmail.com wrote:
On Wed, Aug 26, 2015 at 7:32 AM, Charles R Harris charlesr.har...@gmail.com
wrote:
On Wed, Aug 26, 2015 at 7:31 AM, Charles R Harris
charlesr.har...@gmail.com wrote:
On Wed, Aug 26, 2015 at 7:11 AM, Antoine Pitrou solip...@pitrou.net
wrote:
On Tue, 25 Aug 2015 10:26:02 -0600
Charles R Harris charlesr.har...@gmail.com wrote:
Hi All,
The silence after the 1.10 beta has been eerie. Consequently, I'm
thinking
of making a first release candidate this weekend. If you haven't yet
tested
the beta, please do so. It would be good to discover as many problems
as we
can before the first release.
Has typing of ufunc parameters become much stricter? I can't find
anything in the release notes, but see (1.10b1):
arr = np.linspace(0, 5, 10)
out = np.empty_like(arr, dtype=np.intp)
np.round(arr, out=out)
Traceback (most recent call last):
File stdin, line 1, in module
File
/home/antoine/np110/lib/python3.4/site-packages/numpy/core/fromnumeric.py,
line 2778, in round_
return round(decimals, out)
TypeError: ufunc 'rint' output (typecode 'd') could not be coerced to
provided output parameter (typecode 'l') according to the casting rule
''same_kind''
It used to work (1.9):
arr = np.linspace(0, 5, 10)
out = np.empty_like(arr, dtype=np.intp)
np.round(arr, out=out)
array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
out
array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
The default casting mode has been changed. I think this has been raising
a warning since 1.7 and was mentioned as a future change in 1.10, but you
are right, it needs to be mentioned in the 1.10 release notes.
Make that warned of in the 1.9.0 release notes.
Here it is in 1.9.0 with deprecation warning made visible.
```
In [3]: import warnings
In [4]: warnings.simplefilter('always')
In [5]: arr = np.linspace(0, 5, 10)
In [6]: out = np.empty_like(arr, dtype=np.intp)
In [7]: np.round(arr, out=out)
/home/charris/.local/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2640:
DeprecationWarning: Implicitly casting between incompatible kinds. In a
future numpy release, this will raise an error. Use casting=unsafe if
this is intentional.
return round(decimals, out)
Out[7]: array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
```
Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Defining a white noise process using numpy
Daniel Bliss wrote: Hi all, Can anyone give me some advice for translating this equation into code using numpy? eta(t) = lim(dt - 0) N(0, 1/sqrt(dt)), where N(a, b) is a Gaussian random variable of mean a and variance b**2. This is a heuristic definition of a white noise process. Thanks, Dan You want noise with infinite variance? That doesn't make sense. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Defining a white noise process using numpy
On Thu, Aug 27, 2015 at 12:51 AM Daniel Bliss daniel.p.bl...@gmail.com wrote: Can anyone give me some advice for translating this equation into code using numpy? eta(t) = lim(dt - 0) N(0, 1/sqrt(dt)), where N(a, b) is a Gaussian random variable of mean a and variance b**2. This is a heuristic definition of a white noise process. This is an abstract definition. How to express it in numpy will depend on what you want to do with it. The easiest and most likely thing you could want would be a time series, with N time steps dt, in which sample i is the average value of the white noise process from i*dt to (i+1)*dt. This is very easy to write in numpy: 1/np.sqrt(dt) * np.random.randn(N) Anne ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1.10.0rc1
The change also seems to have made datetime64 computations stricter:
np.datetime64('2010') - np.datetime64('2000-01-01')
numpy.timedelta64(3653,'D')
np.datetime64('2010') - np.datetime64('2000-01-01T00:00:00Z')
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: Cannot cast ufunc subtract input from dtype('M8[Y]') to
dtype('M8[s]') with casting rule 'same_kind'
Regards
Antoine.
On Wed, 26 Aug 2015 07:52:09 -0600
Charles R Harris charlesr.har...@gmail.com wrote:
On Wed, Aug 26, 2015 at 7:32 AM, Charles R Harris charlesr.har...@gmail.com
wrote:
On Wed, Aug 26, 2015 at 7:31 AM, Charles R Harris
charlesr.har...@gmail.com wrote:
On Wed, Aug 26, 2015 at 7:11 AM, Antoine Pitrou solip...@pitrou.net
wrote:
On Tue, 25 Aug 2015 10:26:02 -0600
Charles R Harris charlesr.har...@gmail.com wrote:
Hi All,
The silence after the 1.10 beta has been eerie. Consequently, I'm
thinking
of making a first release candidate this weekend. If you haven't yet
tested
the beta, please do so. It would be good to discover as many problems
as we
can before the first release.
Has typing of ufunc parameters become much stricter? I can't find
anything in the release notes, but see (1.10b1):
arr = np.linspace(0, 5, 10)
out = np.empty_like(arr, dtype=np.intp)
np.round(arr, out=out)
Traceback (most recent call last):
File stdin, line 1, in module
File
/home/antoine/np110/lib/python3.4/site-packages/numpy/core/fromnumeric.py,
line 2778, in round_
return round(decimals, out)
TypeError: ufunc 'rint' output (typecode 'd') could not be coerced to
provided output parameter (typecode 'l') according to the casting rule
''same_kind''
It used to work (1.9):
arr = np.linspace(0, 5, 10)
out = np.empty_like(arr, dtype=np.intp)
np.round(arr, out=out)
array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
out
array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
The default casting mode has been changed. I think this has been raising
a warning since 1.7 and was mentioned as a future change in 1.10, but you
are right, it needs to be mentioned in the 1.10 release notes.
Make that warned of in the 1.9.0 release notes.
Here it is in 1.9.0 with deprecation warning made visible.
```
In [3]: import warnings
In [4]: warnings.simplefilter('always')
In [5]: arr = np.linspace(0, 5, 10)
In [6]: out = np.empty_like(arr, dtype=np.intp)
In [7]: np.round(arr, out=out)
/home/charris/.local/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2640:
DeprecationWarning: Implicitly casting between incompatible kinds. In a
future numpy release, this will raise an error. Use casting=unsafe if
this is intentional.
return round(decimals, out)
Out[7]: array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
```
Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Thu, Aug 27, 2015 at 9:36 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Wed, Aug 26, 2015 at 11:46 PM, Stefan van der Walt stef...@berkeley.edu wrote: Hi Matthew On 2015-08-26 10:50:47, Matthew Brett matthew.br...@gmail.com wrote: In short, the core structure seems to be characteristically associated with a conservatism and lack of vision that causes the project to stagnate. Can you describe how a democratic governance structure would look? It's not clear from the discussions linked where successful examples are to be found. Ah yes - as I was writing at the top of the xfree86 summary, it's difficult to assess governance models, because you cannot tell if a project that has a particular governance model would have been more successful with another model. For example, would clang be competing so successfully with gcc, if gcc had had a different governance model? Would Apache be further ahead of the many competitors in the web-server space with different management? Difficult to know. The advantage of studying forks is that they usually arise from disagreements about how a project is managed. All other things being equal, we might expect a fork to fail, given the general aversion to forks and the considerable new work that has to be done to get one going. So, if a fork succeeds in the long term, that is probably an indication that the governance / management of the fork is indeed an improvement on the previous model. So, in answer to your question, it's difficult to know if a particular governance model is successful. It isn't enough that a project has lasted, or is still active, because there are so many factors in play. On the other hand, I think it is possible to point to models that have a tendency to fail in particular ways, and the by-invitation meritocratic 'core' group (I think this is close to the 'steering committee' in our current draft) is the model that failed for NetBSD and XFree86, with a particular pattern of poor or absent accountability and lack of project vision. Sorry to follow up on my own email, but: I'm just speculating here, without data, but I suspect one the key elements that led to the decline and fall of NetBSD and XFree86 was the perception that there was no way for the community to depose the government. It seems these projects managed to combine aspects of the dictatorship model, with lots of emphasis on personal loyalty and expected gratitude, with a dysfunctional oligarchy, in which no-one felt able or willing to change the project direction, when the project was failing. The other problem with the meritocracy / invitation model, is that some people are terrible managers. In the XFree86 project, for example, I think David Dawes did a terrible job of guiding the project when it ran into trouble. He was in the position he was in because of his huge commitment and contributions to the project, but I think he was not the right person to manage the project. The standard 'core' model, doesn't take that into account. For example, I suspect that, if we had a David Dawes, no matter how terrible we thought they were at managing, we would feel obliged to put them onto the steering committee. It is much easier to count or review commits than it is to assess someone for their qualities as a leader or manager. We most of us would hate to be the person to make that assessment, and it's very tempting to negotiate ourselves into a world-view where this assessment is not necessary. So, I speculate, that a good governance model would have: * one 'president' who has to take final responsibility for all decisions; * this president might well have a fixed term, maybe with limits on the number of terms they can serve. * the president would be chosen by community vote and explicitly on the basis that they were good managers as well as coders; * for the presidential election, the candidates should set out what their vision for the project is, and how they plan to achieve that vision; The point about these features is that we explicitly emphasize accountability, vision and management ability. Instead of a small number of people being in the position of assessing their peers for their ability to manage, the whole community (somehow defined) takes responsibility for that assessment, therefore making it easier to think about without distracting issues of personal loyalty or implied obligation. See you, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Aug 27, 2015, at 9:36 AM, Matthew Brett matthew.br...@gmail.com wrote: So, in answer to your question, it's difficult to know if a particular governance model is successful. It isn't enough that a project has lasted, or is still active, because there are so many factors in play. On the other hand, I think it is possible to point to models that have a tendency to fail in particular ways, and the by-invitation meritocratic 'core' group (I think this is close to the 'steering committee' in our current draft) is the model that failed for NetBSD and XFree86, with a particular pattern of poor or absent accountability and lack of project vision. Anecdotes about two projects is not compelling evidence of anything unless you can also point to a comparison of the corresponding success rate. Two failures out of three is suggestive. Two failures out of three hundred is significantly less interesting. More useful would be actual details of an alternative proposal or pointers to examples of alternative arrangements that could be modeled. Bryan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
Hi, On Thu, Aug 27, 2015 at 10:15 AM, Bryan Van de Ven bry...@continuum.io wrote: On Aug 27, 2015, at 9:36 AM, Matthew Brett matthew.br...@gmail.com wrote: So, in answer to your question, it's difficult to know if a particular governance model is successful. It isn't enough that a project has lasted, or is still active, because there are so many factors in play. On the other hand, I think it is possible to point to models that have a tendency to fail in particular ways, and the by-invitation meritocratic 'core' group (I think this is close to the 'steering committee' in our current draft) is the model that failed for NetBSD and XFree86, with a particular pattern of poor or absent accountability and lack of project vision. Anecdotes about two projects is not compelling evidence of anything unless you can also point to a comparison of the corresponding success rate. Two failures out of three is suggestive. Two failures out of three hundred is significantly less interesting. More useful would be actual details of an alternative proposal or pointers to examples of alternative arrangements that could be modeled. Unfortunately, I don't think we have much choice but to do our best in sifting through the anecdotal evidence we have available, weak and contradictory as it is. Successful forks in large projects are pretty rare, and as I was arguing before, they are particularly useful as evidence about governance models. In the case of the 'core' model, we have some compelling testimony from someone with a great deal of experience: Much of this early structure (CVS, web site, cabal [core group], etc.) was copied verbatim by other open source (this term not being in wide use yet) projects -- even the form of the project name and the term core. This later became a kind of standard template for starting up an open source project. [...] I'm sorry to say that I helped create this problem, and that most of the projects which modeled themselves after NetBSD (probably due to its high popularity in 1993 and 1994) have suffered similar problems. FreeBSD and XFree86, for example, have both forked successor projects (Dragonfly and X.org) for very similar reasons. http://mail-index.netbsd.org/netbsd-users/2006/08/30/0016.html Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Aug 27, 2015, at 10:22 AM, Matthew Brett matthew.br...@gmail.com wrote: In the case of the 'core' model, we have some compelling testimony from someone with a great deal of experience: Much of this early structure (CVS, web site, cabal [core group], etc.) was copied verbatim by other open source (this term not being in wide use yet) projects -- even the form of the project name and the term core. This later became a kind of standard template for starting up an open source project. [...] I'm sorry to say that I helped create this problem, and that most of the projects which modeled themselves after NetBSD (probably due to its high popularity in 1993 and 1994) have suffered similar problems. FreeBSD and XFree86, for example, have both forked successor projects (Dragonfly and X.org) for very similar reasons. Who goes on to propose: 7) The core group must be replaced with people who are actually competent and dedicated enough to review proposals, accept feedback, and make good decisions. More to the point, though, the core group must only act when *needed* -- most technical decisions should be left to the community to hash out; it must not preempt the community from developing better solutions. (This is how the core group worked during most of the project's growth period.) Which, AFAICT, is exactly in line with the Numpy proposal: During the everyday project activities, council members participate in all discussions, code review and other project activities as peers with all other Contributors and the Community. In these everyday activities, Council Members do not have any special power or privilege through their membership on the Council. ... However, the Council's primary responsibility is to facilitate the ordinary community-based decision making procedure described above. If we ever have to step in and formally override the community for the health of the Project, then we will do so, but we will consider reaching this point to indicate a failure in our leadership. Bryan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
Hi, On Wed, Aug 26, 2015 at 11:46 PM, Stefan van der Walt stef...@berkeley.edu wrote: Hi Matthew On 2015-08-26 10:50:47, Matthew Brett matthew.br...@gmail.com wrote: In short, the core structure seems to be characteristically associated with a conservatism and lack of vision that causes the project to stagnate. Can you describe how a democratic governance structure would look? It's not clear from the discussions linked where successful examples are to be found. Ah yes - as I was writing at the top of the xfree86 summary, it's difficult to assess governance models, because you cannot tell if a project that has a particular governance model would have been more successful with another model. For example, would clang be competing so successfully with gcc, if gcc had had a different governance model? Would Apache be further ahead of the many competitors in the web-server space with different management? Difficult to know. The advantage of studying forks is that they usually arise from disagreements about how a project is managed. All other things being equal, we might expect a fork to fail, given the general aversion to forks and the considerable new work that has to be done to get one going. So, if a fork succeeds in the long term, that is probably an indication that the governance / management of the fork is indeed an improvement on the previous model. So, in answer to your question, it's difficult to know if a particular governance model is successful. It isn't enough that a project has lasted, or is still active, because there are so many factors in play. On the other hand, I think it is possible to point to models that have a tendency to fail in particular ways, and the by-invitation meritocratic 'core' group (I think this is close to the 'steering committee' in our current draft) is the model that failed for NetBSD and XFree86, with a particular pattern of poor or absent accountability and lack of project vision. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [DRAFT] numpy governance document
Hi, On Thu, Aug 27, 2015 at 7:33 AM, Nathaniel Smith n...@pobox.com wrote: Hi all, Here's a first draft of a governance document for NumPy. Thanks for this. I wasn't sure from your email whether you were asking for feedback as to whether this was the right governance model? I mean that - for code - I think the usual procedure would be to discuss various potential solutions on the mailing list, and then follow up with something like a NEP that lays out the various alternatives with their pros and cons. But I have the impression here that you consider the general form to be set, and that you are asking for comments on the detail. Is that right? Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
Hi Sebastian On 2015-08-27 14:45:50, Sebastian Berg sebast...@sipsolutions.net wrote: Agreed. Are not PEP's/NEP's just that (and could possibly be formalized more, not sure how much they are in the current proposal) in some sense? Since they have a sponsor/author who can be said to be assigned to it/responsible once accepted. I would consider a collection of NEPs for the following year to be such a thing. When implementing a bigger plan, it does help to have one person who owns the entire vision at the helm, pushing it forward. I think of it like a symphony orchestra: we agree to play the same music, and then let the director oversee the execution of the piece as a whole. The governance has to be create as little hassle as possible and it should be simple/short enough to quickly understand. I completely agree; and I think bikeshedding (everyone gets to argue their point of view, regardless of the stakes) can be the antithesis to productive focus. Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
