Re: [Numpy-discussion] What is consensus anyway
On Wed, Apr 25, 2012 at 4:02 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Apr 24, 2012 at 8:56 PM, Fernando Perez fperez@gmail.com wrote: On Tue, Apr 24, 2012 at 6:12 PM, Charles R Harris charlesr.har...@gmail.com wrote: I admit to a certain curiosity about your own involvement in FOSS projects, and I know I'm not alone in this. Google shows several years of discussion on Monotone, but I have no idea what your contributions were Seriously??? Please, let's rise above this. We discuss people's opinions *on their technical merit alone*, regardless of the background of the person presenting them. I don't care if Linus himself shows up on the list with a bad idea, it should be shot down; and if someone we'd never heard of brings up a valid point, we should respect it. The day we start checking credentials at the door is the day this project will die as an open source effort. Or at least I think so, but perhaps I don't have enough 'commit credits' in my account for my opinion to matter... Fernando, I'm not checking credentials, I'm curious. Nathaniel has experience with FOSS projects, unlike us first timers, and I'd like to know what that experience was and what he learned from it. He has also mentioned Graydon Hoare in connection with RUST, and since Graydon was the prime mover in Monotone I'd like to know the story of the project. Yeah, I don't want to get into resumes and such here, since it'd be hard to avoid turning it into one of those whose has a bigger FOSS pecking-order contests, which I find both unpleasant and counter-productive. If I've learned anything useful from experience, then I've already tried to summarize it here (and really, experience may or may not guarantee any kind of wisdom). If you want to swap war stories, ask me some day over a $BEVERAGE :-). After sleeping on it, I was wondering if part of your objection to the consensus stuff is just to the word veto? Would you feel more comfortable if it was phrased like, the maintainers have noticed that trying to pick and choose on contentious issues tends to come back and bite them, so they've decided that they will not accept changes unless they have reasonable certainty that all substantive objections from the userbase have been worked through and resolved? It means the same thing in the end, but perhaps makes clearer how the power actually works. -- Nathaniel ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
On Wed, Apr 25, 2012 at 4:07 AM, Nathaniel Smith n...@pobox.com wrote: On Wed, Apr 25, 2012 at 4:02 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Apr 24, 2012 at 8:56 PM, Fernando Perez fperez@gmail.com wrote: On Tue, Apr 24, 2012 at 6:12 PM, Charles R Harris charlesr.har...@gmail.com wrote: I admit to a certain curiosity about your own involvement in FOSS projects, and I know I'm not alone in this. Google shows several years of discussion on Monotone, but I have no idea what your contributions were Seriously??? Please, let's rise above this. We discuss people's opinions *on their technical merit alone*, regardless of the background of the person presenting them. I don't care if Linus himself shows up on the list with a bad idea, it should be shot down; and if someone we'd never heard of brings up a valid point, we should respect it. The day we start checking credentials at the door is the day this project will die as an open source effort. Or at least I think so, but perhaps I don't have enough 'commit credits' in my account for my opinion to matter... Fernando, I'm not checking credentials, I'm curious. Nathaniel has experience with FOSS projects, unlike us first timers, and I'd like to know what that experience was and what he learned from it. He has also mentioned Graydon Hoare in connection with RUST, and since Graydon was the prime mover in Monotone I'd like to know the story of the project. Yeah, I don't want to get into resumes and such here, since it'd be hard to avoid turning it into one of those whose has a bigger FOSS pecking-order contests, which I find both unpleasant and counter-productive. If I've learned anything useful from experience, then I've already tried to summarize it here (and really, experience may or may not guarantee any kind of wisdom). If you want to swap war stories, ask me some day over a $BEVERAGE :-). Well, you have already appealed to the authority of greater experience, so it's a bit late to declare disinterest in the subject ;) I mean, at this point I really would like to see how big your FOSS is. After sleeping on it, I was wondering if part of your objection to the consensus stuff is just to the word veto? Would you feel more comfortable if it was phrased like, the maintainers have noticed that trying to pick and choose on contentious issues tends to come back and bite them, so they've decided that they will not accept changes unless they have reasonable certainty that all substantive objections from the userbase have been worked through and resolved? It means the same thing in the end, but perhaps makes clearer how the power actually works. I don't agree here. People work on open source to scratch an itch, so the process of making a contribution needs to be easy. Widespread veto makes it more difficult and instead of opening up the process, closes it down. There is less freedom, not more. That is one of the reasons that the smaller scikits attract people, they have more freedom to do what they want and fewer people to answer to. Scipy also has some of that advantage because there are a number of packages to choose from. The more strict the process and the more people to please, the less appealing the environment becomes. This can be observed in practice and the voluntary nature of FOSS amplifies the effect. But in the end, someone has to write the code. Steve McConnell (Code Complete) estimates that even in carefully planned projects code construction will take up 60-80 percent of the time and effort. And if the code isn't written, nothing else matters much. That is why people who write code are essential to a project, no amount of structure will take their place. And here again the voluntary nature of FOSS comes into play, folks can't be ordered to do the work. It can be suggested that certain things be done, and the desire to work with the group will lead people to do work they wouldn't consider doing for themselves, but unless they are interested in a particular feature they won't generally be motivated to sit down and devote the effort needed to get it done just because someone else wants it. And they will rightly be offended if anyone demands that they volunteer their work to implement some feature in a particular way. They have to be led there, not pushed. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
On Wed, Apr 25, 2012 at 06:03:25AM -0600, Charles R Harris wrote: Well, you have already appealed to the authority of greater experience, so it's a bit late to declare disinterest in the subject ;) I mean, at this point I really would like to see how big your FOSS is. Chuck, I am not sure that this is helpful for the discussion. I think that it is a great discussion to have in real life, as it is one of those in which all participants can learn a lot, but on a mailing list with a wider diffusion, it can very easily drift in a pissing contest. Gaël ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
On Wed, Apr 25, 2012 at 1:03 PM, Charles R Harris charlesr.har...@gmail.com wrote: That is one of the reasons that the smaller scikits attract people, they have more freedom to do what they want and fewer people to answer to. Scipy also has some of that advantage because there are a number of packages to choose from. The more strict the process and the more people to please, the less appealing the environment becomes. A quick look shows ~100,000 downloads of 1.6.1 via PyPI. SF.net shows 600,000 numpy downloads in the last 12 months. I'm afraid the numpy developers have a lot of people to please, whether they like it or not :-). OTOH I'm still confused at what kind of strictness you're worried about in practice. Not too many of those people actually show up on the mailing list, and usually the problem is convincing those that *do* show up into actually expressing their needs rather than just assuming that real developers must know better. Fernando spoke eloquently in this thread in support of consensus, and IPython doesn't seem to be laboring under a strict process that's driving away developers. AFAICT whole-heartedly adopting the consensus idea would only have actually altered one (!) decision in the project to date, which is not exactly jack-booted as these things go. - N ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] A crazy masked-array thought
The masked array discussions have brought up all sorts of interesting topics - too many to usefully list here - but there's one aspect I haven't spotted yet. Perhaps that's because it's flat out wrong, or crazy, or just too awkward to be helpful. But ... Shouldn't masked arrays (MA) be a superclass of the plain-old-array (POA)? In the library I'm working on, the introduction of MAs (via numpy.ma) required us to sweep through the library and make a fair few changes. That's not the sort of thing one would normally expect from the introduction of a subclass. Putting aside the ABI issue, would it help downstream API compatibility if the POA was a subclass of the MA? Code that's expecting/casting-to a POA might continue to work and, where appropriate, could be upgraded in their own time to accept MAs. Richard Hattersley ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] linalg.lstsq
Hello, is there weighted version of linalg.lstsq available? In my case, b is a (N,K) matrix, so i can't use manual scaling of x and b. greetings Till ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
I don't agree here. People work on open source to scratch an itch, so the process of making a contribution needs to be easy. Widespread veto makes it more difficult and instead of opening up the process, closes it down. There is less freedom, not more. That is one of the reasons that the smaller scikits attract people, they have more freedom to do what they want and fewer people to answer to. Scipy also has some of that advantage because there are a number of packages to choose from. The more strict the process and the more people to please, the less appealing the environment becomes. This can be observed in practice and the voluntary nature of FOSS amplifies the effect. It is true that it is easier to get developers to contribute to small projects where they can control exactly what happens and not have to appeal to a wider audience to get code changed and committed. This effect is well-illustrated by the emergence of scikits in the presence of SciPy. However, the idea that people work on open source to scratch an itch is incomplete. This is certainly one of the reasons volunteers work on open source.There are many people, however, that work on open source as part of their job. In the particular instance of the missing data support, Mark did much of the work as part of his job. It wasn't just to scratch an itch. So, we should not make assumptions on the basis of this incomplete model. NumPy is far-beyond the mode of a few people scratching an itch. It is in wide-spread use. It is a large project with a great deal of history and a diverse user-community. It needs people full-time to help maintain it.It needs maintainers who listen actively to anyone who will express their concerns cogently.It needs maintainers who recognize that any concern that somebody expresses is typically not a unique view. We cannot expect to find people like that who are just interested in scratching an itch and always working for free. Most projects suffer from lack of feedback. We should be worried about how to get more feedback and input from *just users* and be very sensitive to anyone feeling like their legitimate concerns are not being heard. Most people, rather than express their concerns, will just work-around the problem, write their own stuff, or move on to other languages and approaches. Your point about somebody writing the code is absolutely true, I would just suggest that the view that FOSS is always just volunteer labor needs to expand. People do work full time on FOSS as part of their job. We need to bring that to NumPy. I know of at least 2 other people besides me who are actively trying to make this possible. At Continuum we offer the opportunity to work on NumPy. We plan to continue this. We are hiring.In this context, I'm especially interested in making sure that it's not just the developers who get to decide what happens to NumPy. Nathaniel has clarified very well what veto-power really means. It's not absolute, it just means that users who write clear arguments get listened to actively. It doesn't replace the need for developers with wisdom and understanding of user-experiences, but active listening is a useful skill that we could all improve on: http://en.wikipedia.org/wiki/Active_listening A list full of bright, interested, active listeners is the kind of culture we need on this list. It's the kind of attitude we need from maintainers of NumPy. -Travis But in the end, someone has to write the code. Steve McConnell (Code Complete) estimates that even in carefully planned projects code construction will take up 60-80 percent of the time and effort. And if the code isn't written, nothing else matters much. That is why people who write code are essential to a project, no amount of structure will take their place. And here again the voluntary nature of FOSS comes into play, folks can't be ordered to do the work. It can be suggested that certain things be done, and the desire to work with the group will lead people to do work they wouldn't consider doing for themselves, but unless they are interested in a particular feature they won't generally be motivated to sit down and devote the effort needed to get it done just because someone else wants it. And they will rightly be offended if anyone demands that they volunteer their work to implement some feature in a particular way. They have to be led there, not pushed. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] linalg.lstsq
On Wed, Apr 25, 2012 at 12:13 PM, Till Stensitzki mail.t...@gmx.de wrote: Hello, is there weighted version of linalg.lstsq available? In my case, b is a (N,K) matrix, so i can't use manual scaling of x and b. What shape are the weights in this case? I'm not that familiar with problems with an N x K b matrix. Skipper ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
Hi, On Wed, Apr 25, 2012 at 9:39 AM, Travis Oliphant tra...@continuum.io wrote: I don't agree here. People work on open source to scratch an itch, so the process of making a contribution needs to be easy. Widespread veto makes it more difficult and instead of opening up the process, closes it down. There is less freedom, not more. That is one of the reasons that the smaller scikits attract people, they have more freedom to do what they want and fewer people to answer to. Scipy also has some of that advantage because there are a number of packages to choose from. The more strict the process and the more people to please, the less appealing the environment becomes. This can be observed in practice and the voluntary nature of FOSS amplifies the effect. It is true that it is easier to get developers to contribute to small projects where they can control exactly what happens and not have to appeal to a wider audience to get code changed and committed. This effect is well-illustrated by the emergence of scikits in the presence of SciPy. However, the idea that people work on open source to scratch an itch is incomplete. Do you agree that Numpy has not been very successful in recruiting and maintaining new developers compared to its large user-base? Compared to - say - Sympy? Why do you think this is? Would you consider asking that question directly on list and asking for the most honest possible answers? Best, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
Do you agree that Numpy has not been very successful in recruiting and maintaining new developers compared to its large user-base? Compared to - say - Sympy? Why do you think this is? I don't know about SymPy. But in my view (and I'm just a typical user of NumPy), numpy seems to be at the base of what people actually need to do. I would assume most users of numpy actually use it because it's underlying piece of software, i.e. SciPy. It provides convenient, fast array structures to do maths. I would assume that most users see numpy as infrastructure, they write their own code on top of it. As a normal user of numpy, I wouldn't know where it would need improvement to suit my needs because it already does all I need. (Okay, masked arrays are something which could definitely improve, but that's another story.) This is different from other, higher-level FOSS projects, which are closer to end user final requirements, where end users might be more compelled to contribute because it's closer to what they're actually doing. For example, I just wrote two enhancements to scipy.interpolate, which were / will be merged recently / soon. Plus, numpy is a lot of C code, and to me (again, as a user) it seems more complicated to contribute because things are not as isolated. Just my 2 ct. Andreas. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
On 4/25/2012 4:51 PM, Andreas H. wrote: I would assume that most users see numpy as infrastructure, they write their own code on top of it. As a normal user of numpy, I wouldn't know where it would need improvement to suit my needs because it already does all I need. (Okay, masked arrays are something which could definitely improve, but that's another story.) This is different from other, higher-level FOSS projects, Thank you Andreas. I was debating whether to explain exactly this, to point out that I found Matthew's question inappropriately aggressive, or both. Now I can do both in a flash. But I find I would also like to once again say thank you to the developers, who have given us an amazing piece of software. I would add that I am impressed by the deep respect they show each other even when dealing with hard issues. Alan Isaac Just another grateful user for many years. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
I too have to agree with Andreas. I have been using Numpy for years in my work, but am not versed in C so I don't even understand what numpy is doing under the hood. I too would only be able to contribute to the code at the python level, or as Andreas said, at improving SciPy packages and other Numpy-based projects. One area that you may be able to get more help from the general user base is with publicity, tutorials and word-of-mouth. I had recently shown Numpy to a friend who was versed in matlab, and he was really impressed because Numpy is easily incorporated into more general Python scripts. I've worked a lot with the Enthought Tool Suite and shown off some of that to my colleagues. They are impressed at the streamlined code-to-visuals process although I don't think they even realize that Numpy is responsible for all the numerics in the program. To this end, I think outreach would be helpful in recruiting new programmers. Once they understand that Numpy does a lot at the C-level and that it is not strictly a Python feature, they may realize its something that they can contribute to. On Wed, Apr 25, 2012 at 5:04 PM, Alan G Isaac alan.is...@gmail.com wrote: On 4/25/2012 4:51 PM, Andreas H. wrote: I would assume that most users see numpy as infrastructure, they write their own code on top of it. As a normal user of numpy, I wouldn't know where it would need improvement to suit my needs because it already does all I need. (Okay, masked arrays are something which could definitely improve, but that's another story.) This is different from other, higher-level FOSS projects, Thank you Andreas. I was debating whether to explain exactly this, to point out that I found Matthew's question inappropriately aggressive, or both. Now I can do both in a flash. But I find I would also like to once again say thank you to the developers, who have given us an amazing piece of software. I would add that I am impressed by the deep respect they show each other even when dealing with hard issues. Alan Isaac Just another grateful user for many years. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
Do you agree that Numpy has not been very successful in recruiting and maintaining new developers compared to its large user-base? Compared to - say - Sympy? Why do you think this is? I think it's mostly because it's infrastructure that is a means to an end. I certainly wasn't excited to have to work on NumPy originally, when my main interest was SciPy.I've come to love the interesting plateau that NumPy lives on.But, I think it mostly does the job it is supposed to do. The fact that it is in C is also not very sexy. It is also rather complicated with a lot of inter-related parts. I think NumPy could do much, much more --- but getting there is going to be a challenge of execution and education. You can get to know the code base. It just takes some time and patience. You also have to be comfortable with compilers and building software just to tweak the code. Would you consider asking that question directly on list and asking for the most honest possible answers? I'm always interested in honest answers and welcome any sincere perspective. -Travis ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
Hi, On Wed, Apr 25, 2012 at 2:35 PM, Travis Oliphant tra...@continuum.io wrote: Do you agree that Numpy has not been very successful in recruiting and maintaining new developers compared to its large user-base? Compared to - say - Sympy? Why do you think this is? I think it's mostly because it's infrastructure that is a means to an end. I certainly wasn't excited to have to work on NumPy originally, when my main interest was SciPy. I've come to love the interesting plateau that NumPy lives on. But, I think it mostly does the job it is supposed to do. The fact that it is in C is also not very sexy. It is also rather complicated with a lot of inter-related parts. I think NumPy could do much, much more --- but getting there is going to be a challenge of execution and education. You can get to know the code base. It just takes some time and patience. You also have to be comfortable with compilers and building software just to tweak the code. Would you consider asking that question directly on list and asking for the most honest possible answers? I'm always interested in honest answers and welcome any sincere perspective. Of course, there are potential explanations: 1) Numpy is too low-level for most people 2) The C code is too complicated 3) It's fine already, more or less are some obvious ones. I would say there are the easy answers. But of course, the easy answer may not be the right answer. It may not be easy to get right answer [1]. As you can see from Alan Isaac's reply on this thread, even asking the question can be taken as being in bad faith. In that situation, I think you'll find it hard to get sincere replies. Best, Matthew [1] http://en.wikipedia.org/wiki/Good_to_Great ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
On Wed, Apr 25, 2012 at 5:54 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Wed, Apr 25, 2012 at 2:35 PM, Travis Oliphant tra...@continuum.io wrote: Do you agree that Numpy has not been very successful in recruiting and maintaining new developers compared to its large user-base? Compared to - say - Sympy? Why do you think this is? I think it's mostly because it's infrastructure that is a means to an end. I certainly wasn't excited to have to work on NumPy originally, when my main interest was SciPy. I've come to love the interesting plateau that NumPy lives on. But, I think it mostly does the job it is supposed to do. The fact that it is in C is also not very sexy. It is also rather complicated with a lot of inter-related parts. I think NumPy could do much, much more --- but getting there is going to be a challenge of execution and education. You can get to know the code base. It just takes some time and patience. You also have to be comfortable with compilers and building software just to tweak the code. Would you consider asking that question directly on list and asking for the most honest possible answers? I'm always interested in honest answers and welcome any sincere perspective. Of course, there are potential explanations: 1) Numpy is too low-level for most people 2) The C code is too complicated 3) It's fine already, more or less are some obvious ones. I would say there are the easy answers. But of course, the easy answer may not be the right answer. It may not be easy to get right answer [1]. As you can see from Alan Isaac's reply on this thread, even asking the question can be taken as being in bad faith. In that situation, I think you'll find it hard to get sincere replies. I don't see why this shouldn't be the sincere replies, I think these easy answers are also the right answer for most people. maybe I would add 4) writing code for a few hundred thousand users is a big responsibility and a bit scary Except for a few core c developers, most contributors contribute to parts of numpy, best example Pierre and masked arrays, or specific functions. Life goes on for most developers in the application areas, I guess. For example I'm very glad about the time that Pauli is spending on scipy. numpy is great [1] Josef Best, Matthew [1] http://en.wikipedia.org/wiki/Good_to_Great [1] http://sourceforge.net/projects/numpy/files/stats/timeline?dates=2000-01-11+to+2012-04-25 http://qa.debian.org/popcon.php?package=python-numpy ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
On Wednesday, April 25, 2012, Matthew Brett wrote: Hi, On Wed, Apr 25, 2012 at 2:35 PM, Travis Oliphant tra...@continuum.iojavascript:; wrote: Do you agree that Numpy has not been very successful in recruiting and maintaining new developers compared to its large user-base? Compared to - say - Sympy? Why do you think this is? I think it's mostly because it's infrastructure that is a means to an end. I certainly wasn't excited to have to work on NumPy originally, when my main interest was SciPy.I've come to love the interesting plateau that NumPy lives on.But, I think it mostly does the job it is supposed to do. The fact that it is in C is also not very sexy. It is also rather complicated with a lot of inter-related parts. I think NumPy could do much, much more --- but getting there is going to be a challenge of execution and education. You can get to know the code base. It just takes some time and patience. You also have to be comfortable with compilers and building software just to tweak the code. Would you consider asking that question directly on list and asking for the most honest possible answers? I'm always interested in honest answers and welcome any sincere perspective. Of course, there are potential explanations: 1) Numpy is too low-level for most people 2) The C code is too complicated 3) It's fine already, more or less are some obvious ones. I would say there are the easy answers. But of course, the easy answer may not be the right answer. It may not be easy to get right answer [1]. As you can see from Alan Isaac's reply on this thread, even asking the question can be taken as being in bad faith. In that situation, I think you'll find it hard to get sincere replies. As with anything, the phrasing of a question makes a world of a difference with regards to replies. Ask any pollster. When phrased correctly, I would not have any doubt about the sincerely of replies, and I would not worry about previewed hostility -- when phrased correctly. As the questioner, the onus is upon you to gauge the community and adjust the question appropriately. I think the fact that we engage in these discussions show that we value and care about each others perceptions and opinions with regards to numpy. Cheers! Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
Hi, On Wed, Apr 25, 2012 at 1:35 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Wed, Apr 25, 2012 at 9:39 AM, Travis Oliphant tra...@continuum.io wrote: I don't agree here. People work on open source to scratch an itch, so the process of making a contribution needs to be easy. Widespread veto makes it more difficult and instead of opening up the process, closes it down. There is less freedom, not more. That is one of the reasons that the smaller scikits attract people, they have more freedom to do what they want and fewer people to answer to. Scipy also has some of that advantage because there are a number of packages to choose from. The more strict the process and the more people to please, the less appealing the environment becomes. This can be observed in practice and the voluntary nature of FOSS amplifies the effect. It is true that it is easier to get developers to contribute to small projects where they can control exactly what happens and not have to appeal to a wider audience to get code changed and committed. This effect is well-illustrated by the emergence of scikits in the presence of SciPy. However, the idea that people work on open source to scratch an itch is incomplete. Do you agree that Numpy has not been very successful in recruiting and maintaining new developers compared to its large user-base? Compared to - say - Sympy? Why do you think this is? Would you consider asking that question directly on list and asking for the most honest possible answers? Aha - I now realize that I was reading too quickly under the influence (again) of too much caffeine, and missed this part of Travis' email: In this context, I'm especially interested in making sure that it's not just the developers who get to decide what happens to NumPy. Nathaniel has clarified very well what veto-power really means. It's not absolute, it just means that users who write clear arguments get listened to actively. It doesn't replace the need for developers with wisdom and understanding of user-experiences, but active listening is a useful skill that we could all improve on: http://en.wikipedia.org/wiki/Active_listeningA list full of bright, interested, active listeners is the kind of culture we need on this list. It's the kind of attitude we need from maintainers of NumPy. which mostly answers my worry, and I apologize for pushing on an open door. See you, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
On Wed, Apr 25, 2012 at 10:54 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Wed, Apr 25, 2012 at 2:35 PM, Travis Oliphant tra...@continuum.io wrote: Do you agree that Numpy has not been very successful in recruiting and maintaining new developers compared to its large user-base? Compared to - say - Sympy? Why do you think this is? I think it's mostly because it's infrastructure that is a means to an end. I certainly wasn't excited to have to work on NumPy originally, when my main interest was SciPy.I've come to love the interesting plateau that NumPy lives on.But, I think it mostly does the job it is supposed to do. The fact that it is in C is also not very sexy. It is also rather complicated with a lot of inter-related parts. I think NumPy could do much, much more --- but getting there is going to be a challenge of execution and education. You can get to know the code base. It just takes some time and patience. You also have to be comfortable with compilers and building software just to tweak the code. Would you consider asking that question directly on list and asking for the most honest possible answers? I'm always interested in honest answers and welcome any sincere perspective. Of course, there are potential explanations: 1) Numpy is too low-level for most people 2) The C code is too complicated 3) It's fine already, more or less are some obvious ones. I would say there are the easy answers. But of course, the easy answer may not be the right answer. It may not be easy to get right answer [1]. As you can see from Alan Isaac's reply on this thread, even asking the question can be taken as being in bad faith. In that situation, I think you'll find it hard to get sincere replies. While I don't think jumping into NumPy C code is as difficult as some people made it to be, I think numpy reaped most of the low-hanging fruits, and is now at a stage where it requires massive investment to get significantly better. I would suggest a different question, whose answer may serve as a proxy to uncover the lack of contributions: what needs to be done in NumPy, and how can we make it simpler for newcommers ? Here is an incomplete, unshamelessly biased list: - Less dependencies on CPython internals - Allow for 3rd parties to extend numpy at the C level in more fundamental ways (e.g. I wished something like half-float dtype could be more easily developed out of tree) - Separate memory representation from higher level representation (slicing, broadcasting, etc…), to allow arrays to sit on non-contiguous memory areas, etc… - Test and performance infrastructure so we can track our evolution, get coverage of our C code, etc… - Fix bugs - Better integration with 3rd party on-disk storage (database, etc…) None of that is particularly simple nor has a fast learning curve, except for fixing bugs and maybe some of the infrastructure. I think most of this is necessary for the things Travis talked about a few weeks ago. What could make contributions easier: - different levels of C API documentation (still lacking anything besides reference) - ways to detect early when we break ABI, slightly more obscure platforms (we need good CI, ways to publish binaries that people can easily test, etc...) - improve infrastructure so that we can focus on the things we want to work on (improve the dire situation with bug tracking, etc…) Also, lots of people just don't know/want to know C. But people with say web skills would be welcome: we have a website that could use some help… So ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
Hi, On Wed, Apr 25, 2012 at 3:24 PM, josef.p...@gmail.com wrote: On Wed, Apr 25, 2012 at 5:54 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Wed, Apr 25, 2012 at 2:35 PM, Travis Oliphant tra...@continuum.io wrote: Do you agree that Numpy has not been very successful in recruiting and maintaining new developers compared to its large user-base? Compared to - say - Sympy? Why do you think this is? I think it's mostly because it's infrastructure that is a means to an end. I certainly wasn't excited to have to work on NumPy originally, when my main interest was SciPy. I've come to love the interesting plateau that NumPy lives on. But, I think it mostly does the job it is supposed to do. The fact that it is in C is also not very sexy. It is also rather complicated with a lot of inter-related parts. I think NumPy could do much, much more --- but getting there is going to be a challenge of execution and education. You can get to know the code base. It just takes some time and patience. You also have to be comfortable with compilers and building software just to tweak the code. Would you consider asking that question directly on list and asking for the most honest possible answers? I'm always interested in honest answers and welcome any sincere perspective. Of course, there are potential explanations: 1) Numpy is too low-level for most people 2) The C code is too complicated 3) It's fine already, more or less are some obvious ones. I would say there are the easy answers. But of course, the easy answer may not be the right answer. It may not be easy to get right answer [1]. As you can see from Alan Isaac's reply on this thread, even asking the question can be taken as being in bad faith. In that situation, I think you'll find it hard to get sincere replies. I don't see why this shouldn't be the sincere replies, I think these easy answers are also the right answer for most people. I wasn't saying these replies are not sincere, of course they are factors. I have heard other people give reasons why they didn't enjoy numpy development much, but I can't speak for them, only for me. I have done some numpy development, but very little. I've done a moderate amount of scipy development. I have considered doing more numpy development, in particular, I did want to do some work on the longdouble parts of numpy. Part of the reason I didn't do this was because, when I raised the question on the list, it did not seem there was much interest in a change, or even a real discussion. Partly from the masked array discussions, but not only, it seemed that the process of making decisions was not clear, and there seemed to be as many views about how this was done as there were developers. I suppose I'd summarize the atmosphere, as I have have felt it, as being that numpy was owned by someone else, and I wasn't quite sure who that was, but I was fairly sure it wasn't me. On the other hand, in some projects at least - of which Sympy is the most obvious example, I think it's easy to feel that all of us own Sympy (and I've only made one commit to Sympy, and that of someone else's idea). Adding to that, it does seem to me that the atmosphere on this list get ugly sometimes. In particular it seems to me that there's a sort of conformity that starts to emerge in which people feel it is necessary to praise or criticize people, but not the arguments. I suppose that is because there was a long time during which Travis was not on the list to model what kind of discussion he wanted. I'm glad that has changed now. The reason I keep returning to process, even though it is 'non-technical' - is because it seems to me that the atmosphere that I'm describing will have the strong effect of discouraging enthusiastic developers. It certainly discourages me. I don't think open-source software is just developers scratching an itch, I think it's about community, and the pleasure of working with people you like and trust, to do something you think is important. If I've made that harder, then I am sorry, and I'm very happy to hear why that is, and how I can help. Best, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
On Wed, Apr 25, 2012 at 7:08 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Wed, Apr 25, 2012 at 3:24 PM, josef.p...@gmail.com wrote: On Wed, Apr 25, 2012 at 5:54 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Wed, Apr 25, 2012 at 2:35 PM, Travis Oliphant tra...@continuum.io wrote: Do you agree that Numpy has not been very successful in recruiting and maintaining new developers compared to its large user-base? Compared to - say - Sympy? Why do you think this is? I think it's mostly because it's infrastructure that is a means to an end. I certainly wasn't excited to have to work on NumPy originally, when my main interest was SciPy. I've come to love the interesting plateau that NumPy lives on. But, I think it mostly does the job it is supposed to do. The fact that it is in C is also not very sexy. It is also rather complicated with a lot of inter-related parts. I think NumPy could do much, much more --- but getting there is going to be a challenge of execution and education. You can get to know the code base. It just takes some time and patience. You also have to be comfortable with compilers and building software just to tweak the code. Would you consider asking that question directly on list and asking for the most honest possible answers? I'm always interested in honest answers and welcome any sincere perspective. Of course, there are potential explanations: 1) Numpy is too low-level for most people 2) The C code is too complicated 3) It's fine already, more or less are some obvious ones. I would say there are the easy answers. But of course, the easy answer may not be the right answer. It may not be easy to get right answer [1]. As you can see from Alan Isaac's reply on this thread, even asking the question can be taken as being in bad faith. In that situation, I think you'll find it hard to get sincere replies. I don't see why this shouldn't be the sincere replies, I think these easy answers are also the right answer for most people. I wasn't saying these replies are not sincere, of course they are factors. I have heard other people give reasons why they didn't enjoy numpy development much, but I can't speak for them, only for me. I have done some numpy development, but very little. I've done a moderate amount of scipy development. I have considered doing more numpy development, in particular, I did want to do some work on the longdouble parts of numpy. Part of the reason I didn't do this was because, when I raised the question on the list, it did not seem there was much interest in a change, or even a real discussion. Partly from the masked array discussions, but not only, it seemed that the process of making decisions was not clear, and there seemed to be as many views about how this was done as there were developers. I suppose I'd summarize the atmosphere, as I have have felt it, as being that numpy was owned by someone else, and I wasn't quite sure who that was, but I was fairly sure it wasn't me. On the other hand, in some projects at least - of which Sympy is the most obvious example, I think it's easy to feel that all of us own Sympy (and I've only made one commit to Sympy, and that of someone else's idea). Adding to that, it does seem to me that the atmosphere on this list get ugly sometimes. In particular it seems to me that there's a sort of conformity that starts to emerge in which people feel it is necessary to praise or criticize people, but not the arguments. I suppose that is because there was a long time during which Travis was not on the list to model what kind of discussion he wanted. I'm glad that has changed now. The reason I keep returning to process, even though it is 'non-technical' - is because it seems to me that the atmosphere that I'm describing will have the strong effect of discouraging enthusiastic developers. It certainly discourages me. I don't think open-source software is just developers scratching an itch, I think it's about community, and the pleasure of working with people you like and trust, to do something you think is important. Except for the big changes like NA and datetime, I think the debate is pretty boring. The main problem that I see for discussing technical issues is whether there are many developers really interested in commenting on code and coding. I think it mostly comes down to the discussion on tickets or pull requests. First my own experience with scipy.stats. Most of the time when I was cleaning up scipy.stats, I was alone, except for some helpful comments by Robert. My itch was that the bugs in scipy.stats were bugging me, and I just kept working and committing without code review until the bugs that I thought urgent were gone. Now, with Warren and Ralf also working on scipy.stats it is a lot more fun, since there is actually a regular community of 3 developers. My impression (since I only
Re: [Numpy-discussion] What is consensus anyway
On Apr 25, 2012, at 7:18 PM, josef.p...@gmail.com wrote: Except for the big changes like NA and datetime, I think the debate is pretty boring. The main problem that I see for discussing technical issues is whether there are many developers really interested in commenting on code and coding. I think it mostly comes down to the discussion on tickets or pull requests. This is a very insightful comment. Github has been a great thing for both NumPy and SciPy. However, it has changed the community feel for many because these pull request discussions don't happen on this list. You have to comment on a pull request to get notified of future comments or changes.The process is actually pretty nice, but it does mean you can't just hang out watching this list. You have to look at the pull requests and get involved there. It would be nice if every pull request created a message to this list.Is that even possible? -Travis ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
On Wednesday, April 25, 2012, Travis Oliphant wrote: On Apr 25, 2012, at 7:18 PM, josef.p...@gmail.com javascript:; wrote: Except for the big changes like NA and datetime, I think the debate is pretty boring. The main problem that I see for discussing technical issues is whether there are many developers really interested in commenting on code and coding. I think it mostly comes down to the discussion on tickets or pull requests. This is a very insightful comment. Github has been a great thing for both NumPy and SciPy. However, it has changed the community feel for many because these pull request discussions don't happen on this list. You have to comment on a pull request to get notified of future comments or changes.The process is actually pretty nice, but it does mean you can't just hang out watching this list. You have to look at the pull requests and get involved there. It would be nice if every pull request created a message to this list. Is that even possible? -Travis This ha been a concern of mine for matplotlib as well. The closest I can come is to set up an RSS feed, but all the titles are PR # and a action, so I lose track of which ones I want to view. All devs get an initial email for each PR, but I cant figure out how to get that down to the public list and it is hard to know if another dev took care of the PR or if it is just waiting. Cheers! Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
On 4/25/12 8:11 PM, Travis Oliphant wrote: On Apr 25, 2012, at 7:18 PM, josef.p...@gmail.com wrote: Except for the big changes like NA and datetime, I think the debate is pretty boring. The main problem that I see for discussing technical issues is whether there are many developers really interested in commenting on code and coding. I think it mostly comes down to the discussion on tickets or pull requests. This is a very insightful comment. Github has been a great thing for both NumPy and SciPy. However, it has changed the community feel for many because these pull request discussions don't happen on this list. You have to comment on a pull request to get notified of future comments or changes.The process is actually pretty nice, but it does mean you can't just hang out watching this list. You have to look at the pull requests and get involved there. It would be nice if every pull request created a message to this list.Is that even possible? Sure. Github has a pretty extensive hook system that can notify (via hitting a URL) about lots of events. https://github.com/blog/964-all-of-the-hooks http://developer.github.com/v3/repos/hooks/ I haven't actually used it (just read the docs), so I may be mistaken... Thanks, Jason ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
On Thu, Apr 26, 2012 at 6:41 AM, Travis Oliphant tra...@continuum.io wrote: [snip] It would be nice if every pull request created a message to this list. Is that even possible? That is definitely possible and shouldn't be too hard to do, like Jason said. But that can potentially cause some confusion, with some of the discussion starting off in the mailing list, and some of the discussion happening on the pull-request itself. Are my concerns justified? -- Puneeth ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
On 4/25/12 11:08 PM, Puneeth Chaganti wrote: On Thu, Apr 26, 2012 at 6:41 AM, Travis Oliphanttra...@continuum.io wrote: [snip] It would be nice if every pull request created a message to this list.Is that even possible? That is definitely possible and shouldn't be too hard to do, like Jason said. But that can potentially cause some confusion, with some of the discussion starting off in the mailing list, and some of the discussion happening on the pull-request itself. Are my concerns justified? It wouldn't be too hard to have mailing list replies sent back to the pull request as comments (again, using the github API). Already, if you're on a ticket, you can just reply to a comment email and the reply is put as a comment in the pull request. Jason ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
On Wed, Apr 25, 2012 at 11:08 PM, Puneeth Chaganti puncha...@gmail.com wrote: On Thu, Apr 26, 2012 at 6:41 AM, Travis Oliphant tra...@continuum.io wrote: [snip] It would be nice if every pull request created a message to this list. Is that even possible? That is definitely possible and shouldn't be too hard to do, like Jason said. But that can potentially cause some confusion, with some of the discussion starting off in the mailing list, and some of the discussion happening on the pull-request itself. Are my concerns justified? Related issue: some projects have an user's list and a devel list. It might be worth (re?)considering that option. They have their pros and cons but I think I like the idea of a devel list and seperate help wanted list. Something else that might be helpful for contentious threads is a stack-overflowesque system where readers can vote up responses of others. Sometimes just a i agree i disagree goes a long way, especially when you have many lurkers. On something else that was brought up: I do not consider myself competent/prepared enough to take on development, but it is not the case that I have _never_ felt the temptation. What I have found intimidating and styming is the perceived politics over development issues. The two places where I have felt this are a) on contentious threads on the list and b) what seems like legitimate patches tickets on trac that seem to be languishing for no compelling technical reason. I would be hardpressed to quote specifics, but I have encountered this feeling a few times. For my case it would not have mattered, because I doubt I would have contriuted anything useful. However, it might be the case that more competent lurkers might have felt the same way. The possibility of a patch relegated semipermanently to trac, or the possibility of getting caught up in the politics is bit of a disincentive. This is just an honest perception/observation. I am more of a get on with it, get the code out and rest will resolve itself eventually kind of a guy, thus long political/philosophical/epistemic threads distance me. I know there are legitimate reasons to have this discussions. But it seems to me that they get a bit too wordy here sometimes. My 10E-2. -- srean ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What is consensus anyway
On Wed, Apr 25, 2012 at 6:28 PM, Benjamin Root ben.r...@ou.edu wrote: It would be nice if every pull request created a message to this list. Is that even possible? -Travis This ha been a concern of mine for matplotlib as well. The closest I can come is to set up an RSS feed, but all the titles are PR # and a action, so I lose track of which ones I want to view. Same here for IPython. If anybody figures out a clean solution, please advertise it! I think a bunch of us want the same thing... Cheers, f ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
