Re: [Numpy-discussion] Numpy FFT.FFT slow with certain samples
On Aug 28, 2015 5:17 PM, "Pierre-Andre Noel" wrote: > > I had in mind the use of FFT to do convolutions ( > https://en.wikipedia.org/wiki/Convolution_theorem ). If you do not > zero-pad properly, then the end of the signal may "bleed" on the > beginning, and vice versa. Ah, gotcha! All these things should also be handled nicely in scipy.signal.fftconvolve. Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy FFT.FFT slow with certain samples
> Zero-padding won't help with the non-periodicity, will it? For > that you may want to window instead. Umh, it depends what you use the FFT for. You are right Stéfan when saying that Joseph should probably also use a window to get rid of the high frequencies that will come from the sharp steps at the beginning and end of his signal. I had in mind the use of FFT to do convolutions ( https://en.wikipedia.org/wiki/Convolution_theorem ). If you do not zero-pad properly, then the end of the signal may "bleed" on the beginning, and vice versa. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Fri, Aug 28, 2015 at 3:36 PM, Jaime Fernández del Río < jaime.f...@gmail.com> wrote: > On Fri, Aug 28, 2015 at 2:40 AM, Sebastian Berg < > sebast...@sipsolutions.net> wrote: > >> On Fr, 2015-08-28 at 09:46 +0100, Matthew Brett wrote: >> > Hi, >> > >> > On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fernández del Río >> > wrote: >> > > On Thu, Aug 27, 2015 at 11:06 AM, Matthew Brett < >> matthew.br...@gmail.com> >> > > wrote: >> > >> >> > >> Hi, >> > >> >> > >> On Thu, Aug 27, 2015 at 6:23 PM, wrote: >> > >> > >> > >> > >> > >> > On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett >> > >> > >> > >> > wrote: >> > >> >> >> > >> >> Hi >> > >> >> >> > >> >> On Thu, Aug 27, 2015 at 5:11 PM, wrote: >> > >> >> > >> > >> >> > >> > >> >> > On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett >> > >> >> > >> > >> >> > wrote: >> > >> >> >> >> > >> >> >> Hi, >> > >> >> >> >> > >> >> >> On Thu, Aug 27, 2015 at 3:34 PM, >> wrote: >> > >> >> >> [snip] >> > >> >> >> > I don't really see a problem with "codifying" the status quo. >> > >> >> >> >> > >> >> >> That's an excellent point.If we believe that the current >> > >> >> >> situation >> > >> >> >> is the best possible, both now and in the future, then >> codifying the >> > >> >> >> status quo is an excellent idea. >> > >> >> >> >> > >> >> >> So, we should probably first start by asking ourselves: >> > >> >> >> >> > >> >> >> * what numpy is doing well; >> > >> >> >> * what numpy could do better; >> > >> >> >> >> > >> >> >> and then ask, is there some way we could make it more likely >> we will >> > >> >> >> improve over time. >> > >> >> >> >> > >> >> >> [snip] >> > >> >> >> >> > >> >> >> > As the current debate shows it's possible to have a public >> > >> >> >> > discussion >> > >> >> >> > about >> > >> >> >> > the direction of the project without having to delegate >> providing >> > >> >> >> > a >> > >> >> >> > vision >> > >> >> >> > to a president. >> > >> >> >> >> > >> >> >> The idea of a president that I had in mind, was not someone who >> > >> >> >> makes >> > >> >> >> all decisions, but the person who holds themselves responsible >> for >> > >> >> >> the >> > >> >> >> performance of the project. If the project has a coherent >> vision >> > >> >> >> already, the president has no need to provide one, but it's the >> > >> >> >> president's job to worry about whether we have vision or not, >> and do >> > >> >> >> what they need to, to make sure we don't lose track of that. >> If >> > >> >> >> you >> > >> >> >> don't know it already, I highly recommend Jim Collins' work on >> > >> >> >> 'level >> > >> >> >> 5 leadership' [1] >> > >> >> > >> > >> >> > >> > >> >> > Still doesn't sound like the need for a president to me >> > >> >> > >> > >> >> > " the person who holds themselves responsible for the >> > >> >> > performance of the project" >> > >> >> > >> > >> >> > sounds more like the role of the "core" group (adding plural to >> > >> >> > persons) >> > >> >> > to >> > >> >> > me, and cannot be pushed of to an official president. >> > >> >> >> > >> >> Except that, in the past, having multiple people taking decisions >> has >> > >> >> led to the situation where no-one feels themselves accountable >> for the >> > >> >> result, hence this situation tends to lead to stagnation. >> > >> > >> > >> > >> > >> > Is there any evidence for this? >> > >> >> > >> Oh - dear - that's the key point, but I'm obviously not making it >> > >> clearly enough. Yes there is, and that was the evidence I was >> > >> pointing to before. >> > >> >> > >> But anyway - Sebastian is right - this discussion isn't going >> anywhere >> > >> useful. >> > >> >> > >> So - let's step back. >> > >> >> > >> In thinking about governance, we first need to ask what we want to >> > >> achieve. This includes considering the risks ahead for the project. >> > >> >> > >> So, in the spirit of fruitful discussion, can I ask what y'all >> > >> consider to be the current problems with working on numpy (other than >> > >> the technical ones). What is numpy doing well, and what is it doing >> > >> badly? What risks do we have to plan for in the future? >> > > >> > > >> > > Are you trying to prove the point that consensus doesn't work by >> making it >> > > impossible to reach a consensus on this? ;-) >> > > >> > >> > Forgive me if I use this joke to see if I can get us any further. >> > >> > If this was code, I think this joke would not be funny, because we >> > wouldn't expect to reach consensus without considering all the >> > options, and discussing their pros and cons. >> > >> > Why would that not be useful in the case of forms of governance? >> > >> >> Oh, it is true. I think we (those in the room in Austin) just have >> thought about it a bit already, so now we have to be a bit patient with >> everyone who just saw the plans the first time. But I hope we can agree >> that we should decide on some form of governance in the next few weeks, >> even if it may not be perfect. >> >> My personal problem with your ideas
Re: [Numpy-discussion] Numpy FFT.FFT slow with certain samples
On 2015-08-28 16:20:33, Pierre-Andre Noel wrote: > If your sequence is not meant to be periodic (i.e., if after one > minute there is no reason why the signal should start back at > the beginning right away), then you should do zero-padding. And > while you zero-pad, you can zero-pad to a sequence that is a > power of two, thus preventing awkward factorizations. Zero-padding won't help with the non-periodicity, will it? For that you may want to window instead. Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy FFT.FFT slow with certain samples
If your sequence is not meant to be periodic (i.e., if after one minute there is no reason why the signal should start back at the beginning right away), then you should do zero-padding. And while you zero-pad, you can zero-pad to a sequence that is a power of two, thus preventing awkward factorizations. from numpy.fft import fft from numpy.random import rand from math import log, ceil seq_A = rand(2649674) seq_B = rand(2646070) fft_A = fft(seq_A) #Long fft_B = fft(seq_B) zeropadded_fft_A = fft(seq_A, n=2**(ceil(log(len(seq_A),2))+1)) zeropadded_fft_B = fft(seq_B, n=2**(ceil(log(len(seq_B),2))+1)) You could remove the "+1" above to get faster results, but then that may lead to unwanted frequencies (coming from the fact that fft assumes periodic signals, read online about zero-padding). Have a nice day, Pierre-André On 08/28/2015 12:03 PM, Stefan van der Walt wrote: > > On 2015-08-28 11:51:47, Joseph Codadeen wrote: >> my_1_minute_noise_with_gaps_truncated - Array len is >> 2646070my_1_minute_noise_with_gaps - Array len is 2649674 > > In [6]: from sympy import factorint In [7]: max(factorint(2646070)) > Out[7]: 367 In [8]: max(factorint(2649674)) Out[8]: 1324837 > Those numbers give you some indication of how long the FFT will take > to compute. > > Stéfan > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Fri, Aug 28, 2015 at 2:40 AM, Sebastian Berg wrote: > On Fr, 2015-08-28 at 09:46 +0100, Matthew Brett wrote: > > Hi, > > > > On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fernández del Río > > wrote: > > > On Thu, Aug 27, 2015 at 11:06 AM, Matthew Brett < > matthew.br...@gmail.com> > > > wrote: > > >> > > >> Hi, > > >> > > >> On Thu, Aug 27, 2015 at 6:23 PM, wrote: > > >> > > > >> > > > >> > On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett > > >> > > > >> > wrote: > > >> >> > > >> >> Hi > > >> >> > > >> >> On Thu, Aug 27, 2015 at 5:11 PM, wrote: > > >> >> > > > >> >> > > > >> >> > On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett > > >> >> > > > >> >> > wrote: > > >> >> >> > > >> >> >> Hi, > > >> >> >> > > >> >> >> On Thu, Aug 27, 2015 at 3:34 PM, wrote: > > >> >> >> [snip] > > >> >> >> > I don't really see a problem with "codifying" the status quo. > > >> >> >> > > >> >> >> That's an excellent point.If we believe that the current > > >> >> >> situation > > >> >> >> is the best possible, both now and in the future, then > codifying the > > >> >> >> status quo is an excellent idea. > > >> >> >> > > >> >> >> So, we should probably first start by asking ourselves: > > >> >> >> > > >> >> >> * what numpy is doing well; > > >> >> >> * what numpy could do better; > > >> >> >> > > >> >> >> and then ask, is there some way we could make it more likely we > will > > >> >> >> improve over time. > > >> >> >> > > >> >> >> [snip] > > >> >> >> > > >> >> >> > As the current debate shows it's possible to have a public > > >> >> >> > discussion > > >> >> >> > about > > >> >> >> > the direction of the project without having to delegate > providing > > >> >> >> > a > > >> >> >> > vision > > >> >> >> > to a president. > > >> >> >> > > >> >> >> The idea of a president that I had in mind, was not someone who > > >> >> >> makes > > >> >> >> all decisions, but the person who holds themselves responsible > for > > >> >> >> the > > >> >> >> performance of the project. If the project has a coherent > vision > > >> >> >> already, the president has no need to provide one, but it's the > > >> >> >> president's job to worry about whether we have vision or not, > and do > > >> >> >> what they need to, to make sure we don't lose track of that. > If > > >> >> >> you > > >> >> >> don't know it already, I highly recommend Jim Collins' work on > > >> >> >> 'level > > >> >> >> 5 leadership' [1] > > >> >> > > > >> >> > > > >> >> > Still doesn't sound like the need for a president to me > > >> >> > > > >> >> > " the person who holds themselves responsible for the > > >> >> > performance of the project" > > >> >> > > > >> >> > sounds more like the role of the "core" group (adding plural to > > >> >> > persons) > > >> >> > to > > >> >> > me, and cannot be pushed of to an official president. > > >> >> > > >> >> Except that, in the past, having multiple people taking decisions > has > > >> >> led to the situation where no-one feels themselves accountable for > the > > >> >> result, hence this situation tends to lead to stagnation. > > >> > > > >> > > > >> > Is there any evidence for this? > > >> > > >> Oh - dear - that's the key point, but I'm obviously not making it > > >> clearly enough. Yes there is, and that was the evidence I was > > >> pointing to before. > > >> > > >> But anyway - Sebastian is right - this discussion isn't going anywhere > > >> useful. > > >> > > >> So - let's step back. > > >> > > >> In thinking about governance, we first need to ask what we want to > > >> achieve. This includes considering the risks ahead for the project. > > >> > > >> So, in the spirit of fruitful discussion, can I ask what y'all > > >> consider to be the current problems with working on numpy (other than > > >> the technical ones). What is numpy doing well, and what is it doing > > >> badly? What risks do we have to plan for in the future? > > > > > > > > > Are you trying to prove the point that consensus doesn't work by > making it > > > impossible to reach a consensus on this? ;-) > > > > > > > Forgive me if I use this joke to see if I can get us any further. > > > > If this was code, I think this joke would not be funny, because we > > wouldn't expect to reach consensus without considering all the > > options, and discussing their pros and cons. > > > > Why would that not be useful in the case of forms of governance? > > > > Oh, it is true. I think we (those in the room in Austin) just have > thought about it a bit already, so now we have to be a bit patient with > everyone who just saw the plans the first time. But I hope we can agree > that we should decide on some form of governance in the next few weeks, > even if it may not be perfect. > > My personal problem with your ideas is not that I do not care for the > warnings, but having already spend some time trying to put together this > (and this is nothing weird, this is very common practice in open > source), I personally do not want to spend time inventing something > completely new.
Re: [Numpy-discussion] Numpy FFT.FFT slow with certain samples
On 2015/08/28 10:36 AM, Sebastian Berg wrote: > If you don't mind the extra dependency or licensing and this is an issue > for you, you can try pyfftw (there are likely other similar projects) > which wraps fftw and does not have this problem as far as I know. It > exposes a numpy-like interface. Sort of; that interface returns a function, not the result. fftw is still an fft algorithm, so it is still subject to a huge difference in run time depending on how the input array can be factored. Furthermore, it gets its speed by figuring out how to optimize a calculation for a given size of input array. That initial optimization can be very slow. The overall speed gain is realized only when one saves the result of that optimization, and applies it to many calculations on arrays of the same size. Eric > > - sebastian > > > On Fr, 2015-08-28 at 19:13 +, Joseph Codadeen wrote: >> Great, thanks Stefan and everyone. >> >>> From: stef...@berkeley.edu >>> To: numpy-discussion@scipy.org >>> Date: Fri, 28 Aug 2015 12:03:52 -0700 >>> Subject: Re: [Numpy-discussion] Numpy FFT.FFT slow with certain >> samples >>> >>> >>> On 2015-08-28 11:51:47, Joseph Codadeen >>> wrote: my_1_minute_noise_with_gaps_truncated - Array len is 2646070my_1_minute_noise_with_gaps - Array len is 2649674 >>> >>> In [6]: from sympy import factorint In [7]: >>> max(factorint(2646070)) Out[7]: 367 In [8]: >>> max(factorint(2649674)) Out[8]: 1324837 >>> >>> Those numbers give you some indication of how long the FFT will >>> take to compute. >>> >>> Stéfan >>> ___ >>> NumPy-Discussion mailing list >>> NumPy-Discussion@scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy FFT.FFT slow with certain samples
If you don't mind the extra dependency or licensing and this is an issue for you, you can try pyfftw (there are likely other similar projects) which wraps fftw and does not have this problem as far as I know. It exposes a numpy-like interface. - sebastian On Fr, 2015-08-28 at 19:13 +, Joseph Codadeen wrote: > Great, thanks Stefan and everyone. > > > From: stef...@berkeley.edu > > To: numpy-discussion@scipy.org > > Date: Fri, 28 Aug 2015 12:03:52 -0700 > > Subject: Re: [Numpy-discussion] Numpy FFT.FFT slow with certain > samples > > > > > > On 2015-08-28 11:51:47, Joseph Codadeen > > wrote: > > > my_1_minute_noise_with_gaps_truncated - Array len is > > > 2646070my_1_minute_noise_with_gaps - Array len is 2649674 > > > > In [6]: from sympy import factorint In [7]: > > max(factorint(2646070)) Out[7]: 367 In [8]: > > max(factorint(2649674)) Out[8]: 1324837 > > > > Those numbers give you some indication of how long the FFT will > > take to compute. > > > > Stéfan > > ___ > > NumPy-Discussion mailing list > > NumPy-Discussion@scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy FFT.FFT slow with certain samples
Great, thanks Stefan and everyone. > From: stef...@berkeley.edu > To: numpy-discussion@scipy.org > Date: Fri, 28 Aug 2015 12:03:52 -0700 > Subject: Re: [Numpy-discussion] Numpy FFT.FFT slow with certain samples > > > On 2015-08-28 11:51:47, Joseph Codadeen > wrote: > > my_1_minute_noise_with_gaps_truncated - Array len is > > 2646070my_1_minute_noise_with_gaps - Array len is 2649674 > > In [6]: from sympy import factorint In [7]: > max(factorint(2646070)) Out[7]: 367 In [8]: > max(factorint(2649674)) Out[8]: 1324837 > > Those numbers give you some indication of how long the FFT will > take to compute. > > Stéfan > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy FFT.FFT slow with certain samples
On 2015-08-28 11:51:47, Joseph Codadeen wrote: > my_1_minute_noise_with_gaps_truncated - Array len is > 2646070my_1_minute_noise_with_gaps - Array len is 2649674 In [6]: from sympy import factorint In [7]: max(factorint(2646070)) Out[7]: 367 In [8]: max(factorint(2649674)) Out[8]: 1324837 Those numbers give you some indication of how long the FFT will take to compute. Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy FFT.FFT slow with certain samples
my_1_minute_noise_with_gaps_truncated - Array len is 2646070my_1_minute_noise_with_gaps - Array len is 2649674 > Date: Fri, 28 Aug 2015 14:28:49 -0400 > From: ho...@stsci.edu > To: numpy-discussion@scipy.org > Subject: Re: [Numpy-discussion] Numpy FFT.FFT slow with certain samples > > On 08/28/2015 02:02 PM, Joseph Codadeen wrote: > > > > * my_1_minute_noise_with_gaps_truncated took***30.75620985s* to process. > > * my_1_minute_noise_with_gaps took *22307.13917s*to process. > > > > You didn't say how long those arrays were, but I can make a good guess > that the truncated one had a length that could be factored into small, > prime numbers, while the non-truncated one had a length that was either > prime or could only be factored into large primes. > > Phil > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpty FFT.FFT slow with certain samples
On Fri, Aug 28, 2015 at 11:02 AM, Joseph Codadeen wrote: > Hi, > > I am a numpy newbie. > > I have two wav files, one that numpy takes a long time to process the FFT. > They was created within audacity using white noise and silence for gaps. > > >1. my_1_minute_noise_with_gaps.wav >2. my_1_minute_noise_with_gaps_truncated.wav > > > The files are very similar in the following way; > > >- 1. is white noise with silence gaps on every 15 second interval. >- 2. is 1. but slightly shorter, i.e. I trimmed some ms off the end >but it still has the last gap at 60s. > > > The code I am using processes the file like this; > > framerate, data = scipy.io.wavfile.read(filepath) > right = data[:, 0] > # Align it to be efficient. > if len(right) % 2 != 0: > right = right[range(len(right) - 1)] > noframes = len(right) > fftout = np.fft.fft(right) / noframes# <<< I am timing this cmd > > Using timeit... > > >- my_1_minute_noise_with_gaps_truncated took *30.75620985s* to process. >- my_1_minute_noise_with_gaps took *22307.13917s* to process. > > > Could someone tell me why this behaviour is happening please? > > Sorry I can't attach the files as this email gets bounced but you could > easily create the files yourself. > E.g my last gap width is 59.9995 - 1:00.0005, I repeat this every 15 > seconds. > My truncated file is 1:00.0015s long, non-truncated is 1:00.0833s long > It is almost certainly caused by the number of samples in your signals, i.e. look at what noframes is in one case and the other. You will get best performance when noframes is a power of two, or has a factorization that includes many small integers (2, 3, 5, perhaps also 7), and the worst if the size is a prime number. Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy FFT.FFT slow with certain samples
On 08/28/2015 02:02 PM, Joseph Codadeen wrote: > > * my_1_minute_noise_with_gaps_truncated took***30.75620985s* to process. > * my_1_minute_noise_with_gaps took *22307.13917s*to process. > You didn't say how long those arrays were, but I can make a good guess that the truncated one had a length that could be factored into small, prime numbers, while the non-truncated one had a length that was either prime or could only be factored into large primes. Phil ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Numpty FFT.FFT slow with certain samples
Hi, I am a numpy newbie. I have two wav files, one that numpy takes a long time to process the FFT. They was created within audacity using white noise and silence for gaps. my_1_minute_noise_with_gaps.wavmy_1_minute_noise_with_gaps_truncated.wav The files are very similar in the following way; 1. is white noise with silence gaps on every 15 second interval.2. is 1. but slightly shorter, i.e. I trimmed some ms off the end but it still has the last gap at 60s. The code I am using processes the file like this; framerate, data = scipy.io.wavfile.read(filepath)right = data[:, 0] # Align it to be efficient.if len(right) % 2 != 0:right = right[range(len(right) - 1)]noframes = len(right)fftout = np.fft.fft(right) / noframes# <<< I am timing this cmd Using timeit... my_1_minute_noise_with_gaps_truncated took 30.75620985s to process.my_1_minute_noise_with_gaps took 22307.13917s to process. Could someone tell me why this behaviour is happening please? Sorry I can't attach the files as this email gets bounced but you could easily create the files yourself.E.g my last gap width is 59.9995 - 1:00.0005, I repeat this every 15 seconds.My truncated file is 1:00.0015s long, non-truncated is 1:00.0833s long Thanks. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
On Fr, 2015-08-28 at 09:46 +0100, Matthew Brett wrote: > Hi, > > On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fernández del Río > wrote: > > On Thu, Aug 27, 2015 at 11:06 AM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Thu, Aug 27, 2015 at 6:23 PM, wrote: > >> > > >> > > >> > On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett > >> > > >> > wrote: > >> >> > >> >> Hi > >> >> > >> >> On Thu, Aug 27, 2015 at 5:11 PM, wrote: > >> >> > > >> >> > > >> >> > On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett > >> >> > > >> >> > wrote: > >> >> >> > >> >> >> Hi, > >> >> >> > >> >> >> On Thu, Aug 27, 2015 at 3:34 PM, wrote: > >> >> >> [snip] > >> >> >> > I don't really see a problem with "codifying" the status quo. > >> >> >> > >> >> >> That's an excellent point.If we believe that the current > >> >> >> situation > >> >> >> is the best possible, both now and in the future, then codifying the > >> >> >> status quo is an excellent idea. > >> >> >> > >> >> >> So, we should probably first start by asking ourselves: > >> >> >> > >> >> >> * what numpy is doing well; > >> >> >> * what numpy could do better; > >> >> >> > >> >> >> and then ask, is there some way we could make it more likely we will > >> >> >> improve over time. > >> >> >> > >> >> >> [snip] > >> >> >> > >> >> >> > As the current debate shows it's possible to have a public > >> >> >> > discussion > >> >> >> > about > >> >> >> > the direction of the project without having to delegate providing > >> >> >> > a > >> >> >> > vision > >> >> >> > to a president. > >> >> >> > >> >> >> The idea of a president that I had in mind, was not someone who > >> >> >> makes > >> >> >> all decisions, but the person who holds themselves responsible for > >> >> >> the > >> >> >> performance of the project. If the project has a coherent vision > >> >> >> already, the president has no need to provide one, but it's the > >> >> >> president's job to worry about whether we have vision or not, and do > >> >> >> what they need to, to make sure we don't lose track of that. If > >> >> >> you > >> >> >> don't know it already, I highly recommend Jim Collins' work on > >> >> >> 'level > >> >> >> 5 leadership' [1] > >> >> > > >> >> > > >> >> > Still doesn't sound like the need for a president to me > >> >> > > >> >> > " the person who holds themselves responsible for the > >> >> > performance of the project" > >> >> > > >> >> > sounds more like the role of the "core" group (adding plural to > >> >> > persons) > >> >> > to > >> >> > me, and cannot be pushed of to an official president. > >> >> > >> >> Except that, in the past, having multiple people taking decisions has > >> >> led to the situation where no-one feels themselves accountable for the > >> >> result, hence this situation tends to lead to stagnation. > >> > > >> > > >> > Is there any evidence for this? > >> > >> Oh - dear - that's the key point, but I'm obviously not making it > >> clearly enough. Yes there is, and that was the evidence I was > >> pointing to before. > >> > >> But anyway - Sebastian is right - this discussion isn't going anywhere > >> useful. > >> > >> So - let's step back. > >> > >> In thinking about governance, we first need to ask what we want to > >> achieve. This includes considering the risks ahead for the project. > >> > >> So, in the spirit of fruitful discussion, can I ask what y'all > >> consider to be the current problems with working on numpy (other than > >> the technical ones). What is numpy doing well, and what is it doing > >> badly? What risks do we have to plan for in the future? > > > > > > Are you trying to prove the point that consensus doesn't work by making it > > impossible to reach a consensus on this? ;-) > > > > Forgive me if I use this joke to see if I can get us any further. > > If this was code, I think this joke would not be funny, because we > wouldn't expect to reach consensus without considering all the > options, and discussing their pros and cons. > > Why would that not be useful in the case of forms of governance? > Oh, it is true. I think we (those in the room in Austin) just have thought about it a bit already, so now we have to be a bit patient with everyone who just saw the plans the first time. But I hope we can agree that we should decide on some form of governance in the next few weeks, even if it may not be perfect. My personal problem with your ideas is not that I do not care for the warnings, but having already spend some time trying to put together this (and this is nothing weird, this is very common practice in open source), I personally do not want to spend time inventing something completely new. We must discuss improvements to the document, and even whole different approaches. But for me at least, I need something a little more specific. Maybe I am daft, but I hear "this is a bad idea" without also providing another approach (that seems doable). And I do not buy that it is *that* bad, it is a very common governance structure for open source
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
Hi, On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fernández del Río wrote: > On Thu, Aug 27, 2015 at 11:06 AM, Matthew Brett > wrote: >> >> Hi, >> >> On Thu, Aug 27, 2015 at 6:23 PM, wrote: >> > >> > >> > On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett >> > >> > wrote: >> >> >> >> Hi >> >> >> >> On Thu, Aug 27, 2015 at 5:11 PM, wrote: >> >> > >> >> > >> >> > On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett >> >> > >> >> > wrote: >> >> >> >> >> >> Hi, >> >> >> >> >> >> On Thu, Aug 27, 2015 at 3:34 PM, wrote: >> >> >> [snip] >> >> >> > I don't really see a problem with "codifying" the status quo. >> >> >> >> >> >> That's an excellent point.If we believe that the current >> >> >> situation >> >> >> is the best possible, both now and in the future, then codifying the >> >> >> status quo is an excellent idea. >> >> >> >> >> >> So, we should probably first start by asking ourselves: >> >> >> >> >> >> * what numpy is doing well; >> >> >> * what numpy could do better; >> >> >> >> >> >> and then ask, is there some way we could make it more likely we will >> >> >> improve over time. >> >> >> >> >> >> [snip] >> >> >> >> >> >> > As the current debate shows it's possible to have a public >> >> >> > discussion >> >> >> > about >> >> >> > the direction of the project without having to delegate providing >> >> >> > a >> >> >> > vision >> >> >> > to a president. >> >> >> >> >> >> The idea of a president that I had in mind, was not someone who >> >> >> makes >> >> >> all decisions, but the person who holds themselves responsible for >> >> >> the >> >> >> performance of the project. If the project has a coherent vision >> >> >> already, the president has no need to provide one, but it's the >> >> >> president's job to worry about whether we have vision or not, and do >> >> >> what they need to, to make sure we don't lose track of that. If >> >> >> you >> >> >> don't know it already, I highly recommend Jim Collins' work on >> >> >> 'level >> >> >> 5 leadership' [1] >> >> > >> >> > >> >> > Still doesn't sound like the need for a president to me >> >> > >> >> > " the person who holds themselves responsible for the >> >> > performance of the project" >> >> > >> >> > sounds more like the role of the "core" group (adding plural to >> >> > persons) >> >> > to >> >> > me, and cannot be pushed of to an official president. >> >> >> >> Except that, in the past, having multiple people taking decisions has >> >> led to the situation where no-one feels themselves accountable for the >> >> result, hence this situation tends to lead to stagnation. >> > >> > >> > Is there any evidence for this? >> >> Oh - dear - that's the key point, but I'm obviously not making it >> clearly enough. Yes there is, and that was the evidence I was >> pointing to before. >> >> But anyway - Sebastian is right - this discussion isn't going anywhere >> useful. >> >> So - let's step back. >> >> In thinking about governance, we first need to ask what we want to >> achieve. This includes considering the risks ahead for the project. >> >> So, in the spirit of fruitful discussion, can I ask what y'all >> consider to be the current problems with working on numpy (other than >> the technical ones). What is numpy doing well, and what is it doing >> badly? What risks do we have to plan for in the future? > > > Are you trying to prove the point that consensus doesn't work by making it > impossible to reach a consensus on this? ;-) > Forgive me if I use this joke to see if I can get us any further. If this was code, I think this joke would not be funny, because we wouldn't expect to reach consensus without considering all the options, and discussing their pros and cons. Why would that not be useful in the case of forms of governance? One reason might be that the specific form of governance can have no influence on the long-term health of the project. I am convinced that that is wrong - that the form of governance has a large influence on the long-term health of a project. If there is some possibility that this is true, then it seems to me that we would be foolish not to try and come to some reasoned choice about the form of governance. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
+10 Very well written down ideas Jaime. 2015-08-28 6:59 GMT+02:00 Jaime Fernández del Río : > On Thu, Aug 27, 2015 at 11:06 AM, Matthew Brett > wrote: > >> Hi, >> >> On Thu, Aug 27, 2015 at 6:23 PM, wrote: >> > >> > >> > On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett < >> matthew.br...@gmail.com> >> > wrote: >> >> >> >> Hi >> >> >> >> On Thu, Aug 27, 2015 at 5:11 PM, wrote: >> >> > >> >> > >> >> > On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett >> >> > >> >> > wrote: >> >> >> >> >> >> Hi, >> >> >> >> >> >> On Thu, Aug 27, 2015 at 3:34 PM, wrote: >> >> >> [snip] >> >> >> > I don't really see a problem with "codifying" the status quo. >> >> >> >> >> >> That's an excellent point.If we believe that the current >> situation >> >> >> is the best possible, both now and in the future, then codifying the >> >> >> status quo is an excellent idea. >> >> >> >> >> >> So, we should probably first start by asking ourselves: >> >> >> >> >> >> * what numpy is doing well; >> >> >> * what numpy could do better; >> >> >> >> >> >> and then ask, is there some way we could make it more likely we will >> >> >> improve over time. >> >> >> >> >> >> [snip] >> >> >> >> >> >> > As the current debate shows it's possible to have a public >> discussion >> >> >> > about >> >> >> > the direction of the project without having to delegate providing >> a >> >> >> > vision >> >> >> > to a president. >> >> >> >> >> >> The idea of a president that I had in mind, was not someone who >> makes >> >> >> all decisions, but the person who holds themselves responsible for >> the >> >> >> performance of the project. If the project has a coherent vision >> >> >> already, the president has no need to provide one, but it's the >> >> >> president's job to worry about whether we have vision or not, and do >> >> >> what they need to, to make sure we don't lose track of that. If >> you >> >> >> don't know it already, I highly recommend Jim Collins' work on >> 'level >> >> >> 5 leadership' [1] >> >> > >> >> > >> >> > Still doesn't sound like the need for a president to me >> >> > >> >> > " the person who holds themselves responsible for the >> >> > performance of the project" >> >> > >> >> > sounds more like the role of the "core" group (adding plural to >> persons) >> >> > to >> >> > me, and cannot be pushed of to an official president. >> >> >> >> Except that, in the past, having multiple people taking decisions has >> >> led to the situation where no-one feels themselves accountable for the >> >> result, hence this situation tends to lead to stagnation. >> > >> > >> > Is there any evidence for this? >> >> Oh - dear - that's the key point, but I'm obviously not making it >> clearly enough. Yes there is, and that was the evidence I was >> pointing to before. >> >> But anyway - Sebastian is right - this discussion isn't going anywhere >> useful. >> >> So - let's step back. >> >> In thinking about governance, we first need to ask what we want to >> achieve. This includes considering the risks ahead for the project. >> >> So, in the spirit of fruitful discussion, can I ask what y'all >> consider to be the current problems with working on numpy (other than >> the technical ones). What is numpy doing well, and what is it doing >> badly? What risks do we have to plan for in the future? >> > > > Are you trying to prove the point that consensus doesn't work by making it > impossible to reach a consensus on this? ;-) > > > One thing we are doing very badly is leveraging resources outside of > contributions of work and time from individuals. Getting sponsors to > finance work on what is the cornerstone of just about any Python package > that has to add two numbers together shouldn't be too hard, especially > seeing success stories like Jupyter's, who I believe has several paid > developers working full time. That requires formalizing governance, > because apparently sponsors are a little wary of giving money to "people on > the internet". ;-) Fernando Pérez was extremely emphatic about the size of > the opportunity NumPy was letting slip by not formalizing *any* governance > model. And it is a necessary first step so that e.g. we have the money to, > say a year from now, get the right people together for a couple of days to > figure out a better governance model. I'd argue that money would be better > spent financing a talented developer to advance e.g. Nathaniel's new dtype > system to end all dtype systems, but that's a different story. > > Largely because of the above, even if Nathaniel's document involved > tossing a coin to resolve disputes, I'd rather have that now than something > much better never. Because there really is no alternative to Nathaniel's > write-up of the status quo, other than the status quo without a write-up: > it has taken him two months to put this draft together, **after** we agreed > over several hours of face to face discussion on what the model should be. > And I'm sure he has hated every minute he has had to
