Re: [Numpy-discussion] Where is Jaime?

[Jaime Fernández del Río]

On Sun, Dec 6, 2015 at 4:15 AM, Charles R Harris 
wrote:

>
>
> On Sat, Dec 5, 2015 at 4:49 PM, Jaime Fernández del Río <
> jaime.f...@gmail.com> wrote:
>
>> I'm alive and well: trying to stay afloat on a sea of messaging
>> protocols, Java and Swiss bureaucracy, but doing great aside from that.
>>
>> Jaime
>>
>>
> Glad to hear it. I was beginning to worry...
>
> Java? Poor soul. Anything special about the Swiss bureaucracy? Reminds me
> of the old joke
>

Well, if you don't have a suitcase full of $100 bills, opening a bank
account in Switzerland is surprisingly difficult, especially if you are
moving here from the U.S. If immigration then decides to request from you
documents they already have, thus delaying your residence permit by a few
more weeks, the bank ends up returning your first salary, just about the
same time you have to pay a 3 month rental deposit for a small and
ridiculously expensive apartment. Everything is slowly falling into place,
but it has been an interesting ride.


> *Heaven and Hell*
>
> Heaven Is Where:
>
> The French are the chefs
> The Italians are the lovers
> The British are the police
> The Germans are the mechanics
> And the Swiss make everything run on time
>
>
> Hell is Where:
>
> The British are the chefs
> The Swiss are the lovers
> The French are the mechanics
> The Italians make everything run on time
> And the Germans are the police
>

The trains and trams do seem to run remarkably on time, but I don't think
Eva would be too happy about me setting out to test how good lovers the
Swiss are...

Jaime

-- 
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
de dominación mundial.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] When to stop supporting Python 2.6?

[Charles R Harris]

On Fri, Dec 4, 2015 at 9:29 AM, Bryan Van de Ven 
wrote:

>
>
>
> > On Dec 4, 2015, at 9:49 AM, Charles R Harris 
> wrote:
> >
> >
> >
> > On Fri, Dec 4, 2015 at 2:40 AM, Julian Taylor <
> jtaylor.deb...@googlemail.com> wrote:
> > dropping 3.2: +-0 as it would remove some extra code in our broken py3
> > string handling but not much
> > dropping 3.3: -1 doesn't gain us anything so far I know
> > dropping 2.6: -1, I don't see not enough advantage the only issue I know
> > of is an occasional set literal which gets caught by our test-suite
> > immediately. Besides 2.6 is still the default in RHEL6. But if there is
> > something larger which makes it worthwhile I don't know about I have no
> > objections.
> >
> > My thought is that dropping 2.6 allows a more unified code base between
> Python 2 and Python3. In 2.7 we get
> >
> >   • The syntax for set literals ({1,2,3} is a mutable set).
> >   • Dictionary and set comprehensions ({i: i*2 for i in range(3)}).
> >   • Multiple context managers in a single with statement.
> >   • A new version of the io library, rewritten in C for performance.
> >   • The ordered-dictionary type described in PEP 372: Adding an
> Ordered Dictionary to collections.
> >   • The new "," format specifier described in PEP 378: Format
> Specifier for Thousands Separator.
> >   • The memoryview object.
> >   • A small subset of the importlib module, described below.
> >   • The repr() of a float x is shorter in many cases: it’s now based
> on the shortest decimal string that’s guaranteed to round back to x. As in
> previous versions of Python, it’s guaranteed that float(repr(x)) recovers x.
> >   • Float-to-string and string-to-float conversions are correctly
> rounded. The round() function is also now correctly rounded.
> >   • The PyCapsule type, used to provide a C API for extension
> modules.
> >   • The PyLong_AsLongAndOverflow() C API function.
> > In particular, memoryview and PyCapsule are available. Moving to Python
> 3.3 as a minimum provides unicode literals. Python 3.4 strikes me as the
> end of the Python 3 beginning, with future Python development taking off
> from there.
>
> I'd suggest that anything that unifies the codebase and reduces complexity
> and special cases will not only help current developers, but also lower the
> bar for potential new developers as well. The importance of streamlining
> and reducing the maintenance burden in long-running projects cannot be
> overstated, in my opinion.
>
> I'd also suggest that Numpy is in a unique position proactively encourage
> people to use more reasonable versions of python, and for those that can't
> or won't (yet) it's not like older versions of Numpy will disappear. A
> brief search seems to affirm my feeling that "2.7 + 3.3/3.4" support is
> becoming fairly standard among a wide range of OSS python projects.
>
> Regarding RHEL6 comment above, even Nick Coghlan suggests that is not a
> compelling motivation:
>
>
> http://www.curiousefficiency.org/posts/2015/04/stop-supporting-python26.html
>
> """
> While it's entirely admirable that many upstream developers are generous
> enough to help their end users work around this inertia, in the long run
> doing so is detrimental for everyone concerned, as long term sustaining
> engineering for old releases is genuinely demotivating for upstream
> developers (it's a good job, but a lousy way to spend your free time) and
> for end users, working around institutional inertia this way reduces the
> pressure to actually get the situation addressed properly.
> """
>
>
As a strawman proposal, how about dropping moving to 2.7 and 3.4 minimum
supported version next fall, say around numpy 1.12 or 1.13 depending on how
the releases go.

I would like to here from the scipy folks first.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] array of random numbers fails to construct

[DAVID SAROFF (RIT Student)]

Allan,

I see with a google search on your name that you are in the physics
department at Rutgers. I got my BA in Physics there. 1975. Biological
physics. A thought: Is there an entropy that can be assigned to the dna in
an organism? I don't mean the usual thing, coupled to the heat bath.
Evolution blindly explores metabolic and signalling pathways, and tends
towards disorder, as long as it functions. Someone working out signaling
pathways some years ago wrote that they were senselessly complex, branched
and interlocked. I think that is to be expected. Evolution doesn't find
minimalist, clear, rational solutions. Look at the amazon rain forest. What
are all those beetles and butterflies and frogs for? It is the wrong
question. I think some measure of the complexity could be related to the
amount of time that ecosystem has existed. Similarly for genomes.

On Sun, Dec 6, 2015 at 6:55 PM, Allan Haldane 
wrote:

>
> I've also often wanted to generate large datasets of random uint8 and
> uint16. As a workaround, this is something I have used:
>
> np.ndarray(100, 'u1', np.random.bytes(100))
>
> It has also crossed my mind that np.random.randint and np.random.rand
> could use an extra 'dtype' keyword. It didn't look easy to implement though.
>
> Allan
>
> On 12/06/2015 04:55 PM, DAVID SAROFF (RIT Student) wrote:
>
>> Matthew,
>>
>> That looks right. I'm concluding that the .astype(np.uint8) is applied
>> after the array is constructed, instead of during the process. This
>> random array is a test case. In the production analysis of radio
>> telescope data this is how the data comes in, and there is no  problem
>> with 10GBy files.
>> linearInputData = np.fromfile(dataFile, dtype = np.uint8, count = -1)
>> spectrumArray = linearInputData.reshape(nSpectra,sizeSpectrum)
>>
>>
>> On Sun, Dec 6, 2015 at 4:07 PM, Matthew Brett > > wrote:
>>
>> Hi,
>>
>> On Sun, Dec 6, 2015 at 12:39 PM, DAVID SAROFF (RIT Student)
>> > wrote:
>> > This works. A big array of eight bit random numbers is constructed:
>> >
>> > import numpy as np
>> >
>> > spectrumArray = np.random.randint(0,255,
>> (2**20,2**12)).astype(np.uint8)
>> >
>> >
>> >
>> > This fails. It eats up all 64GBy of RAM:
>> >
>> > spectrumArray = np.random.randint(0,255,
>> (2**21,2**12)).astype(np.uint8)
>> >
>> >
>> > The difference is a factor of two, 2**21 rather than 2**20, for the
>> extent
>> > of the first axis.
>>
>> I think what's happening is that this:
>>
>> np.random.randint(0,255, (2**21,2**12))
>>
>> creates 2**33 random integers, which (on 64-bit) will be of dtype
>> int64 = 8 bytes, giving total size 2 ** (21 + 12 + 6) = 2 ** 39 bytes
>> = 512 GiB.
>>
>> Cheers,
>>
>> Matthew
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org 
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>>
>>
>> --
>> David P. Saroff
>> Rochester Institute of Technology
>> 54 Lomb Memorial Dr, Rochester, NY 14623
>> david.sar...@mail.rit.edu  | (434)
>> 227-6242
>>
>>
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
David P. Saroff
Rochester Institute of Technology
54 Lomb Memorial Dr, Rochester, NY 14623
david.sar...@mail.rit.edu | (434) 227-6242
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] array of random numbers fails to construct

[Allan Haldane]

I've also often wanted to generate large datasets of random uint8 and 
uint16. As a workaround, this is something I have used:


np.ndarray(100, 'u1', np.random.bytes(100))

It has also crossed my mind that np.random.randint and np.random.rand 
could use an extra 'dtype' keyword. It didn't look easy to implement though.


Allan

On 12/06/2015 04:55 PM, DAVID SAROFF (RIT Student) wrote:

Matthew,

That looks right. I'm concluding that the .astype(np.uint8) is applied
after the array is constructed, instead of during the process. This
random array is a test case. In the production analysis of radio
telescope data this is how the data comes in, and there is no  problem
with 10GBy files.
linearInputData = np.fromfile(dataFile, dtype = np.uint8, count = -1)
spectrumArray = linearInputData.reshape(nSpectra,sizeSpectrum)


On Sun, Dec 6, 2015 at 4:07 PM, Matthew Brett > wrote:

Hi,

On Sun, Dec 6, 2015 at 12:39 PM, DAVID SAROFF (RIT Student)
> wrote:
> This works. A big array of eight bit random numbers is constructed:
>
> import numpy as np
>
> spectrumArray = np.random.randint(0,255, (2**20,2**12)).astype(np.uint8)
>
>
>
> This fails. It eats up all 64GBy of RAM:
>
> spectrumArray = np.random.randint(0,255, (2**21,2**12)).astype(np.uint8)
>
>
> The difference is a factor of two, 2**21 rather than 2**20, for the extent
> of the first axis.

I think what's happening is that this:

np.random.randint(0,255, (2**21,2**12))

creates 2**33 random integers, which (on 64-bit) will be of dtype
int64 = 8 bytes, giving total size 2 ** (21 + 12 + 6) = 2 ** 39 bytes
= 512 GiB.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org 
https://mail.scipy.org/mailman/listinfo/numpy-discussion




--
David P. Saroff
Rochester Institute of Technology
54 Lomb Memorial Dr, Rochester, NY 14623
david.sar...@mail.rit.edu  | (434)
227-6242



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Where is Jaime?

[Nathaniel Smith]

On Dec 6, 2015 6:03 PM, "Peter Creasey" 
wrote:
>
> >
> > Is the interp fix in the google pipeline or do we need a workaround?
> >
>
> Oooh, if someone is looking at changing interp, is there any chance
> that fp could be extended to take complex128 rather than just float
> values? I.e. so that I could write:
>
> >>> y = interp(mu, theta, m)
> rather than
> >>> y = interp(mu, theta, m.real) + 1.0j*interp(mu, theta, m.imag)
>
> which *sounds* like it might be simple and more (Num)pythonic.

That sounds like an excellent improvement and you should submit a PR
implementing it :-).

"The interp fix" in question though is a regression in 1.10 that's blocking
1.10.2, and needs a quick minimal fix asap.

-n
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] array of random numbers fails to construct

[Matthew Brett]

Hi,

On Sun, Dec 6, 2015 at 12:39 PM, DAVID SAROFF (RIT Student)
 wrote:
> This works. A big array of eight bit random numbers is constructed:
>
> import numpy as np
>
> spectrumArray = np.random.randint(0,255, (2**20,2**12)).astype(np.uint8)
>
>
>
> This fails. It eats up all 64GBy of RAM:
>
> spectrumArray = np.random.randint(0,255, (2**21,2**12)).astype(np.uint8)
>
>
> The difference is a factor of two, 2**21 rather than 2**20, for the extent
> of the first axis.

I think what's happening is that this:

np.random.randint(0,255, (2**21,2**12))

creates 2**33 random integers, which (on 64-bit) will be of dtype
int64 = 8 bytes, giving total size 2 ** (21 + 12 + 6) = 2 ** 39 bytes
= 512 GiB.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] array of random numbers fails to construct

[DAVID SAROFF (RIT Student)]

Matthew,

That looks right. I'm concluding that the .astype(np.uint8) is applied
after the array is constructed, instead of during the process. This random
array is a test case. In the production analysis of radio telescope data
this is how the data comes in, and there is no  problem with 10GBy files.
linearInputData = np.fromfile(dataFile, dtype = np.uint8, count = -1)
spectrumArray = linearInputData.reshape(nSpectra,sizeSpectrum)


On Sun, Dec 6, 2015 at 4:07 PM, Matthew Brett 
wrote:

> Hi,
>
> On Sun, Dec 6, 2015 at 12:39 PM, DAVID SAROFF (RIT Student)
>  wrote:
> > This works. A big array of eight bit random numbers is constructed:
> >
> > import numpy as np
> >
> > spectrumArray = np.random.randint(0,255, (2**20,2**12)).astype(np.uint8)
> >
> >
> >
> > This fails. It eats up all 64GBy of RAM:
> >
> > spectrumArray = np.random.randint(0,255, (2**21,2**12)).astype(np.uint8)
> >
> >
> > The difference is a factor of two, 2**21 rather than 2**20, for the
> extent
> > of the first axis.
>
> I think what's happening is that this:
>
> np.random.randint(0,255, (2**21,2**12))
>
> creates 2**33 random integers, which (on 64-bit) will be of dtype
> int64 = 8 bytes, giving total size 2 ** (21 + 12 + 6) = 2 ** 39 bytes
> = 512 GiB.
>
> Cheers,
>
> Matthew
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
David P. Saroff
Rochester Institute of Technology
54 Lomb Memorial Dr, Rochester, NY 14623
david.sar...@mail.rit.edu | (434) 227-6242
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] array of random numbers fails to construct

[Jaime Fernández del Río]

On Sun, Dec 6, 2015 at 10:07 PM, Matthew Brett 
wrote:

> Hi,
>
> On Sun, Dec 6, 2015 at 12:39 PM, DAVID SAROFF (RIT Student)
>  wrote:
> > This works. A big array of eight bit random numbers is constructed:
> >
> > import numpy as np
> >
> > spectrumArray = np.random.randint(0,255, (2**20,2**12)).astype(np.uint8)
> >
> >
> >
> > This fails. It eats up all 64GBy of RAM:
> >
> > spectrumArray = np.random.randint(0,255, (2**21,2**12)).astype(np.uint8)
> >
> >
> > The difference is a factor of two, 2**21 rather than 2**20, for the
> extent
> > of the first axis.
>
> I think what's happening is that this:
>
> np.random.randint(0,255, (2**21,2**12))
>
> creates 2**33 random integers, which (on 64-bit) will be of dtype
> int64 = 8 bytes, giving total size 2 ** (21 + 12 + 6) = 2 ** 39 bytes
> = 512 GiB.
>

8 is only 2**3, so it is "just" 64 GiB, which also explains why the half
sized array does work, but yes, that is most likely what's happening.

Jaime

>
> Cheers,
>
> Matthew
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
de dominación mundial.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Where is Jaime?

[Charles R Harris]

On Sun, Dec 6, 2015 at 1:40 AM, Jaime Fernández del Río <
jaime.f...@gmail.com> wrote:

> On Sun, Dec 6, 2015 at 4:15 AM, Charles R Harris <
> charlesr.har...@gmail.com> wrote:
>
>>
>>
>> On Sat, Dec 5, 2015 at 4:49 PM, Jaime Fernández del Río <
>> jaime.f...@gmail.com> wrote:
>>
>>> I'm alive and well: trying to stay afloat on a sea of messaging
>>> protocols, Java and Swiss bureaucracy, but doing great aside from that.
>>>
>>> Jaime
>>>
>>>
>> Glad to hear it. I was beginning to worry...
>>
>> Java? Poor soul. Anything special about the Swiss bureaucracy? Reminds me
>> of the old joke
>>
>
> Well, if you don't have a suitcase full of $100 bills, opening a bank
> account in Switzerland is surprisingly difficult, especially if you are
> moving here from the U.S. If immigration then decides to request from you
> documents they already have, thus delaying your residence permit by a few
> more weeks, the bank ends up returning your first salary, just about the
> same time you have to pay a 3 month rental deposit for a small and
> ridiculously expensive apartment. Everything is slowly falling into place,
> but it has been an interesting ride.
>
>

The cash economy is nothing to sniff at ;) It is big in NYC and other
places with high taxes and bureaucratic meddling. Cash was one of the great
inventions.

Is the interp fix in the google pipeline or do we need a workaround?

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] array of random numbers fails to construct

[DAVID SAROFF (RIT Student)]

This works. A big array of eight bit random numbers is constructed:

import numpy as np

spectrumArray = np.random.randint(0,255, (2**20,2**12)).astype(np.uint8)



This fails. It eats up all 64GBy of RAM:

spectrumArray = np.random.randint(0,255, (2**21,2**12)).astype(np.uint8)


The difference is a factor of two, 2**21 rather than 2**20, for the extent
of the first axis.

-- 
David P. Saroff
Rochester Institute of Technology
54 Lomb Memorial Dr, Rochester, NY 14623
david.sar...@mail.rit.edu | (434) 227-6242
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion