Re: [Numpy-discussion] np.longlong casts to int

2012-02-23 Thread Pierre Haessig
Hi, Le 23/02/2012 02:24, Matthew Brett a écrit : Luckily I was in fact using longdouble in the live code, I had never exotic floating point precision, so thanks for your post which made me take a look at docstring and documentation. If I got it right from the docstring, 'np.longdouble',

Re: [Numpy-discussion] python geospatial package?

2012-02-23 Thread Vincent Schut
On 02/22/2012 10:45 PM, Chao YUE wrote: Hi all, Is anyone using some python geospatial package that can do jobs like intersection, etc. the job is like you automatically extract a region on a global map etc. thanks and cheers, Chao Chao, shapely would do this, though I found it had a

Re: [Numpy-discussion] np.longlong casts to int

2012-02-23 Thread Francesc Alted
On Feb 23, 2012, at 3:06 AM, Pierre Haessig wrote: Hi, Le 23/02/2012 02:24, Matthew Brett a écrit : Luckily I was in fact using longdouble in the live code, I had never exotic floating point precision, so thanks for your post which made me take a look at docstring and documentation. If I

Re: [Numpy-discussion] np.longlong casts to int

2012-02-23 Thread Nathaniel Smith
On Thu, Feb 23, 2012 at 11:40 AM, Francesc Alted franc...@continuum.io wrote: Exactly.  I'd update this to read: float96    96 bits.  Only available on 32-bit (i386) platforms. float128  128 bits.  Only available on 64-bit (AMD64) platforms. Except float96 is actually 80 bits. (Usually?) Plus

Re: [Numpy-discussion] np.longlong casts to int

2012-02-23 Thread Pierre Haessig
Le 23/02/2012 12:40, Francesc Alted a écrit : However, I was surprised that float128 is not mentioned in the array of available types in the user guide. http://docs.scipy.org/doc/numpy/user/basics.types.html Is there a specific reason for this absence, or is just about visiting the

Re: [Numpy-discussion] np.longlong casts to int

2012-02-23 Thread Francesc Alted
On Feb 23, 2012, at 5:43 AM, Nathaniel Smith wrote: On Thu, Feb 23, 2012 at 11:40 AM, Francesc Alted franc...@continuum.io wrote: Exactly. I'd update this to read: float9696 bits. Only available on 32-bit (i386) platforms. float128 128 bits. Only available on 64-bit (AMD64)

Re: [Numpy-discussion] np.longlong casts to int

2012-02-23 Thread Francesc Alted
On Feb 23, 2012, at 6:06 AM, Francesc Alted wrote: On Feb 23, 2012, at 5:43 AM, Nathaniel Smith wrote: On Thu, Feb 23, 2012 at 11:40 AM, Francesc Alted franc...@continuum.io wrote: Exactly. I'd update this to read: float9696 bits. Only available on 32-bit (i386) platforms.

Re: [Numpy-discussion] python geospatial package?

2012-02-23 Thread Kiko
2012/2/23 Vincent Schut sc...@sarvision.nl On 02/22/2012 10:45 PM, Chao YUE wrote: Hi all, Is anyone using some python geospatial package that can do jobs like intersection, etc. the job is like you automatically extract a region on a global map etc. thanks and cheers, Chao

[Numpy-discussion] Special matrices with structure?

2012-02-23 Thread Jaakko Luttinen
Hi! I was wondering whether it would be easy/possible/reasonable to have classes for arrays that have special structure in order to use less memory and speed up some computations? For instance: - symmetric matrix could be stored in almost half the memory required by a non-symmetric matrix -

Re: [Numpy-discussion] np.longlong casts to int

2012-02-23 Thread Matthew Brett
Hi, On Thu, Feb 23, 2012 at 4:23 AM, Francesc Alted franc...@continuum.io wrote: On Feb 23, 2012, at 6:06 AM, Francesc Alted wrote: On Feb 23, 2012, at 5:43 AM, Nathaniel Smith wrote: On Thu, Feb 23, 2012 at 11:40 AM, Francesc Alted franc...@continuum.io wrote: Exactly.  I'd update this to

Re: [Numpy-discussion] np.longlong casts to int

2012-02-23 Thread Charles R Harris
On Thu, Feb 23, 2012 at 5:23 AM, Francesc Alted franc...@continuum.iowrote: On Feb 23, 2012, at 6:06 AM, Francesc Alted wrote: On Feb 23, 2012, at 5:43 AM, Nathaniel Smith wrote: On Thu, Feb 23, 2012 at 11:40 AM, Francesc Alted franc...@continuum.io wrote: Exactly. I'd update this to

Re: [Numpy-discussion] Special matrices with structure?

2012-02-23 Thread Dag Sverre Seljebotn
On 02/23/2012 05:50 AM, Jaakko Luttinen wrote: Hi! I was wondering whether it would be easy/possible/reasonable to have classes for arrays that have special structure in order to use less memory and speed up some computations? For instance: - symmetric matrix could be stored in almost half

Re: [Numpy-discussion] Special matrices with structure?

2012-02-23 Thread Dag Sverre Seljebotn
On 02/23/2012 09:47 AM, Dag Sverre Seljebotn wrote: On 02/23/2012 05:50 AM, Jaakko Luttinen wrote: Hi! I was wondering whether it would be easy/possible/reasonable to have classes for arrays that have special structure in order to use less memory and speed up some computations? For

Re: [Numpy-discussion] np.longlong casts to int

2012-02-23 Thread Pierre Haessig
Le 23/02/2012 17:28, Charles R Harris a écrit : That's correct. They are both extended precision (80 bits), but aligned on 32bit/64bit boundaries respectively. Sun provides a true quad precision, also called float128, while on PPC long double is an odd combination of two doubles. This is

Re: [Numpy-discussion] np.longlong casts to int

2012-02-23 Thread Matthew Brett
Hi, On Thu, Feb 23, 2012 at 10:11 AM, Pierre Haessig pierre.haes...@crans.org wrote: Le 23/02/2012 17:28, Charles R Harris a écrit : That's correct. They are both extended precision (80 bits), but aligned on 32bit/64bit boundaries respectively. Sun provides a true quad precision, also called

Re: [Numpy-discussion] np.longlong casts to int

2012-02-23 Thread Mark Wiebe
On Thu, Feb 23, 2012 at 10:42 AM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Thu, Feb 23, 2012 at 10:11 AM, Pierre Haessig pierre.haes...@crans.org wrote: Le 23/02/2012 17:28, Charles R Harris a écrit : That's correct. They are both extended precision (80 bits), but aligned on

Re: [Numpy-discussion] np.longlong casts to int

2012-02-23 Thread Matthew Brett
Hi, On Thu, Feb 23, 2012 at 10:45 AM, Mark Wiebe mwwi...@gmail.com wrote: On Thu, Feb 23, 2012 at 10:42 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Feb 23, 2012 at 10:11 AM, Pierre Haessig pierre.haes...@crans.org wrote: Le 23/02/2012 17:28, Charles R Harris a écrit :

Re: [Numpy-discussion] np.longlong casts to int

2012-02-23 Thread Mark Wiebe
On Thu, Feb 23, 2012 at 10:55 AM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Thu, Feb 23, 2012 at 10:45 AM, Mark Wiebe mwwi...@gmail.com wrote: On Thu, Feb 23, 2012 at 10:42 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Feb 23, 2012 at 10:11 AM, Pierre

[Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Wes McKinney
dear all, I haven't read all 180 e-mails, but I didn't see this on Travis's initial list. All of the existing flat file reading solutions I have seen are not suitable for many applications, and they compare very unfavorably to tools present in other languages, like R. Here are some of the main

[Numpy-discussion] mkl usage

2012-02-23 Thread Neal Becker
Is mkl only used for linear algebra? Will it speed up e.g., elementwise transendental functions? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] mkl usage

2012-02-23 Thread Francesc Alted
On Feb 23, 2012, at 1:33 PM, Neal Becker wrote: Is mkl only used for linear algebra? Will it speed up e.g., elementwise transendental functions? Yes, MKL comes with VML that has this type of optimizations: http://software.intel.com/sites/products/documentation/hpc/mkl/vml/vmldata.htm Also,

Re: [Numpy-discussion] mkl usage

2012-02-23 Thread Pauli Virtanen
23.02.2012 20:44, Francesc Alted kirjoitti: On Feb 23, 2012, at 1:33 PM, Neal Becker wrote: Is mkl only used for linear algebra? Will it speed up e.g., elementwise transendental functions? Yes, MKL comes with VML that has this type of optimizations: And also no, in the sense that Numpy

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Pauli Virtanen
Hi, 23.02.2012 20:32, Wes McKinney kirjoitti: [clip] To be clear: I'm going to do this eventually whether or not it happens in NumPy because it's an existing problem for heavy pandas users. I see no reason why the code can't emit structured arrays, too, so we might as well have a common

Re: [Numpy-discussion] np.longlong casts to int

2012-02-23 Thread Pierre Haessig
Le 23/02/2012 20:08, Mark Wiebe a écrit : +1, I think it's good for its name to correspond to the name in C/C++, so that when people search for information on it they will find the relevant information more easily. With a bunch of NumPy-specific aliases, it just creates more hassle for

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Travis Oliphant
This is actually on my short-list as well --- it just didn't make it to the list. In fact, we have someone starting work on it this week. It is his first project so it will take him a little time to get up to speed on it, but he will contact Wes and work with him and report progress to this

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Wes McKinney
On Thu, Feb 23, 2012 at 3:08 PM, Travis Oliphant tra...@continuum.io wrote: This is actually on my short-list as well --- it just didn't make it to the list. In fact, we have someone starting work on it this week.  It is his first project so it will take him a little time to get up to speed

Re: [Numpy-discussion] mkl usage

2012-02-23 Thread Neal Becker
Pauli Virtanen wrote: 23.02.2012 20:44, Francesc Alted kirjoitti: On Feb 23, 2012, at 1:33 PM, Neal Becker wrote: Is mkl only used for linear algebra? Will it speed up e.g., elementwise transendental functions? Yes, MKL comes with VML that has this type of optimizations: And also no,

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Warren Weckesser
On Thu, Feb 23, 2012 at 2:08 PM, Travis Oliphant tra...@continuum.iowrote: This is actually on my short-list as well --- it just didn't make it to the list. In fact, we have someone starting work on it this week. It is his first project so it will take him a little time to get up to speed

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Erin Sheldon
Wes - I designed the recfile package to fill this need. It might be a start. Some features: - the ability to efficiently read any subset of the data without loading the whole file. - reads directly into a recarray, so no overheads. - object oriented interface, mimicking

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Wes McKinney
On Thu, Feb 23, 2012 at 3:19 PM, Warren Weckesser warren.weckes...@enthought.com wrote: On Thu, Feb 23, 2012 at 2:08 PM, Travis Oliphant tra...@continuum.io wrote: This is actually on my short-list as well --- it just didn't make it to the list. In fact, we have someone starting work on it

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Wes McKinney
On Thu, Feb 23, 2012 at 3:23 PM, Erin Sheldon erin.shel...@gmail.com wrote: Wes - I designed the recfile package to fill this need.  It might be a start. Some features:    - the ability to efficiently read any subset of the data without      loading the whole file.    - reads directly

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Éric Depagne
Le jeudi 23 février 2012 21:24:28, Wes McKinney a écrit : That would indeed be great. Reading large files is a real pain whatever the python method used. BTW, could you tell us what you mean by large files? cheers, Éric. Sweet, between this, Continuum folks, and me and my guys I think we

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Erin Sheldon
Excerpts from Wes McKinney's message of Thu Feb 23 15:24:44 -0500 2012: On Thu, Feb 23, 2012 at 3:23 PM, Erin Sheldon erin.shel...@gmail.com wrote: I designed the recfile package to fill this need.  It might be a start. Can you relicense as BSD-compatible? If required, that would be fine with

Re: [Numpy-discussion] mkl usage

2012-02-23 Thread Francesc Alted
On Feb 23, 2012, at 2:19 PM, Neal Becker wrote: Pauli Virtanen wrote: 23.02.2012 20:44, Francesc Alted kirjoitti: On Feb 23, 2012, at 1:33 PM, Neal Becker wrote: Is mkl only used for linear algebra? Will it speed up e.g., elementwise transendental functions? Yes, MKL comes with VML

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Pierre Haessig
Le 23/02/2012 20:32, Wes McKinney a écrit : If anyone wants to get involved in this particular problem right now, let me know! Hi Wes, I'm totally out of the implementations issues you described, but I have some million-lines-long CSV files so that I experience some slowdown when loading those.

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Pierre Haessig
Le 23/02/2012 21:08, Travis Oliphant a écrit : I think loadtxt is now the 3rd or 4th text-reading interface I've seen in NumPy. Ok, now I understand why I got confused ;-) -- Pierre signature.asc Description: OpenPGP digital signature ___

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Wes McKinney
On Thu, Feb 23, 2012 at 3:31 PM, Éric Depagne e...@depagne.org wrote: Le jeudi 23 février 2012 21:24:28, Wes McKinney a écrit : That would indeed be great. Reading large files is a real pain whatever the python method used. BTW, could you tell us what you mean by large files? cheers,

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Erin Sheldon
Excerpts from Wes McKinney's message of Thu Feb 23 15:45:18 -0500 2012: Reasonably wide CSV files with hundreds of thousands to millions of rows. I have a separate interest in JSON handling but that is a different kind of problem, and probably just a matter of forking ultrajson and having it

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Wes McKinney
On Thu, Feb 23, 2012 at 3:55 PM, Erin Sheldon erin.shel...@gmail.com wrote: Excerpts from Wes McKinney's message of Thu Feb 23 15:45:18 -0500 2012: Reasonably wide CSV files with hundreds of thousands to millions of rows. I have a separate interest in JSON handling but that is a different kind

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Gael Varoquaux
On Thu, Feb 23, 2012 at 04:07:04PM -0500, Wes McKinney wrote: In this last case for example, around 500 MB of RAM is taken up for an array that should only be about 80-90MB. If you're a data scientist working in Python, this is _not good_. But why, oh why, are people storing big data in CSV?

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Éric Depagne
But why, oh why, are people storing big data in CSV? Well, that's what scientist do :-) Éric. G ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion Un clavier azerty en vaut deux

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Robert Kern
On Thu, Feb 23, 2012 at 21:09, Gael Varoquaux gael.varoqu...@normalesup.org wrote: On Thu, Feb 23, 2012 at 04:07:04PM -0500, Wes McKinney wrote: In this last case for example, around 500 MB of RAM is taken up for an array that should only be about 80-90MB. If you're a data scientist working in

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Erin Sheldon
Excerpts from Wes McKinney's message of Thu Feb 23 16:07:04 -0500 2012: That's pretty good. That's faster than pandas's csv-module+Cython approach almost certainly (but I haven't run your code to get a read on how much my hardware makes a difference), but that's not shocking at all: In [1]:

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Benjamin Root
On Thu, Feb 23, 2012 at 3:14 PM, Robert Kern robert.k...@gmail.com wrote: On Thu, Feb 23, 2012 at 21:09, Gael Varoquaux gael.varoqu...@normalesup.org wrote: On Thu, Feb 23, 2012 at 04:07:04PM -0500, Wes McKinney wrote: In this last case for example, around 500 MB of RAM is taken up for an

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Wes McKinney
On Thu, Feb 23, 2012 at 4:20 PM, Erin Sheldon erin.shel...@gmail.com wrote: Excerpts from Wes McKinney's message of Thu Feb 23 16:07:04 -0500 2012: That's pretty good. That's faster than pandas's csv-module+Cython approach almost certainly (but I haven't run your code to get a read on how much

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Pierre Haessig
Le 23/02/2012 22:38, Benjamin Root a écrit : labmate/officemate/advisor is using Excel... ... or an industrial partner with its windows-based software that can export (when it works) some very nice field data from a proprietary Honeywell data logger. CSV data is better than no data ! (and better

[Numpy-discussion] Problem Building Numpy with Python 2.7.1 and OS X 10.7.3

2012-02-23 Thread Patrick Armstrong
Hi there, I'm having a problem building NumPy on Python 2.7.1 and OS X 10.7.3. Here is my build log: https://gist.github.com/1895377 Does anyone have any idea what might be happening? I get a very similar error when compiling with clang. Installing a binary really isn't an option for me due

[Numpy-discussion] Announcing Theano 0.5

2012-02-23 Thread Pascal Lamblin
=== Announcing Theano 0.5 === This is a major version, with lots of new features, bug fixes, and some interface changes (deprecated or potentially misleading features were removed). Upgrading to Theano 0.5 is recommended for everyone, but you

Re: [Numpy-discussion] np.longlong casts to int

2012-02-23 Thread Matthew Brett
Hi, On Thu, Feb 23, 2012 at 2:56 PM, Pierre Haessig pierre.haes...@crans.org wrote: Le 23/02/2012 20:08, Mark Wiebe a écrit : +1, I think it's good for its name to correspond to the name in C/C++, so that when people search for information on it they will find the relevant information more

[Numpy-discussion] Test survey that I have been putting together

2012-02-23 Thread Travis Oliphant
Hey all, I would like to gather concrete information about NumPy users and have some data to look at regarding the user base and features that are of interest. We have been putting together a survey that I would love feedback on from members of this list. If you have time and are

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Drew Frank
For convenience, here's a link to the mailing list thread on this topic from a couple months ago: http://thread.gmane.org/gmane.comp.python.numeric.general/47094 . Drew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Paul Anton Letnes
As others on this list, I've also been confused a bit by the prolific numpy interfaces to reading text. Would it be an idea to create some sort of object oriented solution for this purpose? reader = np.FileReader('my_file.txt') reader.loadtxt() # for backwards compat.; np.loadtxt could