Re: [Numpy-discussion] ANN: HDF5 for Python 2.5.0
On 9 Apr 2015, at 9:41 pm, Andrew Collette wrote: > >> Congrats! Also btw, you might want to switch to a new subject line format >> for these emails -- the mention of Python 2.5 getting hdf5 support made me >> do a serious double take before I figured out what was going on, and 2.6 and >> 2.7 will be even worse :-) > > Ha! Didn't even think of that. For our next release I guess we'll > have to go straight to h5py 3.5. You may have to hurry though ;-) "Monday, March 30, 2015 Python 3.5.0a3 has been released. This is the third alpha release of Python 3.5, which will be the next major release of Python. Python 3.5 is still under heavy development and is far from complete.” 3 alpha releases in 7 weeks… On a more serious note though, h5py 2.5.x in the subject would be perfectly clear enough, I think, and also help to distinguish from pytables releases. Derek ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: HDF5 for Python 2.5.0
> Congrats! Also btw, you might want to switch to a new subject line format > for these emails -- the mention of Python 2.5 getting hdf5 support made me > do a serious double take before I figured out what was going on, and 2.6 and > 2.7 will be even worse :-) Ha! Didn't even think of that. For our next release I guess we'll have to go straight to h5py 3.5. Andrew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: HDF5 for Python 2.5.0
On Apr 9, 2015 2:41 PM, "Nathaniel Smith" wrote: > > (Off-list) Doh, we do reply-to munging, don't we. Oh well. -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: HDF5 for Python 2.5.0
On Thu, Apr 9, 2015 at 2:41 PM, Nathaniel Smith wrote: > (Off-list) > > Congrats! Also btw, you might want to switch to a new subject line format > for these emails -- the mention of Python 2.5 getting hdf5 support made me > do a serious double take before I figured out what was going on, and 2.6 and > 2.7 will be even worse :-) (offlist) I also had to read the subject line and the first paragraph several times to see who is using python 2.5 Josef :} > > On Apr 9, 2015 2:07 PM, "Andrew Collette" wrote: >> >> Announcing HDF5 for Python (h5py) 2.5.0 >> >> >> The h5py team is happy to announce the availability of h5py 2.5.0. >> >> This release introduces experimental support for the highly-anticipated >> "Single Writer Multiple Reader" (SWMR) feature in the upcoming HDF5 1.10 >> release. SWMR allows sharing of a single HDF5 file between multiple >> processes without the complexity of MPI or multiprocessing-based >> solutions. >> >> This is an experimental feature that should NOT be used in production >> code. We are interested in getting feedback from the broader community >> with respect to performance and the API design. >> >> For more details, check out the h5py user guide: >> http://docs.h5py.org/en/latest/swmr.html >> >> SWMR support was contributed by Ulrik Pedersen. >> >> >> What's h5py? >> >> >> The h5py package is a Pythonic interface to the HDF5 binary data format. >> >> It lets you store huge amounts of numerical data, and easily manipulate >> that data from NumPy. For example, you can slice into multi-terabyte >> datasets stored on disk, as if they were real NumPy arrays. Thousands of >> datasets can be stored in a single file, categorized and tagged however >> you want. >> >> Documentation is at: >> >> http://docs.h5py.org >> >> >> Changes >> --- >> >> * Experimental SWMR support >> * Group and AttributeManager classes now inherit from the appropriate ABCs >> * Fixed an issue with 64-bit float VLENS >> * Cython warning cleanups related to "const" >> * Entire code base ported to "six"; 2to3 removed from setup.py >> >> >> Acknowledgements >> --- >> >> This release incorporates changes from, among others: >> >> * Ulrik Pedersen >> * James Tocknell >> * Will Parkin >> * Antony Lee >> * Peter H. Li >> * Peter Colberg >> * Ghislain Antony Vaillant >> >> >> Where to get it >> --- >> >> Downloads, documentation, and more are available at the h5py website: >> >> http://www.h5py.org >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: HDF5 for Python 2.5.0
(Off-list) Congrats! Also btw, you might want to switch to a new subject line format for these emails -- the mention of Python 2.5 getting hdf5 support made me do a serious double take before I figured out what was going on, and 2.6 and 2.7 will be even worse :-) On Apr 9, 2015 2:07 PM, "Andrew Collette" wrote: > Announcing HDF5 for Python (h5py) 2.5.0 > > > The h5py team is happy to announce the availability of h5py 2.5.0. > > This release introduces experimental support for the highly-anticipated > "Single Writer Multiple Reader" (SWMR) feature in the upcoming HDF5 1.10 > release. SWMR allows sharing of a single HDF5 file between multiple > processes without the complexity of MPI or multiprocessing-based > solutions. > > This is an experimental feature that should NOT be used in production > code. We are interested in getting feedback from the broader community > with respect to performance and the API design. > > For more details, check out the h5py user guide: > http://docs.h5py.org/en/latest/swmr.html > > SWMR support was contributed by Ulrik Pedersen. > > > What's h5py? > > > The h5py package is a Pythonic interface to the HDF5 binary data format. > > It lets you store huge amounts of numerical data, and easily manipulate > that data from NumPy. For example, you can slice into multi-terabyte > datasets stored on disk, as if they were real NumPy arrays. Thousands of > datasets can be stored in a single file, categorized and tagged however > you want. > > Documentation is at: > > http://docs.h5py.org > > > Changes > --- > > * Experimental SWMR support > * Group and AttributeManager classes now inherit from the appropriate ABCs > * Fixed an issue with 64-bit float VLENS > * Cython warning cleanups related to "const" > * Entire code base ported to "six"; 2to3 removed from setup.py > > > Acknowledgements > --- > > This release incorporates changes from, among others: > > * Ulrik Pedersen > * James Tocknell > * Will Parkin > * Antony Lee > * Peter H. Li > * Peter Colberg > * Ghislain Antony Vaillant > > > Where to get it > --- > > Downloads, documentation, and more are available at the h5py website: > > http://www.h5py.org > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: HDF5 for Python 2.5.0
Announcing HDF5 for Python (h5py) 2.5.0 The h5py team is happy to announce the availability of h5py 2.5.0. This release introduces experimental support for the highly-anticipated "Single Writer Multiple Reader" (SWMR) feature in the upcoming HDF5 1.10 release. SWMR allows sharing of a single HDF5 file between multiple processes without the complexity of MPI or multiprocessing-based solutions. This is an experimental feature that should NOT be used in production code. We are interested in getting feedback from the broader community with respect to performance and the API design. For more details, check out the h5py user guide: http://docs.h5py.org/en/latest/swmr.html SWMR support was contributed by Ulrik Pedersen. What's h5py? The h5py package is a Pythonic interface to the HDF5 binary data format. It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want. Documentation is at: http://docs.h5py.org Changes --- * Experimental SWMR support * Group and AttributeManager classes now inherit from the appropriate ABCs * Fixed an issue with 64-bit float VLENS * Cython warning cleanups related to "const" * Entire code base ported to "six"; 2to3 removed from setup.py Acknowledgements --- This release incorporates changes from, among others: * Ulrik Pedersen * James Tocknell * Will Parkin * Antony Lee * Peter H. Li * Peter Colberg * Ghislain Antony Vaillant Where to get it --- Downloads, documentation, and more are available at the h5py website: http://www.h5py.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On Thu, Apr 9, 2015 at 10:11 AM, Alan G Isaac wrote: > > Alan wrote: >>> 3. I admit, my students are NOT using non-boolen fancy indexing on >>> >multidimensional arrays. (As far as I know.) Are yours? The only confusing case is mixing slices and integer array indexing for ndim > 2. The rest looks unsurprising, AFAIR (AFAICS, my last fancy indexing mailing list discussion is at least 4 years old, with Jonathan Taylor. I don't remember when I discovered the usefulness of the axis argument in take which covers many 3 or higher dimensional indexing use cases.) > > > On 4/9/2015 2:22 AM, Nathaniel Smith wrote: >> Well, okay, this would explain it, since integer fancy indexing is >> exactly the confusing case:-) On the plus side, this also means that >> even if pigs started doing barrel-rolls through hell's >> winter-vortex-chilled air tomorrow and we simply removed integer fancy >> indexing, your students would be unaffected:-) > > > Except that they do use statsmodels, which I believe (?) does make use of > integer fancy-indexing. And maybe all work would come to a standstill, because every library is using fancy integer indexing. I still don't know what all constitutes fancy indexing. The two most common use cases for me (statsmodels) are indexing for selecting elements like diag_indices, triu_indices and maybe nonzero, and expanding from a unique array like inverse index in np.unique. And there are just a few, AFAIR, orthogonal indexing cases with broadcasting index arrays to select rectangular pieces of an array. Josef > > Alan > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
> Alan wrote: >> 3. I admit, my students are NOT using non-boolen fancy indexing on >> >multidimensional arrays. (As far as I know.) Are yours? On 4/9/2015 2:22 AM, Nathaniel Smith wrote: > Well, okay, this would explain it, since integer fancy indexing is > exactly the confusing case:-) On the plus side, this also means that > even if pigs started doing barrel-rolls through hell's > winter-vortex-chilled air tomorrow and we simply removed integer fancy > indexing, your students would be unaffected:-) Except that they do use statsmodels, which I believe (?) does make use of integer fancy-indexing. Alan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On 4/9/2015 1:57 AM, Nathaniel Smith wrote: > Do you think there's anything we could be > doing to reduce this kind of adrenaline reaction while still allowing > for relaxed discussion about out-there ideas? numpy3...@scipy.org :-) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] what files to include with compiled fortran extension on Mac
Hello list, I want to send somebody my compiled fortran extension on a Mac (compiled with f2py and gfortran). Problem is that it doesn't work on other Macs unless they also instal xcode (2 GB, yikes!) and gfortran. So apparently there are some additional files missing when I just send the compiled extension. Does anybody know what other files to include or (better) how to compile a fortran extension without needing to send any additional files? Thanks, Mark ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Non-meta indexing improvements discussion
On Thu, Apr 9, 2015 at 8:07 AM, Sebastian Berg wrote: > > On Do, 2015-04-09 at 08:50 +0200, Sebastian Berg wrote: > > 3. I do not know if it possible or useful, but I could imagine a module > > wide switch (similar to __future__ imports) to change the default > > indexing behaviour. > > OK, my suggestion But actually I do not know if I like it all that > much (nor if it can be done) since 1. and 2. seem to me like enough. But > if someone feels strongly about fancy indexing being bad, I wanted to > point out that there may be ways to go down the road to "switch" numpy > without actually switching. I can't think of a way to actually make that work (I can list all the ways I thought of that *don't* work if anyone insists, but it's a tedious dead end), but it also seems to me to be a step backward. Assuming that we have both .ortho_ix and .fancy_ix to work with, it seems to me that the explicitness is a good thing. Even if in this module you only want to exploit one of those semantics, your module's readers live in a wider context where both semantics play a role. Moving your marker of "this is the non-default semantics I am using" to some module or function header away from where the semantics are actually used makes the code harder to read. A newish user trying to read some nontrivial indexing code may come to the list and ask "what exactly is this expression doing here?" and give us just the line of code with the expression (anecdotally, this is usually how this scenario goes down). If we have to answer "it depends; is there an @ortho_indexing decorator at the top of the function?", that's probably a cure worse than the disease. The properties are a good way to provide googleable signposts right where the tricky semantics are being used. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Non-meta indexing improvements discussion
On Do, 2015-04-09 at 08:50 +0200, Sebastian Berg wrote: > Hi all, > > Let me take a shot at summing up some suggestions to make the indexing > less surprising, and maybe we can gather some more in a more > concentrated way now. > Did not want to comment on the first mail > 1. Implement something like `arr.fancy_index[...]` and > `arr.ortho_index[...]` (i.e. Jaimes PR is the start for trying this) > I like this, personally. There is not much to be lost and I fully agree with Robert on this. It opens up a lot of possibilities for us and especially also others. > 2. Add warnings for non-consecutive advanced indexing (i.e. the original > example `arr[0, :, index_array]`). This could be annoying sometimes, but then warnings do not break legacy code, and I think in new code again Robert is right for these cases using arr.fancy_index[...] is more explicit and a nice warning/google help to the confused reader. > > 3. I do not know if it possible or useful, but I could imagine a module > wide switch (similar to __future__ imports) to change the default > indexing behaviour. > OK, my suggestion But actually I do not know if I like it all that much (nor if it can be done) since 1. and 2. seem to me like enough. But if someone feels strongly about fancy indexing being bad, I wanted to point out that there may be ways to go down the road to "switch" numpy without actually switching. - Sebastian > > One more thing, implementing this (especially the "new" indexing) is > non-trivial, so as always help beyond just a discussion is appreciated > and in my opinion the best way to push an actual change to happen sooner > rather then in some far off future. I do not have time for concentrating > much on an implementation for a while myself for a while at least. > > - Sebastian > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On Do, 2015-04-09 at 02:22 -0400, Nathaniel Smith wrote: > On Wed, Apr 8, 2015 at 4:02 PM, Alan G Isaac wrote: > > 1. I use numpy in teaching. > > I have never heard a complaint about its indexing behavior. > > Have you heard such complaints? > > Some observations: > > 1) There's an unrelated thread on numpy-discussion right now in which > a user is baffled by the interaction between slicing and integer fancy > indexing: > http://thread.gmane.org/gmane.comp.python.numeric.general/60321 > And one of the three replies AFAICT also doesn't actually make sense, > in that its explanation relies on broadcasting two arrays with shape > (5,) against each other to produce an array with shape (5, 5). (Which > is not how broadcasting works.) To be fair, though, this isn't the > poster's fault, because they are quoting the documentation! > > 2) Again, entirely by coincidence, literally this week a numpy user at > Berkeley felt spontaneously moved to send a warning message to the > campus py4science list just to warn everyone about the bizarre > behaviour they had stumbled on where arr[0, :, idx] produced > inexplicable results. They had already found the docs and worked out > what was going on, they just felt it was necessary to warn everyone > else to be careful out there. > > 3) I personally regularly get confused by integer fancy indexing. I > actually understand it substantially better due to thinking it through > while reading these threads, but I'm a bit disturbed that I had that > much left to learn. (New key insight: you can think of *scalar* > indexing arr[i, j, k] as a function f(i, j, k) -> value. If you take > that function and make it a ufunc, then you have integer fancy > indexing. ...Though there's still an extra pound of explanation needed > to describe the mixed slice/fancy cases, it at least captures the > basic intuition. Maybe this was already obvious to everyone else, but > it helped me.) > > 4) Even with my New and Improved Explanatory Powers, when this thread > came up chatting with Thomas Kluyver today, I attempted to provide a > simple, accurate description of how numpy indexing works so that the > debate would make sense, and his conclusion was (paraphrased) "okay, > now I don't understand numpy indexing anymore and never did". I say > this not to pick on Thomas, but to make that point that Thomas is a > pretty smart guy so maybe this is actually confusing. (Or maybe I'm > just terrible at explaining things.) > > I actually think the evidence is very very strong that numpy's current > way of mixing integer fancy indexing and slice-based indexing is a > mistake. It's just not clear whether there's anything we can do to > mitigate that mistake (or indeed, what would actually be better even > if we could start over from scratch). (Which we can't.) > I think the best way to think about the mixing is to think about "subspaces" defined by all of the slices which are taken for each individual fancy indexing "element". I.e. each subspaces is something like: new[:, 0, :] = arr[:, fancy1[0], fancy2[0], :] then you iterate the fancy indexes so the subspaces moves ahead: new[:, 1, :] = arr[:, fancy1[1], fancy2[1], :] new[:, 2, :] = arr[:, fancy1[2], fancy2[2], :] and so on. This is also how it is implemented. Plus of course the transposing to the front when the fancy indices are not consecutive and you cannot add the fancy dimensions to where they were. I think you mentioned an error in the docu, I thought I cleared some of them, but proabably that did not make it more understandable sometimes. The whole subspace way of is used, but there is a lot of improvement possible and I would be happy if more feel like stepping up to fill that void, since you do not need to know the implementation details for that. - Sebastian > > 2. One reason I use numpy in teaching is its indexing behavior. > > What specific language provides a better indexing model, > > in your opinion? > > > > 3. I admit, my students are NOT using non-boolen fancy indexing on > > multidimensional arrays. (As far as I know.) Are yours? > > Well, okay, this would explain it, since integer fancy indexing is > exactly the confusing case :-) On the plus side, this also means that > even if pigs started doing barrel-rolls through hell's > winter-vortex-chilled air tomorrow and we simply removed integer fancy > indexing, your students would be unaffected :-) > > -n > signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
