Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015
I thought I'd add a little more specifically about the kind of graphics/point cloud work I'm doing right now at Thinkbox, and how it relates. To echo Francesc's point about NumPy already being an industry standard, within the VFX/graphics industry there is a reference platform definition on Linux, and the most recent iteration of that specifies a version of NumPy. It also includes a bunch of other open source libraries worth taking a look at if you haven't seen them before: http://www.vfxplatform.com/ Point cloud/particle system data, mesh geometry, numerical grids (both dense and sparse), and many other primitive components in graphics are built out of arrays. What NumPy represents for that kind of data is amazing. The extra baggage of an API tied to the CPython GIL can be a hard pill to swallow, though, and this is one of the reasons I'm hopeful that as DyND continues maturing, it can make inroads into places NumPy hasn't been able to. Thanks, Mark On Wed, Aug 26, 2015 at 9:45 AM, Irwin Zaid iz...@continuum.io wrote: Hello everyone, Mark and I thought it would be good to weigh in here and also be explicitly around to discuss DyND. To be clear, neither of us has strong feelings on what NumPy *should* do -- we are both long-time NumPy users and we both see NumPy being around for a while. But, as Francesc mentioned, there is also the open question of where the community should be implementing new features. It would certainly be nice to not have duplication of effort, but a decision like that can only arise naturally from a broad consensus. Travis covered DyND's history and it's relationship with Continuum pretty well, so what's really missing here is what DyND is, where it is going, and how long we think it'll take to get there. We'll try to stick to those topics. We designed DyND to fill what we saw as fundamental gaps in NumPy. These are not only missing features, but also limitations of its architecture. Many of these gaps have been mentioned several times before in this thread and elsewhere, but a brief list would include: better support for missing values, variable-length strings, GPUs, more extensible types, categoricals, more datetime features, ... Some of these were indeed on Nathaniel's list and many of them are already working (albeit sometimes partially) in DyND. And, yes, we strongly feel that NumPy's fundamental dependence on Python itself is a limitation. Why should we not take the fantastic success of NumPy and generalize it across other languages? So, we see DyND is having a twofold purpose. The first is to expand upon the kinds of data that NumPy can represent and do computations upon. The second is to provide a standard array package that can cross the language barrier and easily interoperate between C++, Python, or whatever you want. DyND, at the moment, is quite functional in some areas and lacking a bit in others. There is no doubt that it is still experimental and a bit unstable. But, it has advanced by a lot recently, and we are steadily working towards something like a version 1.0. In fact, DyND's internal C++ architecture stabilized some time ago -- what's missing now is really solid coverage of some common use cases, alongside up-to-date Python bindings and an easy installation process. All of these are in progress and advancing as quick as we can make them. On the other hand, we are also building out some other features. To give just one example that might excite people, DyND now has Numba interoperability -- one can write DyND's equivalent of a ufunc in Python and, with a single decorator, have a broadcasting or reduction callable that gets JITed or (soon) ahead-of-time compiled. Over the next few months, we are hopeful that we can get DyND into a state where it is largely usable by those familiar with NumPy semantics. The reason why we can be a bit more aggressive in our timeline now is because of the great support we are getting from Continuum. With all that said, we are happy to be a part of of any broader conversation involving NumPy and the community. All the best, Irwin and Mark ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015
On Tue, Aug 25, 2015 at 12:21 PM, Antoine Pitrou solip...@pitrou.net wrote: On Tue, 25 Aug 2015 03:03:41 -0700 Nathaniel Smith n...@pobox.com wrote: Supporting third-party dtypes ~ [...] Some features that would become straightforward to implement (e.g. even in third-party libraries) if this were fixed: - missing value support - physical unit tracking (meters / seconds - array of velocity; meters + seconds - error) - better and more diverse datetime representations (e.g. datetimes with attached timezones, or using funky geophysical or astronomical calendars) - categorical data - variable length strings - strings-with-encodings (e.g. latin1) - forward mode automatic differentiation (write a function that computes f(x) where x is an array of float64; pass that function an array with a special dtype and get out both f(x) and f'(x)) - probably others I'm forgetting right now It should also be the opportunity to streamline datetime64 and timedelta64 dtypes. Currently the unit information is IIRC hidden in some weird metadata thing called the PyArray_DatetimeMetaData. Yeah, and PyArray_DatetimeMetaData is an NpyAuxData, which is its own personal little object system implemented in C with its own reference counting system... the design of dtypes has great bones, but the current implementation has a lot of, um, historical baggage. -n -- Nathaniel J. Smith -- http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015
Hi Travis, Thanks for taking the time to write up your thoughts! I have many thoughts in return, but I will try to restrict myself to two main ones :-). 1) On the question of whether work should be directed towards improving NumPy-as-it-is or instead towards a compatibility-breaking replacement: There's plenty of room for debate about whether it's better engineering practice to try and evolve an existing system in place versus starting over, and I guess we have some fundamental disagreements there, but I actually think this debate is a distraction -- we can agree to disagree, because in fact we have to try both. At a practical level: NumPy *is* going to continue to evolve, because it has users and people interested in evolving it; similarly, dynd and other alternatives libraries will also continue to evolve, because they also have people interested in doing it. And at a normative level, this is a good thing! If NumPy and dynd both get better, than that's awesome: the worst case is that NumPy adds the new features that we talked about at the meeting, and dynd simultaneously becomes so awesome that everyone wants to switch to it, and the result of this would be... that those NumPy features are exactly the ones that will make the transition to dynd easier. Or if some part of that plan goes wrong, then well, NumPy will still be there as a fallback, and in the mean time we've actually fixed the major pain points our users are begging us to fix. You seem to be urging us all to make a double-or-nothing wager that your extremely ambitious plans will all work out, with the entire numerical Python ecosystem as the stakes. I think this ambition is awesome, but maybe it'd be wise to hedge our bets a bit? 2) You really emphasize this idea of an ABI-breaking (but not API-breaking) release, and I think this must indicate some basic gap in how we're looking at things. Where I'm getting stuck here is that... I actually can't think of anything important that we can't do now, but could if we were allowed to break ABI compatibility. The kinds of things that break ABI but keep API are like... rearranging what order the fields in a struct fall in, or changing the numeric value of opaque constants like NPY_ARRAY_WRITEABLE. The biggest win I can think of is that we could save a few bytes per array by arranging the fields inside the ndarray struct more optimally, but that's hardly a feature to hang a 2.0 on. You seem to have a vision of this ABI-breaking release as being something very different from that, and I'm not clear on what this vision is. The main reason I personally am against having a big ABI-breaking release is not that I hate ABI breakage a priori, it's that all the big features that I care about and the are users are asking for seem to be ones that... don't actually require doing that. At most they seem to get a mild benefit from breaking some obscure corner cases. So the cost/benefits don't make any sense to me. So: can you give a concrete example of a change you have in mind where breaking ABI would be the key enabler? (I guess you might also be thinking of a separate issue that you sort of allude to: Perhaps we will try to make changes which we think don't involve breaking the ABI, but discover too late that we have failed to fully understand the implications and have broken it by mistake. IIUC this is what happened in the 1.4 timeframe when datetime64 was merged and accidentally renumbered some of the NPY_* constants. Partially I am less worried about this because I have a fair amount of confidence that our review and QA process has improved these days to the point that we would not let a change like that slip through by accident -- we have a lot more active reviewers, people are sensitized to the issues, we've successfully landed intrusive changes like Sebastian's indexing rewrite, ... though this is very much second-hand impressions on my part, and I'd welcome input from folks like Chuck who have a clearer view on how things have changed from then to now. But more importantly, even if this is true, then I can't see how your proposal helps. If we aren't good enough at our jobs to predict when we'll break ABI, then by assumption it makes no sense to pick one release and decide that this is the one time that we'll break ABI.) On Tue, Aug 25, 2015 at 12:00 PM, Travis Oliphant tra...@continuum.io wrote: Thanks for the write-up Nathaniel. There is a lot of great detail and interesting ideas here. I've am very eager to understand how to help NumPy and the wider community move forward however I can (my passions on this have not changed since 1999, though what I myself spend time on has changed). There are a lot of ways to think about approaching this, though. It's hard to get all the ideas on the table, and it was unfortunate we couldn't get everybody wyho are core NumPy devs together in person to have this discussion as there are still a lot of questions unanswered and a lot of thought
[Numpy-discussion] testing numpy with downstream testsuites (was: Re: Notes from the numpy dev meeting at scipy 2015)
[Popping this off to its own thread to try and keep things easier to follow] On Tue, Aug 25, 2015 at 9:52 AM, Nathan Goldbaum nathan12...@gmail.com wrote: - Lament: it would be really nice if we could get more people to test our beta releases, because in practice right now 1.x.0 ends up being where we actually the discover all the bugs, and 1.x.1 is where it actually becomes usable. Which sucks, and makes it difficult to have a solid policy about what counts as a regression, etc. Is there anything we can do about this? Just a note in here - have you all thought about running the test suites for downstream projects as part of the numpy test suite? I don't think it came up, but it's not a bad idea! The main problems I can foresee are: 1) Since we don't know the downstream code, it can be hard to interpret test suite failures. OTOH for changes we're uncertain of we already do often end up running some downstream test suites by hand, so it can only be an improvement on that... 2) Sometimes everyone including downstream agrees that breaking something is actually a good idea and they should just deal, but what do you do then? These both seem solvable though. I guess a good strategy would be to compile a travis-compatible wheel of $PACKAGE version $latest-stable against numpy 1.x, and then in the 1.(x+1) development period numpy would have an additional travis run which, instead of running the numpy test suite, instead does: pip install . pip install $PACKAGE-$latest-stable.whl python -c 'import package; package.test()' # adjust as necessary ? Where $PACKAGE is something like scipy / pandas / astropy / ... matplotlib would be nice but maybe impractical...? Maybe someone else will have objections but it seems like a reasonable idea to me. Want to put together a PR? Asides from fame and fortune and our earnest appreciation, your reward is you get to make sure that the packages you care about are included so that we break them less often in the future ;-). -n -- Nathaniel J. Smith -- http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015
On Tue, Aug 25, 2015 at 5:53 PM, David Cournapeau courn...@gmail.com wrote: Thanks for the good summary Nathaniel. Regarding dtype machinery, I agree casting is the hardest part. Unless the code has changed dramatically, this was the main reason why you could not make most of the dtypes separate from numpy codebase (I tried to move the datetime dtype out of multiarray into a separate C extension some years ago). Being able to separate the dtypes from the multiarray module would be an obvious way to drive the internal API change. For practical reasons I don't imagine we'll ever want to actually move the core dtypes out of multiarray -- if nothing else they will always remain a little bit special, like np.array([1.0, 2.0]) will just know that this should use the float64 dtype. But yeah, in general a good heuristic would be that -- aside from a few limited cases like that -- we want to make built-in dtypes and user-defined dtypes use the same APIs. Regarding the use of cython in numpy, was there any discussion about the compilation/size cost of using cython, and talking to the cython team to improve this ? Or was that considered acceptable with current cython for numpy. I am convinced cleanly separating the low level parts from the python C API plumbing would be the single most important thing one could do to make the codebase more amenable. It's still a more blue-sky idea than that... the discussion was more at the level of is this something that is even worth trying to make work and seeing where the problems are? The big immediate problem, before we got into code size issues, would be that we would need to be able to compile a mix of .pyx files and .c files into a single .so, while cython generated code currently makes some strong assumptions about how each .pyx file will live in its own .so. From playing around with it I suspect the first version of making this work will be klugey indeed. But yeah, the thing to do would be for someone to dig in and make the kluges and then decide how to clean them up once you know where they are. -n -- Nathaniel J. Smith -- http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015
On Wed, Aug 26, 2015 at 10:11 AM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 26 Aug 2015 16:45:51 + (UTC) Irwin Zaid iz...@continuum.io wrote: So, we see DyND is having a twofold purpose. The first is to expand upon the kinds of data that NumPy can represent and do computations upon. The second is to provide a standard array package that can cross the language barrier and easily interoperate between C++, Python, or whatever you want. One possible limitation is that the lingua franca for language interoperability is C, not C++. DyND doesn't have to be written in C, but exposing a nice C API may help make it attractive to the various language runtimes out there. (even those languages whose runtime doesn't have a compile-time interface to C generally have some kind of cffi or ctypes equivalent to load external C routines at runtime) I kind of like the path LLVM has chosen here, of a stable C API and an unstable C++ API. This has both pros and cons though, so I'm not sure what will be right for DyND in the long term. -Mark Regards Antoine. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy helper function for __getitem__?
Indeed, the helper function I wrote for xray was not designed to handle None/np.newaxis or non-1d Boolean indexers, because those are not valid indexers for xray objects. I think it could be straightforwardly extended to handle None simply by not counting them towards the total number of dimensions. On Tue, Aug 25, 2015 at 8:41 AM, Fabien fabien.mauss...@gmail.com wrote: I think that Stephan's function for xray is very useful. A possible improvement (probably at a certain performance cost) would be to be able to provide a shape instead of a number of dimensions. The output would then be slices with valid start and ends. Current behavior: In[9]: expanded_indexer(slice(None), 2) Out[9]: (slice(None, None, None), slice(None, None, None)) With shape: In[9]: expanded_indexer(slice(None), (3, 4)) Out[9]: (slice(0, 4, 1), slice(0, 5, 1)) But if nobody needed something like this before me, I think that I might have a design problem in my code (still quite new to python). Glad you found it helpful! Python's slice object has the indices method which implements this logic, e.g., In [15]: s = slice(None, 10) In [16]: s.indices(100) Out[16]: (0, 10, 1) Cheers, Stephan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015
On Wed, 26 Aug 2015 16:45:51 + (UTC) Irwin Zaid iz...@continuum.io wrote: So, we see DyND is having a twofold purpose. The first is to expand upon the kinds of data that NumPy can represent and do computations upon. The second is to provide a standard array package that can cross the language barrier and easily interoperate between C++, Python, or whatever you want. One possible limitation is that the lingua franca for language interoperability is C, not C++. DyND doesn't have to be written in C, but exposing a nice C API may help make it attractive to the various language runtimes out there. (even those languages whose runtime doesn't have a compile-time interface to C generally have some kind of cffi or ctypes equivalent to load external C routines at runtime) Regards Antoine. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015
26.08.2015, 14:14, Francesc Alted kirjoitti: [clip] 2015-08-25 12:03 GMT+02:00 Nathaniel Smith n...@pobox.com: Let's focus on evolving numpy as far as we can without major break-the-world changes (no numpy 2.0, at least in the foreseeable future). And, as a target for that evolution, let's change our focus from numpy as NumPy is the library that gives you the np.ndarray object (plus some attached infrastructure), to NumPy provides the standard framework for working with arrays and array-like objects in Python Sorry to disagree here, but in my opinion NumPy *already* provides the standard framework for working with arrays and array-like objects in Python as its huge popularity shows. If what you mean is that there are too many efforts trying to provide other, specialized data containers (things like DataFrame in pandas, DataArray/Dataset in xarray or carray/ctable in bcolz just to mention a few), then let me say that I am of the opinion that there can't be a silver bullet for tackling all the problems that the PyData community is facing. My reading of the above was that this was about multimethods, and allowing different types of containers to interoperate beyond the array interface and Python's builtin operator hooks. The exact performance details of course vary, and an algorithm written for in-memory arrays just fails for too large on-disk or distributed arrays. However, a case for a minimal common API probably could be made, esp. in algorithms mainly relying on linear algebra. This is to a degree different from subclassing, as many of the array-like objects you might want do not have a simple strided memory model. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015
On Wed, Aug 26, 2015 at 6:11 PM, Antoine Pitrou solip...@pitrou.net wrote: One possible limitation is that the lingua franca for language interoperability is C, not C++. DyND doesn't have to be written in C, but exposing a nice C API may help make it attractive to the various language runtimes out there. That is absolutely true and a C API is on the long-term roadmap. At the moment, a C API is not needed for DyND to be stable and usable from Python, which is one reason we aren't doing it now. Irwin ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
Hi, Splitting this one off too because it's a rather different discussion, although related. On Tue, Aug 25, 2015 at 11:03 AM, Nathaniel Smith n...@pobox.com wrote: [snip] Formalizing our governance/decision making == This was a major focus of discussion. At a high level, the consensus was to steal IPython's governance document (IPEP 29) and modify it to remove its use of a BDFL as a backstop to normal community consensus-based decision, and replace it with a new backstop based on Apache-project-style consensus voting amongst the core team. Here's a plea to avoid a 'core' structure if at all possible. Historically it seems to have some severe risks, and experienced people have blamed this structure for the decline of various projects including NetBSD and Xfree86, summaries here: http://asterisk.dynevor.org/melting-core.html http://asterisk.dynevor.org/xfree-forked.html In short, the core structure seems to be characteristically associated with a conservatism and lack of vision that causes the project to stagnate. There's also evidence from the NetBSD / OpenBSD split [1] and the XFree86 / X.org split [2] - that the core structure can lead to bad decisions being taken in private that no or few members of the core group are prepared to defend. I guess what is happening is that distributed responsibility leads to poor accountability, and therefore poor decisions. So, I hope very much we can avoid that trap in our own governance. Best, Matthew [1] http://mail-index.netbsd.org/netbsd-users/1994/12/23/.html [2] http://www.xfree86.org/pipermail/forum/2003-March/001997.html ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015
Hello everyone, Mark and I thought it would be good to weigh in here and also be explicitly around to discuss DyND. To be clear, neither of us has strong feelings on what NumPy *should* do -- we are both long-time NumPy users and we both see NumPy being around for a while. But, as Francesc mentioned, there is also the open question of where the community should be implementing new features. It would certainly be nice to not have duplication of effort, but a decision like that can only arise naturally from a broad consensus. Travis covered DyND's history and it's relationship with Continuum pretty well, so what's really missing here is what DyND is, where it is going, and how long we think it'll take to get there. We'll try to stick to those topics. We designed DyND to fill what we saw as fundamental gaps in NumPy. These are not only missing features, but also limitations of its architecture. Many of these gaps have been mentioned several times before in this thread and elsewhere, but a brief list would include: better support for missing values, variable-length strings, GPUs, more extensible types, categoricals, more datetime features, ... Some of these were indeed on Nathaniel's list and many of them are already working (albeit sometimes partially) in DyND. And, yes, we strongly feel that NumPy's fundamental dependence on Python itself is a limitation. Why should we not take the fantastic success of NumPy and generalize it across other languages? So, we see DyND is having a twofold purpose. The first is to expand upon the kinds of data that NumPy can represent and do computations upon. The second is to provide a standard array package that can cross the language barrier and easily interoperate between C++, Python, or whatever you want. DyND, at the moment, is quite functional in some areas and lacking a bit in others. There is no doubt that it is still experimental and a bit unstable. But, it has advanced by a lot recently, and we are steadily working towards something like a version 1.0. In fact, DyND's internal C++ architecture stabilized some time ago -- what's missing now is really solid coverage of some common use cases, alongside up-to-date Python bindings and an easy installation process. All of these are in progress and advancing as quick as we can make them. On the other hand, we are also building out some other features. To give just one example that might excite people, DyND now has Numba interoperability -- one can write DyND's equivalent of a ufunc in Python and, with a single decorator, have a broadcasting or reduction callable that gets JITed or (soon) ahead-of-time compiled. Over the next few months, we are hopeful that we can get DyND into a state where it is largely usable by those familiar with NumPy semantics. The reason why we can be a bit more aggressive in our timeline now is because of the great support we are getting from Continuum. With all that said, we are happy to be a part of of any broader conversation involving NumPy and the community. All the best, Irwin and Mark ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
Hi Matthew On 2015-08-26 10:50:47, Matthew Brett matthew.br...@gmail.com wrote: In short, the core structure seems to be characteristically associated with a conservatism and lack of vision that causes the project to stagnate. Can you describe how a democratic governance structure would look? It's not clear from the discussions linked where successful examples are to be found. Thanks Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Defining a white noise process using numpy
Hi all, Can anyone give me some advice for translating this equation into code using numpy? eta(t) = lim(dt - 0) N(0, 1/sqrt(dt)), where N(a, b) is a Gaussian random variable of mean a and variance b**2. This is a heuristic definition of a white noise process. Thanks, Dan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] testing numpy with downstream testsuites (was: Re: Notes from the numpy dev meeting at scipy 2015)
As a Matplotlib developer I try to test our code manually with all betas and rc of new numpy versions. (And already pushed fixed a few new deprecation warnings with 1.10beta1 which otherwise passes our test suite. I forgot to report this back since there were no issues to report ) However, we could actually do this automatically if numpy betas were uploaded as prereleases on pypi. We are already using Travis's allow failure mode to test python 3.5 betas and rc's along with all our dependencies installed with `pip --pre` https://pip.pypa.io/en/latest/reference/pip_install.html#pre-release-versions Putting prereleases on pypi would thus automate most of the testing of new Numpy versions for us. Best Jens ons. 26. aug. 2015 kl. 07.59 skrev Nathaniel Smith n...@pobox.com: [Popping this off to its own thread to try and keep things easier to follow] On Tue, Aug 25, 2015 at 9:52 AM, Nathan Goldbaum nathan12...@gmail.com wrote: - Lament: it would be really nice if we could get more people to test our beta releases, because in practice right now 1.x.0 ends up being where we actually the discover all the bugs, and 1.x.1 is where it actually becomes usable. Which sucks, and makes it difficult to have a solid policy about what counts as a regression, etc. Is there anything we can do about this? Just a note in here - have you all thought about running the test suites for downstream projects as part of the numpy test suite? I don't think it came up, but it's not a bad idea! The main problems I can foresee are: 1) Since we don't know the downstream code, it can be hard to interpret test suite failures. OTOH for changes we're uncertain of we already do often end up running some downstream test suites by hand, so it can only be an improvement on that... 2) Sometimes everyone including downstream agrees that breaking something is actually a good idea and they should just deal, but what do you do then? These both seem solvable though. I guess a good strategy would be to compile a travis-compatible wheel of $PACKAGE version $latest-stable against numpy 1.x, and then in the 1.(x+1) development period numpy would have an additional travis run which, instead of running the numpy test suite, instead does: pip install . pip install $PACKAGE-$latest-stable.whl python -c 'import package; package.test()' # adjust as necessary ? Where $PACKAGE is something like scipy / pandas / astropy / ... matplotlib would be nice but maybe impractical...? Maybe someone else will have objections but it seems like a reasonable idea to me. Want to put together a PR? Asides from fame and fortune and our earnest appreciation, your reward is you get to make sure that the packages you care about are included so that we break them less often in the future ;-). -n -- Nathaniel J. Smith -- http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015
On Mi, 2015-08-26 at 00:05 -0700, Nathaniel Smith wrote: On Tue, Aug 25, 2015 at 5:53 PM, David Cournapeau courn...@gmail.com wrote: Thanks for the good summary Nathaniel. Regarding dtype machinery, I agree casting is the hardest part. Unless the code has changed dramatically, this was the main reason why you could not make most of the dtypes separate from numpy codebase (I tried to move the datetime dtype out of multiarray into a separate C extension some years ago). Being able to separate the dtypes from the multiarray module would be an obvious way to drive the internal API change. For practical reasons I don't imagine we'll ever want to actually move the core dtypes out of multiarray -- if nothing else they will always remain a little bit special, like np.array([1.0, 2.0]) will just know that this should use the float64 dtype. But yeah, in general a good heuristic would be that -- aside from a few limited cases like that -- we want to make built-in dtypes and user-defined dtypes use the same APIs. Well, casting is the conceptional hardest part. Marrying it to the rest of numpy is probably just as hard ;). With the chance of not having thought this through enough, maybe some points about the general discussion. I think I would like some more clarity of what we want and especially *need* [1]. From SciPy, there were two things I particularly remember: 1. the dtype/scalar issue 2. making an interface to make array-likes interaction more sane (this I think can go quite far, and we are already going part of it) The dtypes/scalars seem a particularly dark corner of numpy and if it is feasible for us to replace it with something new, then I would be willing to do some breaks for it (admittingly, given protest, I would back down from that and another solution would be needed). The point for me is, I currently think a dtype/scalar could get numpy a big way, especially from the point of view of downstream packages. Of course it would be harder to do in numpy then in something new, but it should also be of much more immediate use. Maybe I am going a bit too far with this right now, but I could imagine that if we cannot clean up the dtype/scalars, numpy may indeed be doomed or at least a brick slowing down a lot of other people. And if it is not possible to do this without a numpy 2, then likely that is the way to go. But I am not convinced we should aim to fix all the other stuff at the same time. I am afraid it would just accumulate to grow over everyones heads. In other words, I think if we can muster the resources I would like to see this problem attacked within numpy. If this proves impossible a new dtype abstraction may well be reason for numpy 2, or used by a DyND or similar? But I do believe we should not give up on Numpy here from the start, at least I do not see a compelling reason to do. Instead giving up on numpy seems like the last way out of a misery. And much of the different opinions to me seem to be whether we think this will clearly happen or not or has already happened (or maybe whether it is too costly to do in numpy). Cleaning it up, would open doors to many things. Note that I think it would make the numpy source much less scary, because I think it is the one big piece of code that is maybe not clearly a separate chunk [2]. After making it sane, I would argue that numpy does become much more maintainable and extensible. From my current view, probably enough so for a long time. Also, I think it would give us abstraction to make different/new projects work together better and if done well enough, some grand new project set to replace numpy could reuse it. Of course it is entirely possible that more things need to be changed in numpy and that some others would be just as hard or even harder to do. But if we can identify this as the one big thing that gets us 90% then I refuse to give up hope of doing it in numpy just yet. - Sebastian [1] Travis has said quite a lot about it, but it is not yet clear to me what is a priority/real pain point. Take datashape for example. By now I think that the datashape is likely a good idea to make structured arrays nicer, since it moves the structured part into the array object and not the dtype, which makes sense to me. However, I am not convinced that the datashape is something that would make numpy a compelling amount better. In fact I could imagine that for many things it would make it unnecessarily more complicated for users. [2] Take indexing, I like to think I did not break that much when redoing it (except on purpose, which I hope did not create much trouble). In some sense indexing was simple to redo, because it does not overlap at all with anything else directly. If we get dtypes/scalars more separated, I think we are at a point where this is possible with pretty much any part of numpy. Regarding the use of cython in numpy, was there any discussion about the compilation/size cost of using cython, and
[Numpy-discussion] SHA256 mismatch on SourceForge downloads
Hello, The SourceForge download page for 1.10.0b1 mentions: 89e467cec774527dd254c1e039801726db1367433053801f0d8bc68deac74009 numpy-1.10.0b1.tar.gz But after downloading the file I get: $ sha256sum numpy-1.10.0b1.tar.gz 855695405092686264dc8ce7b3f5c939a6cf1a5639833e841a5bb6fb799cd6a8 numpy-1.10.0b1.tar.gz Also, since SouceForge doesn't provide any HTTPS downloads (it actually redirects HTTPS to HTTP (*)), this all looks a bit pointless. (*) seems like SourceForge is becoming a poster child of worst practices... Regards Antoine. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] SHA256 mismatch on SourceForge downloads
The file is also not signed so the checksums are not trustworthy anyway. Please sign the releases as we did in the past. On 08/26/2015 10:28 AM, Antoine Pitrou wrote: Hello, The SourceForge download page for 1.10.0b1 mentions: 89e467cec774527dd254c1e039801726db1367433053801f0d8bc68deac74009 numpy-1.10.0b1.tar.gz But after downloading the file I get: $ sha256sum numpy-1.10.0b1.tar.gz 855695405092686264dc8ce7b3f5c939a6cf1a5639833e841a5bb6fb799cd6a8 numpy-1.10.0b1.tar.gz Also, since SouceForge doesn't provide any HTTPS downloads (it actually redirects HTTPS to HTTP (*)), this all looks a bit pointless. (*) seems like SourceForge is becoming a poster child of worst practices... Regards Antoine. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] SHA256 mismatch on SourceForge downloads
On Wed, Aug 26, 2015 at 2:28 AM, Antoine Pitrou solip...@pitrou.net wrote: Hello, The SourceForge download page for 1.10.0b1 mentions: 89e467cec774527dd254c1e039801726db1367433053801f0d8bc68deac74009 numpy-1.10.0b1.tar.gz But after downloading the file I get: $ sha256sum numpy-1.10.0b1.tar.gz 855695405092686264dc8ce7b3f5c939a6cf1a5639833e841a5bb6fb799cd6a8 numpy-1.10.0b1.tar.gz Also, since SouceForge doesn't provide any HTTPS downloads (it actually redirects HTTPS to HTTP (*)), this all looks a bit pointless. (*) seems like SourceForge is becoming a poster child of worst practices... I know what happened there The original tarball generated by numpy-vendor was missing a file, so I uploaded new tar and zip files but neglected to change the sha256 signature. My bad. I'll try to do better for the 1.10.0rc1 Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015
Hi, Thanks Nathaniel and others for sparking this discussion as I think it is very timely. 2015-08-25 12:03 GMT+02:00 Nathaniel Smith n...@pobox.com: Let's focus on evolving numpy as far as we can without major break-the-world changes (no numpy 2.0, at least in the foreseeable future). And, as a target for that evolution, let's change our focus from numpy as NumPy is the library that gives you the np.ndarray object (plus some attached infrastructure), to NumPy provides the standard framework for working with arrays and array-like objects in Python Sorry to disagree here, but in my opinion NumPy *already* provides the standard framework for working with arrays and array-like objects in Python as its huge popularity shows. If what you mean is that there are too many efforts trying to provide other, specialized data containers (things like DataFrame in pandas, DataArray/Dataset in xarray or carray/ctable in bcolz just to mention a few), then let me say that I am of the opinion that there can't be a silver bullet for tackling all the problems that the PyData community is facing. The libraries using specialized data containers (pandas, xray, bcolz...) may have more or less machinery on top of them so that conversion to NumPy not necessarily happens internally (many times we don't want conversions for efficiency), but it is the capability of producing NumPy arrays out of them (or parts of them) what makes these specialized containers to be incredible more useful to users because they can use NumPy to fill the missing gaps, or just use NumPy as an intermediate container that acts as input for other libraries. On the subject on why I don't think a universal data container is feasible for PyData, you just have to have a look at how many data structures Python is providing in the language itself (tuples, lists, dicts, sets...), and how many are added in the standard library (like those in the collections sub-package). Every data container is designed to do a couple of things (maybe three) well, but for other use cases it is the responsibility of the user to choose the more appropriate depending on her needs. In the same vein, I also think that it makes little sense to try to come with a standard solution that is going to satisfy everyone's need. IMHO, and despite all efforts, neither NumPy, NumPy 2.0, DyND, bcolz or any other is going to offer the universal data container. Instead of that, let me summarize what users/developers like me need from NumPy for continue creating more specialized data containers: 1) Keep NumPy simple. NumPy is the truly cornerstone of PyData right now, and it will be for the foreseeable future, so please keep it usable and *minimal*. Before adding any more feature the increase in complexity should carefully weighted. 2) Make NumPy more flexible. Any rewrite that allows arrays or dtypes to be subclassed and extended more easily will be a huge win. *But* if in order to allow flexibility you have to make NumPy much more complex, then point 1) should prevail. 3) Make of NumPy a sustainable project. Historically NumPy depended on heroic efforts of individuals to make it what it is now: *an industry standard*. But individual efforts, while laudable, are not enough, so please, please, please continue the effort of constituting a governance team that ensures the future of NumPy (and with it, the whole PyData community). Finally, the question on whether NumPy 2.0 or projects like DyND should be chosen instead for implementing new features is still legitimate, and while I have my own opinions (favourable to DyND), I still see (such is the price of technological debt) a distant future where we will find NumPy as we know it, allowing more innovation to happen in Python Data space. Again, thanks to all those braves that are allowing others to build on top of NumPy's shoulders. -- Francesc Alted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] testing numpy with downstream testsuites (was: Re: Notes from the numpy dev meeting at scipy 2015)
Hi, On Wed, Aug 26, 2015 at 7:59 AM, Nathaniel Smith n...@pobox.com wrote: [Popping this off to its own thread to try and keep things easier to follow] On Tue, Aug 25, 2015 at 9:52 AM, Nathan Goldbaum nathan12...@gmail.com wrote: - Lament: it would be really nice if we could get more people to test our beta releases, because in practice right now 1.x.0 ends up being where we actually the discover all the bugs, and 1.x.1 is where it actually becomes usable. Which sucks, and makes it difficult to have a solid policy about what counts as a regression, etc. Is there anything we can do about this? Just a note in here - have you all thought about running the test suites for downstream projects as part of the numpy test suite? I don't think it came up, but it's not a bad idea! The main problems I can foresee are: 1) Since we don't know the downstream code, it can be hard to interpret test suite failures. OTOH for changes we're uncertain of we already do often end up running some downstream test suites by hand, so it can only be an improvement on that... 2) Sometimes everyone including downstream agrees that breaking something is actually a good idea and they should just deal, but what do you do then? These both seem solvable though. I guess a good strategy would be to compile a travis-compatible wheel of $PACKAGE version $latest-stable against numpy 1.x, and then in the 1.(x+1) development period numpy would have an additional travis run which, instead of running the numpy test suite, instead does: pip install . pip install $PACKAGE-$latest-stable.whl python -c 'import package; package.test()' # adjust as necessary ? Where $PACKAGE is something like scipy / pandas / astropy / ... matplotlib would be nice but maybe impractical...? Maybe someone else will have objections but it seems like a reasonable idea to me. Want to put together a PR? Asides from fame and fortune and our earnest appreciation, your reward is you get to make sure that the packages you care about are included so that we break them less often in the future ;-). One simple way to get going would be for the release manager to trigger a build from this repo: https://github.com/matthew-brett/travis-wheel-builder This build would then upload a wheel to: http://travis-wheels.scikit-image.org/ The upstream packages would have a test grid which included an entry with something like: pip install -f http://travis-wheels.scikit-image.org --pre numpy Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1.10.0rc1
On Wed, Aug 26, 2015 at 7:32 AM, Charles R Harris charlesr.har...@gmail.com
wrote:
On Wed, Aug 26, 2015 at 7:31 AM, Charles R Harris
charlesr.har...@gmail.com wrote:
On Wed, Aug 26, 2015 at 7:11 AM, Antoine Pitrou solip...@pitrou.net
wrote:
On Tue, 25 Aug 2015 10:26:02 -0600
Charles R Harris charlesr.har...@gmail.com wrote:
Hi All,
The silence after the 1.10 beta has been eerie. Consequently, I'm
thinking
of making a first release candidate this weekend. If you haven't yet
tested
the beta, please do so. It would be good to discover as many problems
as we
can before the first release.
Has typing of ufunc parameters become much stricter? I can't find
anything in the release notes, but see (1.10b1):
arr = np.linspace(0, 5, 10)
out = np.empty_like(arr, dtype=np.intp)
np.round(arr, out=out)
Traceback (most recent call last):
File stdin, line 1, in module
File
/home/antoine/np110/lib/python3.4/site-packages/numpy/core/fromnumeric.py,
line 2778, in round_
return round(decimals, out)
TypeError: ufunc 'rint' output (typecode 'd') could not be coerced to
provided output parameter (typecode 'l') according to the casting rule
''same_kind''
It used to work (1.9):
arr = np.linspace(0, 5, 10)
out = np.empty_like(arr, dtype=np.intp)
np.round(arr, out=out)
array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
out
array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
The default casting mode has been changed. I think this has been raising
a warning since 1.7 and was mentioned as a future change in 1.10, but you
are right, it needs to be mentioned in the 1.10 release notes.
Make that warned of in the 1.9.0 release notes.
Here it is in 1.9.0 with deprecation warning made visible.
```
In [3]: import warnings
In [4]: warnings.simplefilter('always')
In [5]: arr = np.linspace(0, 5, 10)
In [6]: out = np.empty_like(arr, dtype=np.intp)
In [7]: np.round(arr, out=out)
/home/charris/.local/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2640:
DeprecationWarning: Implicitly casting between incompatible kinds. In a
future numpy release, this will raise an error. Use casting=unsafe if
this is intentional.
return round(decimals, out)
Out[7]: array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
```
Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015
On Wed, Aug 26, 2015 at 1:41 AM, Nathaniel Smith n...@pobox.com wrote: Hi Travis, Thanks for taking the time to write up your thoughts! I have many thoughts in return, but I will try to restrict myself to two main ones :-). 1) On the question of whether work should be directed towards improving NumPy-as-it-is or instead towards a compatibility-breaking replacement: There's plenty of room for debate about whether it's better engineering practice to try and evolve an existing system in place versus starting over, and I guess we have some fundamental disagreements there, but I actually think this debate is a distraction -- we can agree to disagree, because in fact we have to try both. Yes, on this we agree. I think NumPy can improve *and* we can have new innovative array objects. I don't disagree about that. At a practical level: NumPy *is* going to continue to evolve, because it has users and people interested in evolving it; similarly, dynd and other alternatives libraries will also continue to evolve, because they also have people interested in doing it. And at a normative level, this is a good thing! If NumPy and dynd both get better, than that's awesome: the worst case is that NumPy adds the new features that we talked about at the meeting, and dynd simultaneously becomes so awesome that everyone wants to switch to it, and the result of this would be... that those NumPy features are exactly the ones that will make the transition to dynd easier. Or if some part of that plan goes wrong, then well, NumPy will still be there as a fallback, and in the mean time we've actually fixed the major pain points our users are begging us to fix. You seem to be urging us all to make a double-or-nothing wager that your extremely ambitious plans will all work out, with the entire numerical Python ecosystem as the stakes. I think this ambition is awesome, but maybe it'd be wise to hedge our bets a bit? You are mis-characterizing my view. I think NumPy can evolve (though I would personally rather see a bigger change to the underlying system like I outlined before).But, I don't believe it can even evolve easily in the direction needed without breaking ABI and that insisting on not breaking it or even putting too much effort into not breaking it will continue to create less-optimal solutions that are harder to maintain and do not take advantage of knowledge this community now has. I'm also very concerned that 'evolving' NumPy will create a situation where there are regular semantic and subtle API changes that will cause NumPy to be less stable for it's user-base.I've watched this happen. This at a time that people are already looking around for new and different approaches anyway. 2) You really emphasize this idea of an ABI-breaking (but not API-breaking) release, and I think this must indicate some basic gap in how we're looking at things. Where I'm getting stuck here is that... I actually can't think of anything important that we can't do now, but could if we were allowed to break ABI compatibility. The kinds of things that break ABI but keep API are like... rearranging what order the fields in a struct fall in, or changing the numeric value of opaque constants like NPY_ARRAY_WRITEABLE. The biggest win I can think of is that we could save a few bytes per array by arranging the fields inside the ndarray struct more optimally, but that's hardly a feature to hang a 2.0 on. You seem to have a vision of this ABI-breaking release as being something very different from that, and I'm not clear on what this vision is. We already broke the ABI with date-time changes --- it's still broken for a certain percentage of users last I checked.So, part of my disagreement is that we've tried this and it didn't work --- even though smart people thought it would.I've had to deal with this personally and I'm not enthusiastic about having to deal with this for the next 5 years because of even more attempts to make changes while not breaking the ABI.I think the group is more careful now --- but I still think the API is broad enough and uses of NumPy deep enough that the effort involved in trying not to break the ABI is just not worth the effort (because it's a non-feature today).Adding new dtypes without breaking the ABI is tricky (and to do it without breaking the ABI is ugly). I also continue to believe that putting out a new ABI-breaking NumPy will allow re-compiling *once* (with some porting changes needed) and not subtle breakages requiring code-changes every time a release is made.If subtle changes aren't made, then the new features won't come. Right now, I'd rather have stability from NumPy than new features. New features can come from other libraries. One specific change that could easily be made in NumPy 2.0 (the current code but with an ABI change) is that Dtypes should become true type objects and array-scalars (which are the current
Re: [Numpy-discussion] 1.10.0rc1
On Tue, 25 Aug 2015 10:26:02 -0600 Charles R Harris charlesr.har...@gmail.com wrote: Hi All, The silence after the 1.10 beta has been eerie. Consequently, I'm thinking of making a first release candidate this weekend. If you haven't yet tested the beta, please do so. It would be good to discover as many problems as we can before the first release. Has typing of ufunc parameters become much stricter? I can't find anything in the release notes, but see (1.10b1): arr = np.linspace(0, 5, 10) out = np.empty_like(arr, dtype=np.intp) np.round(arr, out=out) Traceback (most recent call last): File stdin, line 1, in module File /home/antoine/np110/lib/python3.4/site-packages/numpy/core/fromnumeric.py, line 2778, in round_ return round(decimals, out) TypeError: ufunc 'rint' output (typecode 'd') could not be coerced to provided output parameter (typecode 'l') according to the casting rule ''same_kind'' It used to work (1.9): arr = np.linspace(0, 5, 10) out = np.empty_like(arr, dtype=np.intp) np.round(arr, out=out) array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5]) out array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5]) Regards Antoine. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] testing numpy with downstream testsuites (was: Re: Notes from the numpy dev meeting at scipy 2015)
Pandas has for quite a while has a travis build where we install numpy master and then run our test suite. e.g. here: https://travis-ci.org/pydata/pandas/jobs/77256007 Over the last year this has uncovered a couple of changes which affected pandas (mainly using something deprecated which was turned off :) This was pretty simple to setup. Note that this adds 2+ minutes to the build (though our builds take a while anyhow so its not a big deal). On Wed, Aug 26, 2015 at 7:14 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Wed, Aug 26, 2015 at 7:59 AM, Nathaniel Smith n...@pobox.com wrote: [Popping this off to its own thread to try and keep things easier to follow] On Tue, Aug 25, 2015 at 9:52 AM, Nathan Goldbaum nathan12...@gmail.com wrote: - Lament: it would be really nice if we could get more people to test our beta releases, because in practice right now 1.x.0 ends up being where we actually the discover all the bugs, and 1.x.1 is where it actually becomes usable. Which sucks, and makes it difficult to have a solid policy about what counts as a regression, etc. Is there anything we can do about this? Just a note in here - have you all thought about running the test suites for downstream projects as part of the numpy test suite? I don't think it came up, but it's not a bad idea! The main problems I can foresee are: 1) Since we don't know the downstream code, it can be hard to interpret test suite failures. OTOH for changes we're uncertain of we already do often end up running some downstream test suites by hand, so it can only be an improvement on that... 2) Sometimes everyone including downstream agrees that breaking something is actually a good idea and they should just deal, but what do you do then? These both seem solvable though. I guess a good strategy would be to compile a travis-compatible wheel of $PACKAGE version $latest-stable against numpy 1.x, and then in the 1.(x+1) development period numpy would have an additional travis run which, instead of running the numpy test suite, instead does: pip install . pip install $PACKAGE-$latest-stable.whl python -c 'import package; package.test()' # adjust as necessary ? Where $PACKAGE is something like scipy / pandas / astropy / ... matplotlib would be nice but maybe impractical...? Maybe someone else will have objections but it seems like a reasonable idea to me. Want to put together a PR? Asides from fame and fortune and our earnest appreciation, your reward is you get to make sure that the packages you care about are included so that we break them less often in the future ;-). One simple way to get going would be for the release manager to trigger a build from this repo: https://github.com/matthew-brett/travis-wheel-builder This build would then upload a wheel to: http://travis-wheels.scikit-image.org/ The upstream packages would have a test grid which included an entry with something like: pip install -f http://travis-wheels.scikit-image.org --pre numpy Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1.10.0rc1
On Wed, Aug 26, 2015 at 7:31 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Wed, Aug 26, 2015 at 7:11 AM, Antoine Pitrou solip...@pitrou.net wrote: On Tue, 25 Aug 2015 10:26:02 -0600 Charles R Harris charlesr.har...@gmail.com wrote: Hi All, The silence after the 1.10 beta has been eerie. Consequently, I'm thinking of making a first release candidate this weekend. If you haven't yet tested the beta, please do so. It would be good to discover as many problems as we can before the first release. Has typing of ufunc parameters become much stricter? I can't find anything in the release notes, but see (1.10b1): arr = np.linspace(0, 5, 10) out = np.empty_like(arr, dtype=np.intp) np.round(arr, out=out) Traceback (most recent call last): File stdin, line 1, in module File /home/antoine/np110/lib/python3.4/site-packages/numpy/core/fromnumeric.py, line 2778, in round_ return round(decimals, out) TypeError: ufunc 'rint' output (typecode 'd') could not be coerced to provided output parameter (typecode 'l') according to the casting rule ''same_kind'' It used to work (1.9): arr = np.linspace(0, 5, 10) out = np.empty_like(arr, dtype=np.intp) np.round(arr, out=out) array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5]) out array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5]) The default casting mode has been changed. I think this has been raising a warning since 1.7 and was mentioned as a future change in 1.10, but you are right, it needs to be mentioned in the 1.10 release notes. Make that warned of in the 1.9.0 release notes. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1.10.0rc1
On Wed, Aug 26, 2015 at 7:11 AM, Antoine Pitrou solip...@pitrou.net wrote: On Tue, 25 Aug 2015 10:26:02 -0600 Charles R Harris charlesr.har...@gmail.com wrote: Hi All, The silence after the 1.10 beta has been eerie. Consequently, I'm thinking of making a first release candidate this weekend. If you haven't yet tested the beta, please do so. It would be good to discover as many problems as we can before the first release. Has typing of ufunc parameters become much stricter? I can't find anything in the release notes, but see (1.10b1): arr = np.linspace(0, 5, 10) out = np.empty_like(arr, dtype=np.intp) np.round(arr, out=out) Traceback (most recent call last): File stdin, line 1, in module File /home/antoine/np110/lib/python3.4/site-packages/numpy/core/fromnumeric.py, line 2778, in round_ return round(decimals, out) TypeError: ufunc 'rint' output (typecode 'd') could not be coerced to provided output parameter (typecode 'l') according to the casting rule ''same_kind'' It used to work (1.9): arr = np.linspace(0, 5, 10) out = np.empty_like(arr, dtype=np.intp) np.round(arr, out=out) array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5]) out array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5]) The default casting mode has been changed. I think this has been raising a warning since 1.7 and was mentioned as a future change in 1.10, but you are right, it needs to be mentioned in the 1.10 release notes. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] UTC-based datetime64
Hi, We've found that NumPy uses the local TZ for printing datetime64 timestamps: In [22]: t = datetime.utcnow() In [23]: print t 2015-08-26 11:52:10.662745 In [24]: np.array([t], dtype=datetime64[s]) Out[24]: array(['2015-08-26T13:52:10+0200'], dtype='datetime64[s]') Googling for a way to print UTC out of the box, the best thing I could find is: In [40]: [str(i.item()) for i in np.array([t], dtype=datetime64[s])] Out[40]: ['2015-08-26 11:52:10'] Now, is there a better way to specify that I want the datetimes printed always in UTC? Thanks, -- Francesc Alted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1.10.0rc1
Just a data point, I just tested 1.9.0rc1 (built from source) with
matplotlib master, and things appear to be fine there. In fact, matplotlib
was built against 1.7.x (I was hunting down a regression), and worked
against the 1.9.0 install, so the ABI appears intact.
Cheers!
Ben Root
On Wed, Aug 26, 2015 at 9:52 AM, Charles R Harris charlesr.har...@gmail.com
wrote:
On Wed, Aug 26, 2015 at 7:32 AM, Charles R Harris
charlesr.har...@gmail.com wrote:
On Wed, Aug 26, 2015 at 7:31 AM, Charles R Harris
charlesr.har...@gmail.com wrote:
On Wed, Aug 26, 2015 at 7:11 AM, Antoine Pitrou solip...@pitrou.net
wrote:
On Tue, 25 Aug 2015 10:26:02 -0600
Charles R Harris charlesr.har...@gmail.com wrote:
Hi All,
The silence after the 1.10 beta has been eerie. Consequently, I'm
thinking
of making a first release candidate this weekend. If you haven't yet
tested
the beta, please do so. It would be good to discover as many problems
as we
can before the first release.
Has typing of ufunc parameters become much stricter? I can't find
anything in the release notes, but see (1.10b1):
arr = np.linspace(0, 5, 10)
out = np.empty_like(arr, dtype=np.intp)
np.round(arr, out=out)
Traceback (most recent call last):
File stdin, line 1, in module
File
/home/antoine/np110/lib/python3.4/site-packages/numpy/core/fromnumeric.py,
line 2778, in round_
return round(decimals, out)
TypeError: ufunc 'rint' output (typecode 'd') could not be coerced to
provided output parameter (typecode 'l') according to the casting rule
''same_kind''
It used to work (1.9):
arr = np.linspace(0, 5, 10)
out = np.empty_like(arr, dtype=np.intp)
np.round(arr, out=out)
array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
out
array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
The default casting mode has been changed. I think this has been raising
a warning since 1.7 and was mentioned as a future change in 1.10, but you
are right, it needs to be mentioned in the 1.10 release notes.
Make that warned of in the 1.9.0 release notes.
Here it is in 1.9.0 with deprecation warning made visible.
```
In [3]: import warnings
In [4]: warnings.simplefilter('always')
In [5]: arr = np.linspace(0, 5, 10)
In [6]: out = np.empty_like(arr, dtype=np.intp)
In [7]: np.round(arr, out=out)
/home/charris/.local/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2640:
DeprecationWarning: Implicitly casting between incompatible kinds. In a
future numpy release, this will raise an error. Use casting=unsafe if
this is intentional.
return round(decimals, out)
Out[7]: array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
```
Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1.10.0rc1
On Aug 26, 2015 7:03 PM, Benjamin Root ben.v.r...@gmail.com wrote: Just a data point, I just tested 1.9.0rc1 (built from source) with matplotlib master, and things appear to be fine there. In fact, matplotlib was built against 1.7.x (I was hunting down a regression), and worked against the 1.9.0 install, so the ABI appears intact. 1.9, or 1.10? -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1.10.0rc1
Aw, crap... I looked at the list of tags and saw the rc1... I'll test again in the morning Grumble, grumble... On Aug 26, 2015 10:53 PM, Nathaniel Smith n...@pobox.com wrote: On Aug 26, 2015 7:03 PM, Benjamin Root ben.v.r...@gmail.com wrote: Just a data point, I just tested 1.9.0rc1 (built from source) with matplotlib master, and things appear to be fine there. In fact, matplotlib was built against 1.7.x (I was hunting down a regression), and worked against the 1.9.0 install, so the ABI appears intact. 1.9, or 1.10? -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
