Re: [Numpy-discussion] Refactor fork uses the ./configure, make, make install process.
Sorry for the late reply... I missed this thread. Thanks to Ilan for pointing it out. A variety of comments below... On Sat, Dec 4, 2010 at 10:20 AM, Charles R Harrischarlesr.har...@gmail.com wrote: Just wondering if this is temporary or the intention is to change the build process? I also note that the *.h files in libndarray are not complete and a *lot* of trailing whitespace has crept into the files. For the purposes of our immediate project the intent is to use autoconf since it's widely available and makes building this part Python-independent and easier than working it into both distutils and numscons. Going forward it's certainly open to discussion. Currently all of the .h and .c files are generated as a part of the build rather than being checked in just because it saves a build step. Checking in the intermediate files isn't a problem either. Does the trailing whitespace cause problems? We saw it in the coding guidelines and planned to run a filter over it once the code stabilizes, but none of us had seen a guideline like that before and weren't sure why it was there. On Sat, Dec 4, 2010 at 3:01 PM, Charles R Harris charlesr.har...@gmail.comwrote: On Sat, Dec 4, 2010 at 1:45 PM, Pauli Virtanen p...@iki.fi wrote: On Sat, 04 Dec 2010 14:24:49 -0600, Ilan Schnell wrote: I'm not sure how reasonable it would be to move only libndarray into the master, because I've been working on EPD for the last couple of week. But Jason will know how complete libndarray is. The main question is whether moving it will make things easier or more difficult, I think. It's one tree more to keep track of. In any case, it would be a first part in the merge, and it would split the hunk of changes into two parts. That would be a good thing IMHO. It would also bring a bit more numpy reality to the refactor and since we are implicitly relying on it for the next release sometime next spring the closer to reality it gets the better. *** Technically, the move could be done like this, so that merge tracking still works: refactor--- new-refactor // /libndarray--x / \ start-- master- new-master Looks good to me. Doing this isn't a problem, though I'm not sure if it buys us much. 90% of the changes are the refactoring, moving substantial amounts of code from numpy/core/src/multiarray and /umath into libndarray and then all of the assorted fix-ups. The rest is the .NET interface layer which is isolated in numpy/NumpyDotNet for now. We can leave this directory out, but everything else is the same between libndarray and refactor. Or am I misunderstanding the reason? The current state of the refactor branch is that it passes the bulk of regressions on Python 2.6 and 3.? (Ilan, what version did you use?) and is up-to-date with the master branch. There are a few failing regression test that we need to look at vs. the master branch but less than dozen. Switching to use libndarray is a big ABI+API change, right? If there's an idea to release an ABI-compatible 1.6, wouldn't this end up being more difficult? Maybe I'm misunderstanding this idea. Definitely a big ABI change and effectively a big API change. The API itself should be close to 100% compatible, except that the data structures all change to introduce a new layer of indirection. Code that strictly uses the macro accessors will build fine, but that is turning out to be quite rare. The changes are quite mechanical but still non-trivial for code that directly accesses the structure fields. Changes to Cython as a part of the project take care of some of the work. A new numpy.pdx file is needed and will mask the changes as long as the Python (as opposed to the CPython) interface is used. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Refactor fork uses the ./configure, make, make install process.
On Tue, Dec 7, 2010 at 11:34 AM, Jason McCampbell jmccampb...@enthought.com wrote: Sorry for the late reply... I missed this thread. Thanks to Ilan for pointing it out. A variety of comments below... On Sat, Dec 4, 2010 at 10:20 AM, Charles R Harris charlesr.har...@gmail.com wrote: Just wondering if this is temporary or the intention is to change the build process? I also note that the *.h files in libndarray are not complete and a *lot* of trailing whitespace has crept into the files. For the purposes of our immediate project the intent is to use autoconf since it's widely available and makes building this part Python-independent and easier than working it into both distutils and numscons. Going forward it's certainly open to discussion. Yes, maintaining multiple build systems is a hassle. I'm wondering if we shouldn't remove the scons stuff and stick with distutils until we definitely decide there is a better way. As to autotools, I think it is a fine short term solution for development purposes, but probably needs to be replaced down the road. Currently all of the .h and .c files are generated as a part of the build rather than being checked in just because it saves a build step. Checking in the intermediate files isn't a problem either. The idea of having separate .h files is that you can test compile without a complete build. They might also be helpful in the separate compilation case (I haven't checked). But in any case, the *.h.src files are there just to make maintaining the .h file easier, they shouldn't be used as part of the build. Does the trailing whitespace cause problems? We saw it in the coding guidelines and planned to run a filter over it once the code stabilizes, but none of us had seen a guideline like that before and weren't sure why it was there. It should be cleaned up before anything becomes official. Git can be set up to warn about trailing whitespace. The general guideline is no trailing whitespace. For one thing you end up with repository changes that unintentionally involve whitespace. Most editors can be set up to flag trailing whitespace, which will increase the desire to keep the file clean. On Sat, Dec 4, 2010 at 3:01 PM, Charles R Harris charlesr.har...@gmail.comwrote: On Sat, Dec 4, 2010 at 1:45 PM, Pauli Virtanen p...@iki.fi wrote: On Sat, 04 Dec 2010 14:24:49 -0600, Ilan Schnell wrote: I'm not sure how reasonable it would be to move only libndarray into the master, because I've been working on EPD for the last couple of week. But Jason will know how complete libndarray is. The main question is whether moving it will make things easier or more difficult, I think. It's one tree more to keep track of. In any case, it would be a first part in the merge, and it would split the hunk of changes into two parts. That would be a good thing IMHO. It would also bring a bit more numpy reality to the refactor and since we are implicitly relying on it for the next release sometime next spring the closer to reality it gets the better. *** Technically, the move could be done like this, so that merge tracking still works: refactor--- new-refactor // /libndarray--x / \ start-- master- new-master Looks good to me. Doing this isn't a problem, though I'm not sure if it buys us much. 90% of the changes are the refactoring, moving substantial amounts of code from numpy/core/src/multiarray and /umath into libndarray and then all of the assorted fix-ups. The rest is the .NET interface layer which is isolated in numpy/NumpyDotNet for now. We can leave this directory out, but everything else is the same between libndarray and refactor. Or am I misunderstanding the reason? The idea is to keep things moving along and maybe encourage others to take a bigger role in the merge. We wouldn't touch the current master branch of numpy yet. The current state of the refactor branch is that it passes the bulk of regressions on Python 2.6 and 3.? (Ilan, what version did you use?) and is up-to-date with the master branch. There are a few failing regression test that we need to look at vs. the master branch but less than dozen. Switching to use libndarray is a big ABI+API change, right? If there's an idea to release an ABI-compatible 1.6, wouldn't this end up being more difficult? Maybe I'm misunderstanding this idea. Definitely a big ABI change and effectively a big API change. The API itself should be close to 100% compatible, except that the data structures all change to introduce a new layer of indirection. Code that strictly uses the macro accessors will build fine, but that is turning out to be quite rare. The changes are quite mechanical but still non-trivial for code that directly accesses the structure fields. Changes to Cython as a
Re: [Numpy-discussion] Refactor fork uses the ./configure, make, make install process.
On Wed, Dec 8, 2010 at 2:57 AM, Charles R Harris charlesr.har...@gmail.comwrote: On Tue, Dec 7, 2010 at 11:34 AM, Jason McCampbell jmccampb...@enthought.com wrote: Sorry for the late reply... I missed this thread. Thanks to Ilan for pointing it out. A variety of comments below... On Sat, Dec 4, 2010 at 10:20 AM, Charles R Harris charlesr.har...@gmail.com wrote: Just wondering if this is temporary or the intention is to change the build process? I also note that the *.h files in libndarray are not complete and a *lot* of trailing whitespace has crept into the files. For the purposes of our immediate project the intent is to use autoconf since it's widely available and makes building this part Python-independent and easier than working it into both distutils and numscons. Going forward it's certainly open to discussion. Yes, maintaining multiple build systems is a hassle. I'm wondering if we shouldn't remove the scons stuff and stick with distutils until we definitely decide there is a better way. Why would you want to remove scons before we settle on a final new way of doing things? It's not that much effort to maintain as far as I'm aware, and more useful (at least to me) than distutils. I don't see a reason not to keep it until we have something that's actually better (hopefully bento). Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Refactor fork uses the ./configure, make, make install process.
On Tue, Dec 7, 2010 at 5:36 PM, Ralf Gommers ralf.gomm...@googlemail.comwrote: On Wed, Dec 8, 2010 at 2:57 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Dec 7, 2010 at 11:34 AM, Jason McCampbell jmccampb...@enthought.com wrote: Sorry for the late reply... I missed this thread. Thanks to Ilan for pointing it out. A variety of comments below... On Sat, Dec 4, 2010 at 10:20 AM, Charles R Harris charlesr.har...@gmail.com wrote: Just wondering if this is temporary or the intention is to change the build process? I also note that the *.h files in libndarray are not complete and a *lot* of trailing whitespace has crept into the files. For the purposes of our immediate project the intent is to use autoconf since it's widely available and makes building this part Python-independent and easier than working it into both distutils and numscons. Going forward it's certainly open to discussion. Yes, maintaining multiple build systems is a hassle. I'm wondering if we shouldn't remove the scons stuff and stick with distutils until we definitely decide there is a better way. Why would you want to remove scons before we settle on a final new way of doing things? It's not that much effort to maintain as far as I'm aware, and more useful (at least to me) than distutils. I don't see a reason not to keep it until we have something that's actually better (hopefully bento). Actually, I was waiting to see if you liked scons ;) I don't use it myself but I was wondering if you used it for the releases. I agree it will be interesting to see what David comes up with, I think at the moment he likes waf as a build system to use with bento. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Refactor fork uses the ./configure, make, make install process.
On 12/05/2010 05:57 AM, Dag Sverre Seljebotn wrote: On 12/04/2010 09:11 PM, Charles R Harris wrote: On Sat, Dec 4, 2010 at 12:59 PM, Ilan Schnell ischn...@enthought.com mailto:ischn...@enthought.com wrote: Yes, numpy-refactor builds of top of libndarray. The whole point was that the libndarray is independent of the interface, i.e. the CPython or the IronPython interface, and possibly other (Jython) in the future. Looking at different building/packaging solutions for libndarray, autoconf make things very easy, it's a well established pattern, I'm sure David C. will agree. I know he has expressed reservations about it on non-posix platforms and some large projects have moved away from it. I'm not saying it isn't the best short term solution so you folks can get on with the job, but it may be that long term we will want to look elsewhere. Such as perhaps waf for building libndarray, which seems like it will be much easier to make work nicely with Bento etc. than autoconf (again, speaking long-term). Also, it'd be good to avoid a seperate build system for Windows (problem of keeping changes sync-ed with Visual Studio projects etc. etc.). Is support for visual studio projects a requirement for the refactoring ? If so, the only alternative to keeping changes in sync is to be able to generate the project files from a description, which is not so easy (and quite time consuming). I know of at least two tools doing that: cmake and gpy (the build system used for chrome). cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Refactor fork uses the ./configure, make, make install process.
On Sat, Dec 4, 2010 at 12:59 PM, Ilan Schnell ischn...@enthought.comwrote: Yes, numpy-refactor builds of top of libndarray. The whole point was that the libndarray is independent of the interface, i.e. the CPython or the IronPython interface, and possibly other (Jython) in the future. Looking at different building/packaging solutions for libndarray, autoconf make things very easy, it's a well established pattern, I'm sure David C. will agree. I know he has expressed reservations about it on non-posix platforms and some large projects have moved away from it. I'm not saying it isn't the best short term solution so you folks can get on with the job, but it may be that long term we will want to look elsewhere. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Refactor fork uses the ./configure, make, make install process.
On Sat, Dec 4, 2010 at 12:52 PM, Pauli Virtanen p...@iki.fi wrote: On Sat, 04 Dec 2010 12:21:15 -0700, Charles R Harris wrote: [clip] So does numpy currently build on top of libndarray or is that something for the future also? [clip] It does. If you look how it works, most of the heavy lifting has been moved there, leaving the multiarray module mostly as Python-specific wrappers. Would it unreasonable to move the libndarray stuff to the current master branch of numpy while leaving the rest of things intact? The needed changes to the current core/src could be brought in later. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Refactor fork uses the ./configure, make, make install process.
I'm not sure how reasonable it would be to move only libndarray into the master, because I've been working on EPD for the last couple of week. But Jason will know how complete libndarray is. - Ilan On Sat, Dec 4, 2010 at 2:19 PM, Charles R Harris charlesr.har...@gmail.com wrote: Would it unreasonable to move the libndarray stuff to the current master branch of numpy while leaving the rest of things intact? The needed changes to the current core/src could be brought in later. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Refactor fork uses the ./configure, make, make install process.
On Sat, 04 Dec 2010 14:24:49 -0600, Ilan Schnell wrote: I'm not sure how reasonable it would be to move only libndarray into the master, because I've been working on EPD for the last couple of week. But Jason will know how complete libndarray is. The main question is whether moving it will make things easier or more difficult, I think. It's one tree more to keep track of. In any case, it would be a first part in the merge, and it would split the hunk of changes into two parts. *** Technically, the move could be done like this, so that merge tracking still works: refactor--- new-refactor // /libndarray--x / \ start-- master- new-master -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Refactor fork uses the ./configure, make, make install process.
On 12/04/2010 09:11 PM, Charles R Harris wrote: On Sat, Dec 4, 2010 at 12:59 PM, Ilan Schnell ischn...@enthought.com mailto:ischn...@enthought.com wrote: Yes, numpy-refactor builds of top of libndarray. The whole point was that the libndarray is independent of the interface, i.e. the CPython or the IronPython interface, and possibly other (Jython) in the future. Looking at different building/packaging solutions for libndarray, autoconf make things very easy, it's a well established pattern, I'm sure David C. will agree. I know he has expressed reservations about it on non-posix platforms and some large projects have moved away from it. I'm not saying it isn't the best short term solution so you folks can get on with the job, but it may be that long term we will want to look elsewhere. Such as perhaps waf for building libndarray, which seems like it will be much easier to make work nicely with Bento etc. than autoconf (again, speaking long-term). Also, it'd be good to avoid a seperate build system for Windows (problem of keeping changes sync-ed with Visual Studio projects etc. etc.). Dag Sverre ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Refactor fork uses the ./configure, make, make install process.
On Sat, Dec 4, 2010 at 1:45 PM, Pauli Virtanen p...@iki.fi wrote: On Sat, 04 Dec 2010 14:24:49 -0600, Ilan Schnell wrote: I'm not sure how reasonable it would be to move only libndarray into the master, because I've been working on EPD for the last couple of week. But Jason will know how complete libndarray is. The main question is whether moving it will make things easier or more difficult, I think. It's one tree more to keep track of. In any case, it would be a first part in the merge, and it would split the hunk of changes into two parts. That would be a good thing IMHO. It would also bring a bit more numpy reality to the refactor and since we are implicitly relying on it for the next release sometime next spring the closer to reality it gets the better. *** Technically, the move could be done like this, so that merge tracking still works: refactor--- new-refactor // /libndarray--x / \ start-- master- new-master Looks good to me. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Refactor fork uses the ./configure, make, make install process.
On Sat, Dec 4, 2010 at 12:45 PM, Pauli Virtanen p...@iki.fi wrote: Technically, the move could be done like this, so that merge tracking still works: refactor--- new-refactor // /libndarray--x / \ start-- master- new-master Switching to use libndarray is a big ABI+API change, right? If there's an idea to release an ABI-compatible 1.6, wouldn't this end up being more difficult? Maybe I'm misunderstanding this idea. I looked a little bit at the 1.4.0 ABI issue, and if the only blocking problem was the cast[] array in ArrFuncs, I think that can be worked around without too much difficulty. Would people want an ABI-compatible 1.6 release adding date-time and float16? -Mark ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
