math/atlas-devel build times (was Re: math/py-numpy vs. math/atlas-devel)
On Tue, 10 Nov 2009 15:41:05 +0900 (JST) Maho NAKATA wrote: >I'm willing to apply patches to atlas ports if available. Thanks, Maho. It appears that the most critical change is to make sure the tools can know that math/atlas and math/atlas-devel conflict with each other, so that if one is already installed, the tools won't try to install the other. >I talked with Goto Kazushige (author of GotoBLAS), he told me >that L2 cache handling on FreeBSD is very bad. It may be a reason I would be very interested in knowing in detail why he thinks that. The ATLAS build procedure, however, does use a very crude algorithm for detecting the sizes of L1 and L2 caches that "detects" cache sizes much smaller than they actually are. I think there are some ways of correcting the "detected" sizes, but they involve editing source files and making decisions that go way beyond anything that the FreeBSD ports system is equipped to handle. Assuming cache sizes that are typically 1/2 or 1/4 of the actual sizes will quite naturally give less than optimal results. It is worth noting, though, that this same algorithm for cache size detection is used for building ATLAS on any operating system on i386 or amd64 architectures. >why ATLAS build is so fragile. On Linux machines, it takes only 20 min or so. > As I wrote before, the sensitivity is based upon the variance of a run times. The sample size is quite small, so any sizable differences in even one or two run times can put the variance over the build procedure's tolerance threshhold. On my machine, I always had to edit a source file to increase the number of iterations of each test from 10 to either 100 or 1000 in order to get the sensitivity down to where occasional other activity in the system wouldn't kill the ATLAS build. However, math/atlas-devel in FreeBSD 7.2 surprised me by building cleanly with no such intervention required, which pleased me greatly, of course. I don't know what the authors changed to make this happen. In any case, the variance sensitivity problem ought to be the same, regardless of operating system. The ATLAS build procedure builds an unthreaded version of the library on any system. If more than one logical/physical CPU is detected, it also builds a threaded version. You didn't mention the system configuration that can allegedly build ATLAS in only 20 minutes. I'm building it on a Dell Inspiron XPS, which has a 3.4 GHz P4 Prescott CPU. The L1 caches are 64 KB each, IIRC, and the L2 cache is 1 MB, but ATLAS's build procedure typically decides that the L1 size is 16 KB or less and the L2 size is 256 KB. I have hyperthreading enabled, so the build procedure sees two (logical) CPUs, but the gains from the hyperthreading are very small in this situation. Building ATLAS on a truly dual-cored system ought to give much shorter build times, and building it on a Core i7 system ought to be tremendously faster with or without enabling the i7's reputedly better implemented HT. The P4 Prescott is now a very old chip model. (I bought this machine new five years ago.) The last time I checked, the LINUX kernel still didn't know the difference between logical CPUs and physical CPUs, so it's difficult to see how its cache management in a HT environment could be any better than FreeBSD's. Maybe LINUX has been updated to understand five-year-old technology since I last checked, but that still should only make it closer to FreeBSD's performance, not radically faster than FreeBSD. Scott Bennett, Comm. ASMELG, CFIAG ** * Internet: bennett at cs.niu.edu * ** * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * *-- Gov. John Hancock, New York Journal, 28 January 1790 * ** ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: math/py-numpy vs. math/atlas-devel
Hi all,
I'm willing to apply patches to atlas ports if available.
I talked with Goto Kazushige (author of GotoBLAS), he told me
that L2 cache handling on FreeBSD is very bad. It may be a reason
why ATLAS build is so fragile. On Linux machines, it takes only 20 min or so.
Thanks for good discussions!
Best,
Nakata Maho
From: Scott Bennett
Subject: Re: math/py-numpy vs. math/atlas-devel
Date: Mon, 09 Nov 2009 15:07:55 -0600 (CST)
> On Mon, 9 Nov 2009 20:26:15 + "b. f."
> wrote:
>>On 11/9/09, Scott Bennett wrote:
>>> On Sun, 08 Nov 2009 23:59:29 -0800 Doug Barton
>>> wrote:
>>
>>...
>>
>>> Anyway, the math/py-numpy port now proceeds to build without bothering
>>> with math/atlas. It quickly goes astray when it doesn't recognize that any
>>> of gfortran4[345] has been installed--I guess there no longer is a FORTRAN
>>> compiler included in the base system--and instead tries to use lang/g95,
>>> which has also been installed. Of course, this won't work because the ATLAS
>>> library needs to have been compiled with the same compiler as the programs
>>> that use it.
>>>
>>> ===> Configuring for py26-numpy-1.3.0_2,1
>>> Running from numpy source directory.
>>> [39mF2PY Version 2 [0m
>>> [39mblas_opt_info: [0m
>>> [39mblas_mkl_info: [0m
>>> [39m libraries mkl,vml,guide not found in /usr/lib [0m
>>> [39m libraries mkl,vml,guide not found in /usr/local/lib [0m
>>> [39m libraries mkl,vml,guide not found in
>>> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/../../../ [0m
>>> [39m NOT AVAILABLE [0m
>>> [39m [0m
>>> [39matlas_blas_threads_info: [0m
>>> [39mSetting PTATLAS=ATLAS [0m
>>> [39mSetting PTATLAS=ATLAS [0m
>>> [39mSetting PTATLAS=ATLAS [0m
>>> [39m FOUND: [0m
>>> [39mlibraries = ['alapack_r', 'f77blas_r', 'cblas_r', 'atlas_r'] [0m
>>> [39mlibrary_dirs = ['/usr/local/lib'] [0m
>>> [39mlanguage = c [0m
>>> [39minclude_dirs = ['/usr/local/include'] [0m
>>> [39m [0m
>>> /usr/ports/math/py-numpy/work/numpy-1.3.0/numpy/distutils/command/config.py:361:
>>> DeprecationWarning:
>>> +
>>> Usage of get_output is deprecated: please do not
>>> use it anymore, and avoid configuration checks
>>> involving running executable on the target machine.
>>> +
>>>
>>> DeprecationWarning)
>>> [39mcustomize GnuFCompiler [0m
>>> [32mFound executable /usr/local/bin/gfortran44 [0m
>>> [31mgnu: no Fortran 90 compiler found [0m
>>> [31mgnu: no Fortran 90 compiler found [0m
>>> [39mcustomize Gnu95FCompiler [0m
>>> [39mcustomize Gnu95FCompiler [0m
>>> [39mcustomize Gnu95FCompiler using config [0m
>>> compiling '_configtest.c':
>>>
>>> /* This file is generated from numpy/distutils/system_info.py */
>>> void ATL_buildinfo(void);
>>> int main(void) {
>>> ATL_buildinfo();
>>> return 0;
>>> }
>>> [39mC compiler: gcc44 -DNDEBUG -O2 -fno-strict-aliasing -pipe
>>> -march=prescott -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x10 -O2
>>> -fno-strict-aliasing -pipe -march=prescott -Wl,-rpath=/usr/local/lib/gcc44
>>> -fPIC
>>> [0m
>>> [39mcompile options: '-c' [0m
>>> [39mgcc44: _configtest.c [0m
>>> [39mgcc44 _configtest.o -L/usr/local/lib -lalapack_r -lf77blas_r -lcblas_r
>>> -latlas_r -o _configtest [0m
>>> /usr/bin/ld: _configtest: hidden symbol `__powisf2' in
>>> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/libgcc.a(_powisf2.o)
>>> is referenced by DSO
>>> collect2: ld returned 1 exit status
>>> /usr/bin/ld: _configtest: hidden symbol `__powisf2' in
>>> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/libgcc.a(_powisf2.o)
>>> is referenced by DSO
>>> collect2: ld returned 1 exit status
>>> [39mfailure. [0m
>>> [39mremoving: _configtest.c _configtest.o [0m
>>> [39mStatus: 255 [0m
>>> [39mOutput: [0m
>>> [39m FOUND: [0m
>>> [39mlibraries = ['alapack_r', 'f77blas_r', 'cblas_r', 'atlas_r'] [0m
>>> [39mlibrary_dirs = ['/usr/local/lib'] [0m
>>> [39mlanguage = c [0m
>>> [39mdefine_macros = [('NO_ATLAS_INFO&#
Re: math/py-numpy vs. math/atlas-devel
On Mon, 9 Nov 2009 20:26:15 + "b. f."
wrote:
>On 11/9/09, Scott Bennett wrote:
>> On Sun, 08 Nov 2009 23:59:29 -0800 Doug Barton
>> wrote:
>
>...
>
>> Anyway, the math/py-numpy port now proceeds to build without bothering
>> with math/atlas. It quickly goes astray when it doesn't recognize that any
>> of gfortran4[345] has been installed--I guess there no longer is a FORTRAN
>> compiler included in the base system--and instead tries to use lang/g95,
>> which has also been installed. Of course, this won't work because the ATLAS
>> library needs to have been compiled with the same compiler as the programs
>> that use it.
>>
>> ===> Configuring for py26-numpy-1.3.0_2,1
>> Running from numpy source directory.
>> [39mF2PY Version 2 [0m
>> [39mblas_opt_info: [0m
>> [39mblas_mkl_info: [0m
>> [39m libraries mkl,vml,guide not found in /usr/lib [0m
>> [39m libraries mkl,vml,guide not found in /usr/local/lib [0m
>> [39m libraries mkl,vml,guide not found in
>> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/../../../ [0m
>> [39m NOT AVAILABLE [0m
>> [39m [0m
>> [39matlas_blas_threads_info: [0m
>> [39mSetting PTATLAS=ATLAS [0m
>> [39mSetting PTATLAS=ATLAS [0m
>> [39mSetting PTATLAS=ATLAS [0m
>> [39m FOUND: [0m
>> [39mlibraries = ['alapack_r', 'f77blas_r', 'cblas_r', 'atlas_r'] [0m
>> [39mlibrary_dirs = ['/usr/local/lib'] [0m
>> [39mlanguage = c [0m
>> [39minclude_dirs = ['/usr/local/include'] [0m
>> [39m [0m
>> /usr/ports/math/py-numpy/work/numpy-1.3.0/numpy/distutils/command/config.py:361:
>> DeprecationWarning:
>> +
>> Usage of get_output is deprecated: please do not
>> use it anymore, and avoid configuration checks
>> involving running executable on the target machine.
>> +
>>
>> DeprecationWarning)
>> [39mcustomize GnuFCompiler [0m
>> [32mFound executable /usr/local/bin/gfortran44 [0m
>> [31mgnu: no Fortran 90 compiler found [0m
>> [31mgnu: no Fortran 90 compiler found [0m
>> [39mcustomize Gnu95FCompiler [0m
>> [39mcustomize Gnu95FCompiler [0m
>> [39mcustomize Gnu95FCompiler using config [0m
>> compiling '_configtest.c':
>>
>> /* This file is generated from numpy/distutils/system_info.py */
>> void ATL_buildinfo(void);
>> int main(void) {
>> ATL_buildinfo();
>> return 0;
>> }
>> [39mC compiler: gcc44 -DNDEBUG -O2 -fno-strict-aliasing -pipe
>> -march=prescott -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x10 -O2
>> -fno-strict-aliasing -pipe -march=prescott -Wl,-rpath=/usr/local/lib/gcc44
>> -fPIC
>> [0m
>> [39mcompile options: '-c' [0m
>> [39mgcc44: _configtest.c [0m
>> [39mgcc44 _configtest.o -L/usr/local/lib -lalapack_r -lf77blas_r -lcblas_r
>> -latlas_r -o _configtest [0m
>> /usr/bin/ld: _configtest: hidden symbol `__powisf2' in
>> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/libgcc.a(_powisf2.o)
>> is referenced by DSO
>> collect2: ld returned 1 exit status
>> /usr/bin/ld: _configtest: hidden symbol `__powisf2' in
>> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/libgcc.a(_powisf2.o)
>> is referenced by DSO
>> collect2: ld returned 1 exit status
>> [39mfailure. [0m
>> [39mremoving: _configtest.c _configtest.o [0m
>> [39mStatus: 255 [0m
>> [39mOutput: [0m
>> [39m FOUND: [0m
>> [39mlibraries = ['alapack_r', 'f77blas_r', 'cblas_r', 'atlas_r'] [0m
>> [39mlibrary_dirs = ['/usr/local/lib'] [0m
>> [39mlanguage = c [0m
>> [39mdefine_macros = [('NO_ATLAS_INFO', 2)] [0m
>> [39minclude_dirs = ['/usr/local/include'] [0m
>> [39m [0m
>> [39mlapack_opt_info: [0m
>> [39mlapack_mkl_info: [0m
>> [39mmkl_info: [0m
>> [39m libraries mkl,vml,guide not found in /usr/lib [0m
>> [39m libraries mkl,vml,guide not found in /usr/local/lib [0m
>> [39m libraries mkl,vml,guide not found in
>> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/../../../ [0m
>> [39m NOT AVAILABLE [0m
>> [39m [0m
>> [39m NOT AVAILABLE [0m
>> [39m [0m
>> [39matlas_threads_info: [0m
>> [39mSetting PTATLAS=ATLAS [0m
>> [39m libraries lapack_atlas not found in /usr/local/lib [0m
>> [39mnumpy.distutils.system_info.atlas_threads_info [0m
>> [39mSetting PTATLAS=ATLAS [0m
>> /usr/ports/math/py-numpy/work/numpy-1.3.0/numpy/distutils/system_info.py:999:
>> UserWarning:
>> *
>> Lapack library (from ATLAS) is probably incomplete:
>> size of /usr/local/lib/libalapack_r.so is 3832k (expected >4000k)
>>
>> Follow the instructions in the KNOWN PROBLEMS section of the file
>> numpy/INSTALL.txt.
>> *
>>
>> warnings.warn(message)
>>
>>
>> The above sequence gets more or less repeated several more times, and
>> after much compiling, running of python26, and other activities, the
>> process runs aground as follows.
>>
>>
>> [39mcreating build/temp.freebsd-7.2-STABLE-i386-2
Re: math/py-numpy vs. math/atlas-devel
On 11/9/09, Scott Bennett wrote:
> On Sun, 08 Nov 2009 23:59:29 -0800 Doug Barton
> wrote:
...
> Anyway, the math/py-numpy port now proceeds to build without bothering
> with math/atlas. It quickly goes astray when it doesn't recognize that any
> of gfortran4[345] has been installed--I guess there no longer is a FORTRAN
> compiler included in the base system--and instead tries to use lang/g95,
> which has also been installed. Of course, this won't work because the ATLAS
> library needs to have been compiled with the same compiler as the programs
> that use it.
>
> ===> Configuring for py26-numpy-1.3.0_2,1
> Running from numpy source directory.
> [39mF2PY Version 2 [0m
> [39mblas_opt_info: [0m
> [39mblas_mkl_info: [0m
> [39m libraries mkl,vml,guide not found in /usr/lib [0m
> [39m libraries mkl,vml,guide not found in /usr/local/lib [0m
> [39m libraries mkl,vml,guide not found in
> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/../../../ [0m
> [39m NOT AVAILABLE [0m
> [39m [0m
> [39matlas_blas_threads_info: [0m
> [39mSetting PTATLAS=ATLAS [0m
> [39mSetting PTATLAS=ATLAS [0m
> [39mSetting PTATLAS=ATLAS [0m
> [39m FOUND: [0m
> [39mlibraries = ['alapack_r', 'f77blas_r', 'cblas_r', 'atlas_r'] [0m
> [39mlibrary_dirs = ['/usr/local/lib'] [0m
> [39mlanguage = c [0m
> [39minclude_dirs = ['/usr/local/include'] [0m
> [39m [0m
> /usr/ports/math/py-numpy/work/numpy-1.3.0/numpy/distutils/command/config.py:361:
> DeprecationWarning:
> +
> Usage of get_output is deprecated: please do not
> use it anymore, and avoid configuration checks
> involving running executable on the target machine.
> +
>
> DeprecationWarning)
> [39mcustomize GnuFCompiler [0m
> [32mFound executable /usr/local/bin/gfortran44 [0m
> [31mgnu: no Fortran 90 compiler found [0m
> [31mgnu: no Fortran 90 compiler found [0m
> [39mcustomize Gnu95FCompiler [0m
> [39mcustomize Gnu95FCompiler [0m
> [39mcustomize Gnu95FCompiler using config [0m
> compiling '_configtest.c':
>
> /* This file is generated from numpy/distutils/system_info.py */
> void ATL_buildinfo(void);
> int main(void) {
> ATL_buildinfo();
> return 0;
> }
> [39mC compiler: gcc44 -DNDEBUG -O2 -fno-strict-aliasing -pipe
> -march=prescott -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x10 -O2
> -fno-strict-aliasing -pipe -march=prescott -Wl,-rpath=/usr/local/lib/gcc44
> -fPIC
> [0m
> [39mcompile options: '-c' [0m
> [39mgcc44: _configtest.c [0m
> [39mgcc44 _configtest.o -L/usr/local/lib -lalapack_r -lf77blas_r -lcblas_r
> -latlas_r -o _configtest [0m
> /usr/bin/ld: _configtest: hidden symbol `__powisf2' in
> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/libgcc.a(_powisf2.o)
> is referenced by DSO
> collect2: ld returned 1 exit status
> /usr/bin/ld: _configtest: hidden symbol `__powisf2' in
> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/libgcc.a(_powisf2.o)
> is referenced by DSO
> collect2: ld returned 1 exit status
> [39mfailure. [0m
> [39mremoving: _configtest.c _configtest.o [0m
> [39mStatus: 255 [0m
> [39mOutput: [0m
> [39m FOUND: [0m
> [39mlibraries = ['alapack_r', 'f77blas_r', 'cblas_r', 'atlas_r'] [0m
> [39mlibrary_dirs = ['/usr/local/lib'] [0m
> [39mlanguage = c [0m
> [39mdefine_macros = [('NO_ATLAS_INFO', 2)] [0m
> [39minclude_dirs = ['/usr/local/include'] [0m
> [39m [0m
> [39mlapack_opt_info: [0m
> [39mlapack_mkl_info: [0m
> [39mmkl_info: [0m
> [39m libraries mkl,vml,guide not found in /usr/lib [0m
> [39m libraries mkl,vml,guide not found in /usr/local/lib [0m
> [39m libraries mkl,vml,guide not found in
> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/../../../ [0m
> [39m NOT AVAILABLE [0m
> [39m [0m
> [39m NOT AVAILABLE [0m
> [39m [0m
> [39matlas_threads_info: [0m
> [39mSetting PTATLAS=ATLAS [0m
> [39m libraries lapack_atlas not found in /usr/local/lib [0m
> [39mnumpy.distutils.system_info.atlas_threads_info [0m
> [39mSetting PTATLAS=ATLAS [0m
> /usr/ports/math/py-numpy/work/numpy-1.3.0/numpy/distutils/system_info.py:999:
> UserWarning:
> *
> Lapack library (from ATLAS) is probably incomplete:
> size of /usr/local/lib/libalapack_r.so is 3832k (expected >4000k)
>
> Follow the instructions in the KNOWN PROBLEMS section of the file
> numpy/INSTALL.txt.
> *
>
> warnings.warn(message)
>
>
> The above sequence gets more or less repeated several more times, and
> after much compiling, running of python26, and other activities, the
> process runs aground as follows.
>
>
> [39mcreating build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/fft [0m
> [39mcompile options: '-Inumpy/core/include
> -Ibuild/src.freebsd-7.2-STABLE-i386-2.6/numpy/core/include/numpy
> -Inumpy/core/src -Inumpy/core/include -I/usr/
Re: math/py-numpy vs. math/atlas-devel
b. f. wrote: > 1) +IGNOREME files only work with installed ports (just because the > package database is naturally associated with installed ports doesn't > mean that someone won't create an +IGNOREME for one that isn't > installed), and > 2) when trying to prevent the build or installation of a port that is > not currently installed, the exclusion glob will be compared with the > port directory, rather than the PKGNAME, as is done for installed > packages. I've updated my working copy of the man page with these suggestions, thanks! > And with regard to the selection of exclusion > globs, we should note that a glob that matches too many ports may be > as problematic as one that matches too few. But, of course. :) Regexp creation is both one of the rites of passage for all system administrators and a never-ending journey. If the -x or other glob option is not suitable there are always other options. Doug -- Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: math/py-numpy vs. math/atlas-devel
On Sun, 08 Nov 2009 23:59:29 -0800 Doug Barton wrote: >First, sorry I missed the original problem report, I'm not subscribed >to -questions. Usually ports questions (including those about tools) >are handled on freebsd-po...@freebsd.org, but OTOH if you're not sure >where to send a question it's always better to start on the -questions >list. :) The responses I got the last time I posted something to -ports convinced me not to post there anymore. I'm still subscribed, however, so I still see what transpires there. > >Second, thanks to b.f. for the very thorough analysis of the problem. Yes, indeed, thanks. It was fascinating. >I will respond and trim a bit as I go. > >Third, I've cc'ed maho@ since the genesis of the problem is that >math/atlas and math/atlas-devel aren't setting CONFLICTS when >apparently they should be. maho, if you need any help with this let me >know. I've left the Cc in as well because IIRC, Maho was heavily involved in the upgrade of immense numbers of ports to gfortran42. The next obstacle in math/py-numpy outlined further below may well be a missed relic of that effort. > >b. f. wrote: >> Scott Bennett wrote: >>> I would like to install science/gnudatalanguage but have been running >>> into various obstacles. Lars Engels very kindly just fixed one of them >>> (devel/lasi) (Thanks, Lars!), so now I'm on to the next one, which may not >>> be a showstopper, but it's at least a nuisance. One of the gnudatalanguage >>> dependencies is math/py-numpy, which, in turn, depends upon math/atlas. >>> However, I have math/atlas-devel installed and do not wish to revert to an >>> older, slower version of the ATLAS library. Adding a "-x atlas-\*" onto >>> the portmaster command to build math/py-numpy fails to prevent portmaster >>>from trying to build math/atlas. > >As was pointed out, that definitely won't work. With the glob patterns >for exclusions, or specifying ports on the command line you want to be >as conservative as possible. Also, it's not necessary to include a * >at the end, portmaster will handle that for you. > >>> Creating a >>> /var/db/pkg/atlas-3.8.3_1,1/+IGNOREME file in addition doesn't help. > >+IGNOREME files are only relevant to installed ports. > >>> How >>> can I force math/py-numpy to accept the already installed math/atlas-devel >>> libraries? >>> Thanks in advance for any help! > >The easiest way for you to accomplish this would have been to use '-x >atlas', the second-easiest would have been to use the -i command line >option. See the man page for more information on that. Yes, thanks much to both of you. Cutting the option back to just "-x atlas" did the trick, allowing math/atlas to be skipped. Here's what it looks like giving the desired action: ===>>> Port directory: /usr/ports/math/py-numpy ===>>> Gathering distinfo list for installed ports ===>>> Launching 'make checksum' for math/py-numpy in background ===>>> Gathering dependency list for math/py-numpy from ports ===>>> Starting recursive 'make config' check ===>>> Checking dependency: devel/py-nose ===>>> Launching child to update devel/py-nose math/py-numpy >> devel/py-nose ===>>> Port directory: /usr/ports/devel/py-nose ===>>> Launching 'make checksum' for devel/py-nose in background ===>>> Gathering dependency list for devel/py-nose from ports ===>>> Starting recursive 'make config' check ===>>> Checking dependency: devel/py-setuptools ===>>> Checking dependency: lang/python26 ===>>> Recursive 'make config' check complete for devel/py-nose math/py-numpy >> devel/py-nose ===>>> Continuing 'make config' dependency check for math/py-numpy ===>>> Checking dependency: lang/gcc44 ===>>> Checking dependency: lang/python26 ===>>> Checking dependency: math/atlas ===>>> Skipping math/atlas because it matches the pattern: *atlas* ===>>> Checking dependency: math/suitesparse ===>>> Recursive 'make config' check complete for math/py-numpy ===>>> Starting build for math/py-numpy <<<=== ===>>> Starting check for build dependencies ===>>> Gathering dependency list for math/py-numpy from ports ===>>> Starting dependency check ===>>> Checking dependency: lang/gcc44 ===>>> Checking dependency: lang/python26 ===>>> Checking dependency: math/atlas ===>>> Skipping math/atlas because it matches the pattern: *atlas* ===>>> Checking dependency: math/suitesparse ===>>> Dependency check complete for math/py-numpy ===> Cleaning for py26-numpy-1.3.0_2,1 ===> Found saved configuration for py26-numpy-1.3.0_2,1 ===> Extracting for py26-numpy-1.3.0_2,1 > >> Congratulations, you've managed to defeat three sets of safeguards in >> portmaster. Sigh. It has *always* been that way for me. By age 15 I was already stumbling over compiler bugs, library bugs, operating system bugs, and even the occasional, very peculiar sort of hardware bug. It's enough to make one believe in gremlins developing interests in areas other than aviation. :-} In the great majori
Re: math/py-numpy vs. math/atlas-devel
On 11/9/09, Doug Barton wrote:
> First, sorry I missed the original problem report, I'm not subscribed
> to -questions. Usually ports questions (including those about tools)
> are handled on freebsd-po...@freebsd.org, but OTOH if you're not sure
> where to send a question it's always better to start on the -questions
> list. :)
>
> Second, thanks to b.f. for the very thorough analysis of the problem.
> I will respond and trim a bit as I go.
>
> Third, I've cc'ed maho@ since the genesis of the problem is that
> math/atlas and math/atlas-devel aren't setting CONFLICTS when
> apparently they should be. maho, if you need any help with this let me
> know.
>
> b. f. wrote:
>> Scott Bennett wrote:
>>> I would like to install science/gnudatalanguage but have been running
>>> into various obstacles. Lars Engels very kindly just fixed one of them
>>> (devel/lasi) (Thanks, Lars!), so now I'm on to the next one, which may
>>> not
>>> be a showstopper, but it's at least a nuisance. One of the
>>> gnudatalanguage
>>> dependencies is math/py-numpy, which, in turn, depends upon math/atlas.
>>> However, I have math/atlas-devel installed and do not wish to revert to
>>> an
>>> older, slower version of the ATLAS library. Adding a "-x atlas-\*" onto
>>> the portmaster command to build math/py-numpy fails to prevent portmaster
>>>from trying to build math/atlas.
>
> As was pointed out, that definitely won't work. With the glob patterns
> for exclusions, or specifying ports on the command line you want to be
> as conservative as possible. Also, it's not necessary to include a *
> at the end, portmaster will handle that for you.
>
>>> Creating a
>>> /var/db/pkg/atlas-3.8.3_1,1/+IGNOREME file in addition doesn't help.
>
> +IGNOREME files are only relevant to installed ports.
>
>>> How
>>> can I force math/py-numpy to accept the already installed
>>> math/atlas-devel
>>> libraries?
>>> Thanks in advance for any help!
>
> The easiest way for you to accomplish this would have been to use '-x
> atlas', the second-easiest would have been to use the -i command line
> option. See the man page for more information on that.
>
>> Congratulations, you've managed to defeat three sets of safeguards in
>> portmaster.
>>
>> Problem #1
>>
>> math/atlas and math/atlas-devel don't have proper CONFLICTS entries in
>> their Makefiles (this should be fixed),
>
> Yes, this analysis was essentially correct.
>
>> Problem #2:
>>
>> You're using the -x flag incorrectly. Or portmaster isn't
>> implementing it properly when the port to be excluded is not actually
>> installed, take your pick.
>
> The way that -x is implemented currently really depends on the user's
> definition of the exclude pattern, and is designed to exclude things
> as early as possible so as to avoid what would ultimately become
> wasted effort. It's hard to justify modifying the code to guess that
> something we've already started working on might match what we think
> the user MEANT instead of what they SAID.
>
>> Problem #3:
>>
>> The +IGNOREME checks in portmaster aren't properly implemented for
>> this case, so they fail to prevent math/atlas from being built.
>
> s/properly implemented for/designed to handle/
>
> If you think about this for a second, the entries in /var/db/pkg are
> exclusively related to installed ports, so trying to use an +IGNOREME
> file for this purpose doesn't make sense, although I can sympathize
> with the OPs sense of desperation here. :)
>
>> So what can you do while these problems are being fixed?
>
> Just to be clear, I didn't see anything that needs to be fixed in your
> message, and I hope my explanation makes it clear why. If you think
> that there are actual bugs that need to be fixed please feel free to
> create a thread on -ports to discuss them.
>
The CONFLICTS for math/atlas* should be fixed, as you remarked. As
for portmaster, I'll allow that your choices were reasonable, but it
would be wise to note in the manpage that:
1) +IGNOREME files only work with installed ports (just because the
package database is naturally associated with installed ports doesn't
mean that someone won't create an +IGNOREME for one that isn't
installed), and
2) when trying to prevent the build or installation of a port that is
not currently installed, the exclusion glob will be compared with the
port directory, rather than the PKGNAME, as is done for installed
packages. Using something like:
#!/bin/sh
for _dir in `make -C /usr/ports -V SUBDIR` ; do
for _dir2 in `make -C /usr/ports/${_dir} -V SUBDIR` ; do
_pkgname=`make -C /usr/ports/${_dir}/${_dir2} -V PKGNAME`
case "${_dir}/${_dir2}" in
*${_pkgname%%-[0-9]*}*) ;;
*) echo "${_dir}/${_dir2} ${_pkgname}" ;;
esac ;
done ;
done
one can find a number of ports that might surprise a user that was not
aware of this fact. And with regard to the selection of exclusion
globs, we should note that a glob that matches too many ports may be
as pr
Re: math/py-numpy vs. math/atlas-devel
First, sorry I missed the original problem report, I'm not subscribed to -questions. Usually ports questions (including those about tools) are handled on freebsd-po...@freebsd.org, but OTOH if you're not sure where to send a question it's always better to start on the -questions list. :) Second, thanks to b.f. for the very thorough analysis of the problem. I will respond and trim a bit as I go. Third, I've cc'ed maho@ since the genesis of the problem is that math/atlas and math/atlas-devel aren't setting CONFLICTS when apparently they should be. maho, if you need any help with this let me know. b. f. wrote: > Scott Bennett wrote: >> I would like to install science/gnudatalanguage but have been running >> into various obstacles. Lars Engels very kindly just fixed one of them >> (devel/lasi) (Thanks, Lars!), so now I'm on to the next one, which may not >> be a showstopper, but it's at least a nuisance. One of the gnudatalanguage >> dependencies is math/py-numpy, which, in turn, depends upon math/atlas. >> However, I have math/atlas-devel installed and do not wish to revert to an >> older, slower version of the ATLAS library. Adding a "-x atlas-\*" onto >> the portmaster command to build math/py-numpy fails to prevent portmaster >>from trying to build math/atlas. As was pointed out, that definitely won't work. With the glob patterns for exclusions, or specifying ports on the command line you want to be as conservative as possible. Also, it's not necessary to include a * at the end, portmaster will handle that for you. >> Creating a >> /var/db/pkg/atlas-3.8.3_1,1/+IGNOREME file in addition doesn't help. +IGNOREME files are only relevant to installed ports. >> How >> can I force math/py-numpy to accept the already installed math/atlas-devel >> libraries? >> Thanks in advance for any help! The easiest way for you to accomplish this would have been to use '-x atlas', the second-easiest would have been to use the -i command line option. See the man page for more information on that. > Congratulations, you've managed to defeat three sets of safeguards in > portmaster. > > Problem #1 > > math/atlas and math/atlas-devel don't have proper CONFLICTS entries in > their Makefiles (this should be fixed), Yes, this analysis was essentially correct. > Problem #2: > > You're using the -x flag incorrectly. Or portmaster isn't > implementing it properly when the port to be excluded is not actually > installed, take your pick. The way that -x is implemented currently really depends on the user's definition of the exclude pattern, and is designed to exclude things as early as possible so as to avoid what would ultimately become wasted effort. It's hard to justify modifying the code to guess that something we've already started working on might match what we think the user MEANT instead of what they SAID. > Problem #3: > > The +IGNOREME checks in portmaster aren't properly implemented for > this case, so they fail to prevent math/atlas from being built. s/properly implemented for/designed to handle/ If you think about this for a second, the entries in /var/db/pkg are exclusively related to installed ports, so trying to use an +IGNOREME file for this purpose doesn't make sense, although I can sympathize with the OPs sense of desperation here. :) > So what can you do while these problems are being fixed? Just to be clear, I didn't see anything that needs to be fixed in your message, and I hope my explanation makes it clear why. If you think that there are actual bugs that need to be fixed please feel free to create a thread on -ports to discuss them. hope this helps, Doug PS, Can't resist ... http://dougbarton.us/portmaster-proposal.html -- Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: math/py-numpy vs. math/atlas-devel
Scott Bennett wrote:
>I would like to install science/gnudatalanguage but have been running
>into various obstacles. Lars Engels very kindly just fixed one of them
>(devel/lasi) (Thanks, Lars!), so now I'm on to the next one, which may not
>be a showstopper, but it's at least a nuisance. One of the gnudatalanguage
>dependencies is math/py-numpy, which, in turn, depends upon math/atlas.
>However, I have math/atlas-devel installed and do not wish to revert to an
>older, slower version of the ATLAS library. Adding a "-x atlas-\*" onto
>the portmaster command to build math/py-numpy fails to prevent portmaster
>from trying to build math/atlas. Creating a
>/var/db/pkg/atlas-3.8.3_1,1/+IGNOREME file in addition doesn't help. How
>can I force math/py-numpy to accept the already installed math/atlas-devel
>libraries?
>Thanks in advance for any help!
Congratulations, you've managed to defeat three sets of safeguards in
portmaster.
Problem #1
math/atlas and math/atlas-devel don't have proper CONFLICTS entries in
their Makefiles (this should be fixed), so the following code in
portmaster, from dependency_check ():
1612 conflicts=''
1613 if pm_cd $d_port; then
1614 grep -ql ^CONFLICTS Makefile &&
1615 conflicts=`pm_make_b -V CONFLICTS`
1616 else
1617 fail "Cannot cd to $d_port"
1618 fi
1619 for glob in $conflicts; do
1620 confl_p=`pkg_info -I $glob 2>/dev/null`
1621 if [ -n "$confl_p" ]; then
1622 confl_p=${confl_p%% *}
1623 echo ''
1624 echo "===>>> The dependency
for ${origin}"
1625 echo " seems to be
handled by $confl_p"
1626 echo ''
1627 d_port="$pd/`origin_from_pdb $confl_p`"
1628 fi
1629 done
doesn't substitute your alternative dependency, math/atlas-devel, for
math/atlas.
Problem #2:
You're using the -x flag incorrectly. Or portmaster isn't
implementing it properly when the port to be excluded is not actually
installed, take your pick. It's not completely fool-proof. Since you
don't have math/atlas installed, the code in portmaster's
dependency_check ():
1632 origin="${d_port#$pd/}" ;
iport=`iport_from_origin ${origin}`
1633
1634 if [ -n "$iport" ]; then
1635 check_exclude $iport || continue
1636 else
1637 check_exclude $origin || continue
1638 fi
gives origin="math/atlas", and then uses:
385 iport_from_origin () {
386 local dir
387 dir=`grep -l "@comment ORIGIN:${1}$" $pdb/*/+CONTENTS`
388
389 # It should not happen that more than one port meets this
390 # requirement, but it can if the pkg data is corrupted.
391 dir="${dir%%/+CONTENTS*}"
392 echo ${dir#$pdb/}
393 }
to find that iport="" , so "check_exclude math/atlas" is called. Then,
in accordance with:
1483 check_exclude () {
1484 [ -n "$PM_EXCL" ] || return 0
1485
1486 local pat
1487
1488 for pat in $PM_EXCL; do
1489 case "$1" in
1490 *${pat}*)
1491 if [ -n "$PM_VERBOSE" ]; then
1492 echo "===>>> Skipping $1"
1493 echo " because it matches the
pattern: *${pat}*"
1494 echo ''
1495 fi
1496 return 1 ;;
1497 esac
1498 done
1499
1500 return 0
1501 }
check_exclude tries to match your patterns in PM_EXCL against
"math/atlas". But you've got "atlas-" in PM_EXCL (after portmaster's
globstrip removed the trailing asterisk, negating your attempt protect
it with quotes), which won't match because of the hyphen. It would
have matched if math/atlas were actually installed, in which case
iport="atlas-3.8.3_1,1" and portmaster would have compared iport with
*atlas-*. You must use a port origin glob with the -x flag when the
port that you wish to exclude isn't installed.
Problem #3:
The +IGNOREME checks in portmaster aren't properly implemented for
this case, so they fail to prevent math/atlas from being built. To
see this, Continue along in dependency_check (). Here, since iport="",
dependency_check () next calls "check_interactive math/atlas":
1459 check_interactive () {
1460 [ -n "$INTERACTIVE_UPDATE" ] || return 0
1461
1462 local update_to
1463
1464 [ -n "$2" ] && update_to=" to $2"
1465
1466 case "$INTERACTIVE_
math/py-numpy vs. math/atlas-devel
I would like to install science/gnudatalanguage but have been running into various obstacles. Lars Engels very kindly just fixed one of them (devel/lasi) (Thanks, Lars!), so now I'm on to the next one, which may not be a showstopper, but it's at least a nuisance. One of the gnudatalanguage dependencies is math/py-numpy, which, in turn, depends upon math/atlas. However, I have math/atlas-devel installed and do not wish to revert to an older, slower version of the ATLAS library. Adding a "-x atlas-\*" onto the portmaster command to build math/py-numpy fails to prevent portmaster from trying to build math/atlas. Creating a /var/db/pkg/atlas-3.8.3_1,1/+IGNOREME file in addition doesn't help. How can I force math/py-numpy to accept the already installed math/atlas-devel libraries? Thanks in advance for any help! Scott Bennett, Comm. ASMELG, CFIAG ** * Internet: bennett at cs.niu.edu * ** * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * *-- Gov. John Hancock, New York Journal, 28 January 1790 * ** ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
