Hi Edward,
I just tested the Gentoo machines again using relax-1.3.4 and minf-1.0.2.
Three 32 bit machines I tested completed the test-suite without any
error. Two other machines I previously had in my possession are no
longer available...
However, one 64 bit machine failed for one test, always with the same
values:
====================
FAIL: Constrained Newton opt, GMW Hessian mod, More and Thuente line
search {S2=0.970, te=2048, Rex=0.149}
...
relax> minimise(*args=('newton',), func_tol=1e-25,
max_iterations=10000000, constraints=True, scaling=True, verbosity=1)
Simulation 1
Simulation 2
Simulation 3
relax> monte_carlo.error_analysis(prune=0.0)
Traceback (most recent call last):
File "/home/semor/relax-1.3.4/test_suite/system_tests/model_free.py",
line 610, in test_opt_constr_newton_gmw_mt_S2_0_970_te_2048_Rex_0_149
self.value_test(spin, select, s2, te, rex, chi2, iter, f_count,
g_count, h_count, warning)
File "/home/semor/relax-1.3.4/test_suite/system_tests/model_free.py",
line 1110, in value_test
self.assertEqual(spin.f_count, f_count, msg=mesg)
AssertionError: Optimisation failure.
System: Linux
Release: 2.6.20-gentoo-r7
Version: #1 SMP Sat Apr 28 23:31:52 Local time zone must be set--see zic
Win32 version:
Distribution: gentoo 1.12.13
Architecture: 64bit ELF
Machine: x86_64
Processor: Intel(R) Xeon(R) CPU 5160 @ 3.00GHz
Python version: 2.6.4
numpy version: 1.3.0
s2: 0.9699999999999994
te: 2048.0000000000446
rex: 0.14900000000001615
chi2: 8.3312601381368332e-28
iter: 22
f_count: 91
g_count: 91
h_count: 22
warning: None
====================
Regards,
Séb :)
On 10-02-21 9:00 AM, Edward d'Auvergne wrote:
> Is it different for the different machines, or is it different each
> time on the same machine? If you give a range of numbers for the
> optimisation results, these tests could be relaxed a little.
>
> Cheers,
>
> Edward
>
>
> On 21 February 2010 14:41, Sébastien Morin<[email protected]>
> wrote:
>
>> Hi Ed,
>>
>> I agree with you that this is not an important issue given the small
>> variations observed...
>>
>> I was just still a bit annoyed by this happening on our Gentoo systems...
>>
>> But maybe this is just because of Gentoo itself, as in Gentoo almost
>> everything is compiled locally, so every system is different because of all
>> the variables that can be changed that affect compilation...
>>
>> Ok, let's forget all this !
>>
>> Regards,
>>
>>
>> Séb
>>
>>
>> On 10-02-21 8:32 AM, Edward d'Auvergne wrote:
>>
>>> Hi,
>>>
>>> The code is not parallelised as most optimisation algorithms are not
>>> amenable to parallelisation. There's a lot of research in that field,
>>> but the code here is not along these lines. Do you still see this
>>> problem? Maybe it is a bug in this specific version of the GCC
>>> compiler which created the python executable? Does it occur on
>>> machines with a different Gentoo versions installed? Can you
>>> reproduce the error in a virtual machine? This is a fixed code path
>>> and cannot in any way be different upon different runs of the test
>>> suite. It doesn't change on all the Mandriva installs I have, all the
>>> Macs it has been tested on, or even on the Windows virtual image I use
>>> to build and test relax on Windows. I've even tested it on Solaris
>>> without problems! In any case, this bug is definitely machine
>>> specific and not related to relax itself. Sorry, I don't know what
>>> else I can do to try to track this down. Maybe your CPUs are doing
>>> some strange frequency scaling depending on load, and that is causing
>>> this bizarre behaviour? In any case, this is not an issue for relax
>>> execution and only affects the precision of optimisation in a small
>>> way.
>>>
>>> Regards,
>>>
>>> Edward
>>>
>>>
>>>
>>> On 21 February 2010 05:34, Sébastien Morin<[email protected]>
>>> wrote:
>>>
>>>
>>>> Hi Ed,
>>>>
>>>> This has been a long time since we discussed about this...
>>>>
>>>> However, talking with Olivier last week, we discussed about one
>>>> possibility
>>>> to explain this issue. Is the code in question in some way parallelized,
>>>> i.e. are there multiple processes running at the same time with their
>>>> results being combined subsequently ? If yes, there could be conditions
>>>> in
>>>> which the problem could arise either because of variations in allocated
>>>> memory or cpu that would change the timing between the different
>>>> processes,
>>>> hence affecting the final result...
>>>>
>>>> Does that make sens ?
>>>>
>>>> Olivier, is this what you explained me last week ?
>>>>
>>>>
>>>> Sébastien
>>>>
>>>>
>>>> On 09-09-14 3:30 AM, Edward d'Auvergne wrote:
>>>>
>>>>
>>>>> Hi,
>>>>>
>>>>> I've been trying to work out what is happening, but it is a complete
>>>>> mystery to me. The algorithms are fixed in stone - I coded them
>>>>> myself and you can see it in the minfx code. They are standard
>>>>> optimisation algorithms that obey fixed rules. On the same machine it
>>>>> must, without question, give the same result every time! If it
>>>>> doesn't, something is wrong with the machine, either hardward or
>>>>> software. Would it be possible to install an earlier python and numpy
>>>>> version (maybe 2.5 and 1.2.1 respectively) to see if that makes a
>>>>> difference? Or maybe it is the Linux kernel doing some strange things
>>>>> with the CPU - maybe switching between power profiles causing the CPU
>>>>> floating point math precision to change? Are you 100% sure that all
>>>>> computers give variable results (between each run), and not that they
>>>>> just give a different fixed result each time? Maybe there is a
>>>>> non-fatal kernel bug not triggered by Oliver's hardward?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Edward
>>>>>
>>>>>
>>>>> P.S. A note to others reading this - this problem is not serious for
>>>>> relax's optimisation!
>>>>>
>>>>>
>>>>> 2009/9/4 Sébastien Morin<[email protected]>:
>>>>>
>>>>>
>>>>>
>>>>>> Hi Ed,
>>>>>>
>>>>>> (I added Olivier Fisette in CC as he is quite computer knowledgeable
>>>>>> and
>>>>>> could help us rationalize this issue...)
>>>>>>
>>>>>> This strange behavior was observed for my laptop and the two other
>>>>>> computers in the lab with the failures in the system tests (i.e. for
>>>>>> the
>>>>>> three computers of the bug report).
>>>>>>
>>>>>> I performed some of the different tests proposed on the following page:
>>>>>> ->
>>>>>> http://www.gentoo.org/doc/en/articles/hardware-stability-p1.xml
>>>>>> (tested CPU with infinite rebuild of kernel using gcc for 4
>>>>>> hours)
>>>>>> (tested CPU with cpuburn-1.4 for XXXX hours)
>>>>>> (tested RAM with memtester-4.0.7 for> 6 hours)
>>>>>> to check the CPU and RAM, but did not find anything... Of course, these
>>>>>> tests may not have uncovered potential problems in my CPU and RAM, but
>>>>>> most chances are they are fine. Moreover, the problem being observed
>>>>>> for
>>>>>> three different computers, it would be surprising that hardware
>>>>>> failures
>>>>>> occur in these three machines...
>>>>>>
>>>>>> The three systems run Gentoo Linux with kernel-2.6.30, numpy-1.3.0 and
>>>>>> python-2.6.2. However, the fourth computer to which I have access (for
>>>>>> Olivier: this computer is 'hibou'), and which passes the system tests
>>>>>> properly, also runs Gentoo Linux with kernel 2.6.30, numpy-1.3.0 and
>>>>>> python-2.6.2...
>>>>>>
>>>>>> A potential option could be that some kernel configuration is causing
>>>>>> these problems...
>>>>>>
>>>>>> Another option would be that, although the algorithms are supposedly
>>>>>> fixed, that they are not...
>>>>>>
>>>>>> I could check if the calculations diverge always at the same step and,
>>>>>> if so, try to see what function is problematic...
>>>>>>
>>>>>> Other ideas ?
>>>>>>
>>>>>> Do you know any other minimisation library with which I could test to
>>>>>> see if these computers indeed give rise to changing results or if this
>>>>>> is limited to relax (and minfx) ?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>> Séb :)
>>>>>>
>>>>>>
>>>>>>
>>>>>> Edward d'Auvergne wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> This is very strange, very strange indeed! I've never seen anything
>>>>>>> quite like this. Is it only your laptop that is giving this variable
>>>>>>> result? I'm pretty sure that it's not related to a random seed
>>>>>>> because the optimisation at no point uses random numbers - it is 100%
>>>>>>> fixed, pre-determined, etc. and should never, ever vary (well on
>>>>>>> different machines it will change, but never on the same machine).
>>>>>>> What is the operating system on the laptop? Can you run a ram
>>>>>>> checking program or anything else to diagnose hardware failures?
>>>>>>> Maybe the CPU is overheating? Apart from hardware problems, since you
>>>>>>> never recompile Python or numpy between these tests I cannot think of
>>>>>>> anything else that could possibly cause this.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Edward
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2009/9/3 Sébastien Morin<[email protected]>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Hi Ed,
>>>>>>>>
>>>>>>>> I've just tried what you proposed and observed something quite
>>>>>>>> strange...
>>>>>>>>
>>>>>>>> Here are the results:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> ./relax scripts/optimisation_testing.py> /dev/null
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> (stats from my laptop, different trials, see below)
>>>>>>>> iter 161 147 151
>>>>>>>> f_count 765 620 591
>>>>>>>> g_count 168 152 158
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> ./relax -s
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> (stats from my laptop, different trials, see below)
>>>>>>>> iter 146 159 160 159
>>>>>>>> f_count 708 721 649 673
>>>>>>>> g_count 152 166 167 166
>>>>>>>>
>>>>>>>>
>>>>>>>> Problem 1:
>>>>>>>> The results should be the same in both situations, right ?
>>>>>>>>
>>>>>>>> Problem 2:
>>>>>>>> The results should not vary when the test is done multiple times,
>>>>>>>> right
>>>>>>>> ?
>>>>>>>>
>>>>>>>>
>>>>>>>> I have tested different things to find out why the tests give rise to
>>>>>>>> different results as a function of time...
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> ./relax scripts/optimisation_testing.py> /dev/null
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> If you modify the file "test_suite/system_tests/__init__.py", then
>>>>>>>> the result will be different. By modifying, I mean just comment a few
>>>>>>>> lines in the run() function. (I usually do that when I want to speed
>>>>>>>> up
>>>>>>>> the process of testing a specific issue.) Maybe this behavior is
>>>>>>>> related
>>>>>>>> to random seed based on the code files...
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> ./relax -s
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> This one varies as a function of time without any change. Just
>>>>>>>> doing
>>>>>>>> the test several times in a row will have it varying... Maybe this
>>>>>>>> behavior is related to random seed based on the date and time...
>>>>>>>>
>>>>>>>>
>>>>>>>> Any idea ?
>>>>>>>>
>>>>>>>> If you want, Ed, I could create you an account on one of these
>>>>>>>> strange-behaving computers...
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>>
>>>>>>>> Séb
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Edward d'Auvergne wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I've now written a script so that you can fix this. Try running:
>>>>>>>>>
>>>>>>>>> ./relax scripts/optimisation_testing.py> /dev/null
>>>>>>>>>
>>>>>>>>> This will give you all the info you need, formatted ready for
>>>>>>>>> copying
>>>>>>>>> and pasting into the correct file. This is currently only
>>>>>>>>> 'test_suite/system_tests/model_free.py'. Just paste the
>>>>>>>>> pre-formatted
>>>>>>>>> python comment into the correct test, and add the different values
>>>>>>>>> to
>>>>>>>>> the list of values checked.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Edward
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2009/9/3 Sébastien Morin<[email protected]>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Hi Ed,
>>>>>>>>>>
>>>>>>>>>> I just checked my original mail
>>>>>>>>>> (https://mail.gna.org/public/relax-devel/2009-05/msg00003.html).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> For the failure "FAIL: Constrained BFGS opt, backtracking line
>>>>>>>>>> search
>>>>>>>>>> {S2=0.970, te=2048, Rex=0.149}", the counts were initially as
>>>>>>>>>> follows:
>>>>>>>>>> f_count 386
>>>>>>>>>> g_count 386
>>>>>>>>>> and are now:
>>>>>>>>>> f_count 743 694 761
>>>>>>>>>> g_count 168 172 164
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> For the failure "FAIL: Constrained BFGS opt, More and Thuente line
>>>>>>>>>> search {S2=0.970, te=2048, Rex=0.149}", the counts were initially
>>>>>>>>>> as
>>>>>>>>>> follows:
>>>>>>>>>> f_count 722
>>>>>>>>>> g_count 164
>>>>>>>>>> and are now:
>>>>>>>>>> f_count 375 322 385
>>>>>>>>>> g_count 375 322 385
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The different values given for the "just-measured" parameters
>>>>>>>>>> account
>>>>>>>>>> for the 3 different computers I have access to that give rise to
>>>>>>>>>> these
>>>>>>>>>> two annoying failures...
>>>>>>>>>>
>>>>>>>>>> I wounder if the names of the tests in the original mail were not
>>>>>>>>>> mixed,
>>>>>>>>>> as numbers just measured in the second test seem closer to those
>>>>>>>>>> originally posted in the first test, and vice versa...
>>>>>>>>>>
>>>>>>>>>> Anyway, the problem is that there are variations between the
>>>>>>>>>> different
>>>>>>>>>> machines. Variations are also present for the other parameters (s2,
>>>>>>>>>> te,
>>>>>>>>>> rex, chi2, iter).
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Séb :)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Edward d'Auvergne wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Could you check and see if the numbers are exactly the same as in
>>>>>>>>>>> your
>>>>>>>>>>> original email
>>>>>>>>>>> (https://mail.gna.org/public/relax-devel/2009-05/msg00003.html)?
>>>>>>>>>>> Specifically look at f_count and g_count.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> Edward
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2009/9/2 Sébastien Morin<[email protected]>:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Hi Ed,
>>>>>>>>>>>>
>>>>>>>>>>>> I updated my svn copies to r9432 and checked if the problem was
>>>>>>>>>>>> still
>>>>>>>>>>>> present.
>>>>>>>>>>>>
>>>>>>>>>>>> Unfortunately, it is still present...
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Séb
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Edward d'Auvergne wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ah, yes, there is a reason. I went through and fixed a series
>>>>>>>>>>>>> of
>>>>>>>>>>>>> these optimisation difference issues - in my local svn copy. I
>>>>>>>>>>>>> collected these all together and committed them as one after I
>>>>>>>>>>>>> had
>>>>>>>>>>>>> shut the bugs. This was a few minutes ago at r9426. If you
>>>>>>>>>>>>> update
>>>>>>>>>>>>> and test now, it should work.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Edward
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2009/9/2 Sébastien Morin<[email protected]>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Ed,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I just tested the for the presence of this bug (1.3 repository,
>>>>>>>>>>>>>> r9425)
>>>>>>>>>>>>>> and it seems it is still there...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there a reason why it was closed ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> From the data I have, I guess this bug report should be
>>>>>>>>>>>>>>> re-opened.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Maybe I could try to give more details to help debugging...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Séb :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Edward d Auvergne wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Update of bug #14182 (project relax):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Status: Confirmed => Fixed
>>>>>>>>>>>>>>> Assigned to: None => bugman
>>>>>>>>>>>>>>> Open/Closed: Open => Closed
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________________
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Reply to this item at:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <http://gna.org/bugs/?14182>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> Message sent via/by Gna!
>>>>>>>>>>>>>>> http://gna.org/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Sébastien Morin
>>>>>>>>>>>>>> PhD Student
>>>>>>>>>>>>>> S. Gagné NMR Laboratory
>>>>>>>>>>>>>> Université Laval& PROTEO
>>>>>>>>>>>>>> Québec, Canada
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Sébastien Morin
>>>>>>>>>>>> PhD Student
>>>>>>>>>>>> S. Gagné NMR Laboratory
>>>>>>>>>>>> Université Laval& PROTEO
>>>>>>>>>>>> Québec, Canada
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Sébastien Morin
>>>>>>>>>> PhD Student
>>>>>>>>>> S. Gagné NMR Laboratory
>>>>>>>>>> Université Laval& PROTEO
>>>>>>>>>> Québec, Canada
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>> --
>>>> Sébastien Morin
>>>> PhD Student
>>>> S. Gagné NMR Laboratory
>>>> Université Laval& PROTEO
>>>> Québec, Canada
>>>>
>>>>
>>>>
>>>>
>> --
>> Sébastien Morin
>> PhD Student
>> S. Gagné NMR Laboratory
>> Université Laval& PROTEO
>> Québec, Canada
>>
>>
>>
--
Sébastien Morin
PhD Student
S. Gagné NMR Laboratory
Université Laval& PROTEO
Québec, Canada
_______________________________________________
relax (http://nmr-relax.com)
This is the relax-devel mailing list
[email protected]
To unsubscribe from this list, get a password
reminder, or change your subscription options,
visit the list information page at
https://mail.gna.org/listinfo/relax-devel
