Thanks. Now that I know what to look for, I should be able to figure it
out.
BTW, I switched the script that ultimately runs the mpiexec from tcsh to
bash and the problem went away. Not complaining but do you have any idea
why that might be?
Gene
-- Eugene M Fluder, Jr, PhD
Computational Scientist
Scientific Computing
The Mt. Sinai School of Medicine
One Gustave L. Levy Place, Box 1498
New York, NY 10029-6574
T: 212 659 8608
F: 646 537 8660
E: eugene.flu...@mssm.edu
On 7/6/12 10:57 AM, "Holger Mickler" <holger.mick...@tu-dresden.de> wrote:
>Oh, I just realized that you are probably using the Open MPI version of
>VT which
>builds as part of the Open MPI build. I'm not 100% sure if the
>modification of
>config.h works as laid out, but it should... you need to look out for VT's
>config.h then, not Open MPI's.
>
>Holger
>
>
>On 07/06/2012 04:54 PM, Holger Mickler wrote:
>> Hi Gene,
>>
>> this error is often caused by insufficiently synchronized TSCs (time
>>stamp
>> counter) of different processors/cores.
>> When VT uses the TSC for timing the events (it does that by default),
>>and the
>> processes switch to another core during execution, it may well happen
>>that the
>> next recorded time stamp is earlier in time than the last one.
>>
>> One possibility to avoid this situation is pinning the processes to
>>cores - Open
>> MPI has functionality for realizing this, see
>> http://www.open-mpi.org/faq/?category=tuning#using-paffinity
>>
>> If this is not feasible, you may use another clock source with VT which
>>provides
>> global time. To do this, you need to compile another version of VT. Run
>> configure as usual, then edit config.h: replace the value of
>> #define TIMER [...]
>> with e.g.
>> #define TIMER TIMER_CLOCK_GETTIME
>> or
>> #define TIMER TIMER_GETTIMEOFDAY
>> depending on what is available on your system. Be aware that the
>>resolution of
>> those clocks is not as high as the TSC's.
>>
>> Have a look inside config.h at the place of the mentioned variables -
>>there is
>> some documentation there.
>> Afterwards, compile and install VT. Using this version, you should not
>>encounter
>> the errors anymore.
>>
>> Regards,
>> Holger
>>
>>
>>
>>
>> On 07/06/2012 04:04 PM, Fluder, Eugene wrote:
>>> I got the following error running a VT enabled run of AMBER. This was
>>>reported
>>> in December of 2009 under almost identical conditions but the thread
>>>does not
>>> contain a resolution. I reran the test with VT_UNIFY=no and it
>>>completed
>>> normally. The same error occurred when I ran vtunify separately. Any
>>>help?
>>>
>>> Was this ever resolved?
>>>
>>> Gene
>>>
>>> [fludee01@node7-10 trace_noiox]$ vtunify 8 a
>>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file:
>>>OTF_WBuffer.c, line: 308:
>>> time not increasing. (t= 99459634, p= 6)
>>> vtunify: Error: Could not read events of OTF stream [namestub
>>>./a__ufy.tmp id 6]
>>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file:
>>>OTF_WBuffer.c, line: 308:
>>> time not increasing. (t= 105413860, p= 5)
>>> vtunify: Error: Could not read events of OTF stream [namestub
>>>./a__ufy.tmp id 5]
>>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file:
>>>OTF_WBuffer.c, line: 308:
>>> time not increasing. (t= 103189146, p= 7)
>>> vtunify: Error: Could not read events of OTF stream [namestub
>>>./a__ufy.tmp id 7]
>>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file:
>>>OTF_WBuffer.c, line: 308:
>>> time not increasing. (t= 100509810, p= 8)
>>> vtunify: Error: Could not read events of OTF stream [namestub
>>>./a__ufy.tmp id 8]
>>> vtunify: An error occurred during unifying events - Terminating ...
>>>
>>> -- /Eugene M Fluder, Jr, PhD/
>>> /Computational Scientist/
>>> /Scientific Computing/
>>> /
>>> /
>>> /The Mt. Sinai School of Medicine/
>>> /One Gustave L. Levy Place, Box 1498/
>>>
>>> /New York, NY 10029-6574/
>>>
>>> /
>>> /
>>>
>>> /T: 212 659 8608/
>>>
>>> /F: 646 537 8660/
>>>
>>> /E: eugene.flu...@mssm.edu/
>>>
>>> /
>>> /
>>>
>>> //
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> devel mailing list
>>> de...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> _______________________________________________
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>--
>Dipl.-Inf. Holger Mickler
>
>Technische Universität Dresden
>Center for Information Services
>and High Performance Computing (ZIH)
>01062 Dresden
>Germany
>
>Office: Willers-Bau (WIL) A36
>Tel.: +49 (351) 463-37903
>Fax: +49 (351) 463-37773
>E-Mail: holger.mick...@tu-dresden.de
>
>
>_______________________________________________
>devel mailing list
>de...@open-mpi.org
>http://www.open-mpi.org/mailman/listinfo.cgi/devel
