Can you try adding this to Open-MPI options:
--mca btl ^sm
Full command:
mpiexec --mca btl ^sm -n 40 $RAY -o $OUTDIR -p $LEFT $RIGHT -k 31 \
-search $NCBIDIR/NCBI-Finished-Bacterial-Genomes \
-with-taxonomy $NCBIDIR/Genome-to-Taxon.tsv
$NCBIDIR/TreeOfLife-Edges.tsv $NCBIDIR/Taxon-Names.tsv
This Open-MPI option will disable Open-MPI's "sm" byte transfer layer. "sm"
means shared memory.
All messages will go through "tcp" (using the loopback since it's all on the
same machine),
or through "self".
p.s.: Open-MPI 1.4.3 is really old (2010-10-05). The last release is Open-MPI
1.6.4.
On 02/22/2013 08:06 AM, James Vincent wrote:
> It is OpenMPI 1.4.3. Here is the blurb at the start of a run:
>
>
> MAXKMERLENGTH: 32
> KMER_U64_ARRAY_SIZE: 1
> Maximum coverage depth stored by CoverageDepth: 4294967295
> MAXIMUM_MESSAGE_SIZE_IN_BYTES: 4000 bytes
> FORCE_PACKING = n
> ASSERT = n
> HAVE_LIBZ = n
> HAVE_LIBBZ2 = n
> CONFIG_PROFILER_COLLECT = n
> CONFIG_CLOCK_GETTIME = n
> __linux__ = y
> _MSC_VER = n
> __GNUC__ = y
> RAY_32_BITS = n
> RAY_64_BITS = y
> MPI standard version: MPI 2.1
> MPI library: Open-MPI 1.4.3
> Compiler: GNU gcc/g++ 4.4.6 20120305 (Red Hat 4.4.6-4)
>
> Rank 0: Operating System: Linux (__linux__) POSIX (OS_POSIX)
>
>
> On Fri, Feb 22, 2013 at 8:01 AM, Sébastien Boisvert
> <[email protected]> wrote:
>> Which MPI library are you using ?
>>
>>
>>
>> On 02/22/2013 07:34 AM, James Vincent wrote:
>>> Here are the last 50 lines :
>>>
>>> -bash-4.1$ tail -50 out.run
>>>
>>> OPERATION_incrementReferences operations: 32762
>>> OPERATION_decrementReferences operations: 32386
>>>
>>> OPERATION_purgeVirtualColor operations: 1567
>>> **********************************************************
>>>
>>> Rank 25: assembler memory usage: 138560 KiB
>>> Rank 31 biological abundances 3210 [1/1] [1752/2254] [4/7]
>>> Rank 31 RAY_SLAVE_MODE_ADD_COLORS processed files: 43/56
>>> Rank 31 RAY_SLAVE_MODE_ADD_COLORS processed sequences in file: 3/7
>>> Rank 31 RAY_SLAVE_MODE_ADD_COLORS total processed sequences: 81/107
>>> Rank 31 RAY_SLAVE_MODE_ADD_COLORS processed k-mers for current
>>> sequence: 2341425/0
>>> Rank 31 RAY_SLAVE_MODE_ADD_COLORS total processed k-mers: 149500000
>>> Speed RAY_SLAVE_MODE_ADD_COLORS 20803 units/second
>>>
>>> **********************************************************
>>> Coloring summary
>>> Number of virtual colors: 232
>>> Number of real colors: 3444
>>>
>>> Keys in index: 230
>>> Observed collisions when populating the index: 0
>>> COLOR_NAMESPACE_MULTIPLIER= 10000000000000000
>>>
>>> Operations
>>>
>>> OPERATION_getVirtualColorFrom operations: 35380
>>>
>>> OPERATION_IN_PLACE_ONE_REFERENCE: 31951
>>> OPERATION_NO_VIRTUAL_COLOR_HAS_HASH_CREATION operations: 1730
>>> OPERATION_VIRTUAL_COLOR_HAS_COLORS_FETCH operations: 1506
>>> OPERATION_NO_VIRTUAL_COLOR_HAS_COLORS_CREATION operations: 193
>>>
>>> OPERATION_createVirtualColorFrom operations: 1923
>>>
>>> OPERATION_allocateVirtualColorHandle operations: 1923
>>> OPERATION_NEW_FROM_EMPTY operations: 1692
>>> OPERATION_NEW_FROM_SCRATCH operations: 231
>>>
>>> OPERATION_applyHashOperation operations: 37303
>>> OPERATION_getHash operations: 0
>>>
>>> OPERATION_incrementReferences operations: 35380
>>> OPERATION_decrementReferences operations: 34985
>>>
>>> OPERATION_purgeVirtualColor operations: 1693
>>> **********************************************************
>>>
>>> Rank 31: assembler memory usage: 139452 KiB
>>>
>>> On Fri, Feb 22, 2013 at 7:13 AM, Sébastien Boisvert
>>> <[email protected]> wrote:
>>>> What's the last thing reported in stdout ?
>>>>
>>>> On 02/22/2013 06:23 AM, jjv5 wrote:
>>>>> Hello,
>>>>>
>>>>> I am running ray meta on a shared memory machine with 40 cores and 1TB
>>>>> memory. One very small job with 25K reads finished and gave various
>>>>> taxonomic outputs. Jobs with slightly
>>>>> more input seem to never finish. The output log just stops but there
>>>>> are no errors indicated. Where might one start to look to determine
>>>>> where a job has gone wrong?
>>>>>
>>>>> The command I use is below. There are 100,000 paired reads.
>>>>>
>>>>> mpiexec -n 40 $RAY -o $OUTDIR -p $LEFT $RIGHT -k 31 \
>>>>> -search $NCBIDIR/NCBI-Finished-Bacterial-Genomes \
>>>>> -with-taxonomy $NCBIDIR/Genome-to-Taxon.tsv
>>>>> $NCBIDIR/TreeOfLife-Edges.tsv $NCBIDIR/Taxon-Names.tsv
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Everyone hates slow websites. So do we.
>>>>> Make your web apps faster with AppDynamics
>>>>> Download AppDynamics Lite for free today:
>>>>> http://p.sf.net/sfu/appdyn_d2d_feb
>>>>> _______________________________________________
>>>>> Denovoassembler-users mailing list
>>>>> [email protected]
>>>>> https://lists.sourceforge.net/lists/listinfo/denovoassembler-users
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Everyone hates slow websites. So do we.
>>>> Make your web apps faster with AppDynamics
>>>> Download AppDynamics Lite for free today:
>>>> http://p.sf.net/sfu/appdyn_d2d_feb
>>>> _______________________________________________
>>>> Denovoassembler-users mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/denovoassembler-users
>>
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_d2d_feb
>> _______________________________________________
>> Denovoassembler-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/denovoassembler-users
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb
_______________________________________________
Denovoassembler-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/denovoassembler-users
