Hi, Rafael,
Thanks for your reply. I inlined my comments below:
I’ve been talking to Rafael Larrosa regarding the issue you are reporting. He
has conducted some experiments with your code on Titan (Cray XK7 at ORNL, which
features Cray Gemini interconnect) and there are no communication problem
there. However, we have here in Malaga a cluster based on Infiniband
(ibv-conduit) and the execution fails on that platform. I’ve also confirmed
that udp-conduit does not pose any problem.
I really appreciate your and Rafael Larrosa's experiments. I'm glad to hear
that this is not a problem in Chapel compiler.
Rafael Larrosa told me that he faced the same issue last month and that after
spending more than two weeks tackling the problem he is almost sure that there
is a bug (or a maximum buffer size limitation) in the ibv-conduit
implementation of gasnet. For his code, he found a turning point when the
transferred buffer was 128MBytes: smaller communications work fine, but larger
always fail. He says it was tricky, because when you try to isolate the problem
(i.e. isolate the particular transfer that fails by executing just this single
communication) then the problem vanish. So it will be challenging to chase this
bug.
Sure, now I understand there is a bug in the ibv-conduit implementation of
gasnet. and yes, It seems fixing the bug is very difficult. Actually, an
original benchmark I want to run spawns many tasks by begin statement and each
task does bulk transfer. I can imagine the benchmark exceeds some limit like
Rafael's code. I'm also wondering why the simplified code has this problem.
There might be another problem.
You may want to circumvent the bug by:
1.- Not using bulkComms optimization (-suseBulkTransferStride=false
-suseBulkTransfer=false). —> Slower comms.
2.- Implementing a version of bulkComms that splits big messages into smaller
ones. —> wearisome tinkering
3.- Avoid ibv-conduit —> 207 out of the 500 supercomputers in latest top500
list are based on ibv
4.- Dive into the ibv-conduit implementation —> Probably not your main research
goal
For the time being we are conducting all our experiments on Cray machines, so
we do not plan (and do not have time) to tackle 2 or 4, so we are getting by
with 3.
Exactly, 4 is not my research goal, I 'd choose 3 if a benchmark I would like
to run use bulk transfer. Thanks for your suggestions.
If Rafael wants to chime in, he can probably give you more details and advices,
should you want to debug your code at a lower level.
I would appreciate if he could give me more details. I think I should mention
the bug in my paper or something.
Best,
Akihiro
On Jan 29, 2014, at 4:46 AM, Rafael Asenjo Plaza wrote:
Hi Akihiro,
I’ve been talking to Rafael Larrosa regarding the issue you are reporting. He
has conducted some experiments with your code on Titan (Cray XK7 at ORNL, which
features Cray Gemini interconnect) and there are no communication problem
there. However, we have here in Malaga a cluster based on Infiniband
(ibv-conduit) and the execution fails on that platform. I’ve also confirmed
that udp-conduit does not pose any problem.
Rafael Larrosa told me that he faced the same issue last month and that after
spending more than two weeks tackling the problem he is almost sure that there
is a bug (or a maximum buffer size limitation) in the ibv-conduit
implementation of gasnet. For his code, he found a turning point when the
transferred buffer was 128MBytes: smaller communications work fine, but larger
always fail. He says it was tricky, because when you try to isolate the problem
(i.e. isolate the particular transfer that fails by executing just this single
communication) then the problem vanish. So it will be challenging to chase this
bug.
You may want to circumvent the bug by:
1.- Not using bulkComms optimization (-suseBulkTransferStride=false
-suseBulkTransfer=false). —> Slower comms.
2.- Implementing a version of bulkComms that splits big messages into smaller
ones. —> wearisome tinkering
3.- Avoid ibv-conduit —> 207 out of the 500 supercomputers in latest top500
list are based on ibv
4.- Dive into the ibv-conduit implementation —> Probably not your main research
goal
For the time being we are conducting all our experiments on Cray machines, so
we do not plan (and do not have time) to tackle 2 or 4, so we are getting by
with 3.
If Rafael wants to chime in, he can probably give you more details and advices,
should you want to debug your code at a lower level.
Regards,
Rafa.
El 28/01/2014, a las 19:31, Akihiro Hayashi <[email protected]> escribió:
Hi, Rafael,
Sorry for the delayed reply.
Let me share the program that reproduces the problem. (attached below)
As you can see, the program prints "INVALID? : true" if we get bulk copy transfer error,
otherwise it prints "INVALID?: false".
I get the error when I run the program on 2 locales with ibv-conduit
(mpi-spawner). The input data size is : matrixSize = 2000 and tileSize = 200.
Please let me know if you want the input file.
Note that I don't get the error when I run the program on 1 locale. In
addition, I don't get the error with smaller data size even on 2 or more
locales (e.g 10x10 matrix and 2x2 tile size).
I'm guessing using ibv-conduit and transferring a certain amount of data incurs
this problem.
FYI, using udp-conduit (amudprun) does not show the error.
Please let me know if you have any comments and questions.
Best,
Akihiro
--
use BlockDist;
config const matrixSize: int(32) = -1;
config const tileSize: int(32) = -1;
config const inFile: string = "m_2000.in";
const zero: int(32) = 0;
var tile_array_indices = {zero..tileSize-1,zero..tileSize-1};
class Tile {
var tile_array: [tile_array_indices] real;
}
proc read_2D_array ( fileName: string, matrixSize: int(32) ) {
var input_stream = open (fileName, iomode.r);
var reader = input_stream.reader();
var matrix_index_2D = {0..matrixSize-1, 0..matrixSize-1};
var array: [matrix_index_2D] real;
for ij in matrix_index_2D do {
reader.read(array(ij));
}
input_stream.close();
reader.close();
// if (debug) { writeln("whole array: ",array); }
return array;
}
proc main(): void {
writeln("numLocales : ", numLocales);
var numTiles: int(32) = matrixSize/tileSize;
var numTiles_2: int(64) = matrixSize/tileSize;
var whole_array = read_2D_array(inFile, matrixSize);
var proto_ijk_space = {zero..numTiles_2-1, zero..numTiles_2, zero..numTiles_2};
var ijk_space = proto_ijk_space dmapped Block(boundingBox=proto_ijk_space);
var lkji_tiles: [ijk_space] Tile;
for i in zero..numTiles-1 do {
for j in zero..i do {
on lkji_tiles(i,j,zero).locale do {
var curr_tile: Tile = new Tile();
for (ii,jj) in tile_array_indices do {
curr_tile.tile_array(ii,jj) =
whole_array(i*tileSize+ii,j*tileSize+jj);
}
lkji_tiles(i,j,zero) = curr_tile;
}
}
}
var invalid : bool = false;
for i in zero..numTiles-1 do {
for iB in zero..tileSize-1 do {
for j in zero..i do {
var temp = lkji_tiles(i,j,zero).tile_array;
if(i != j) {
for jB in zero..tileSize-1 do {
if (temp(iB,jB) != lkji_tiles(i, j,
zero).tile_array(iB, jB)) {
invalid = true;
}
}
} else {
for jB in zero..iB do {
if (temp(iB,jB) != lkji_tiles(i, j,
zero).tile_array(iB, jB)) {
invalid = true;
}
}
}
}
}
}
writeln("INVALID? : ", invalid);
}
On Jan 22, 2014, at 1:46 PM, Akihiro Hayashi wrote:
Hi Rafael,
Thanks for your reply.
I inlined my comments below:
May we have a simplified copy of your code (kinda the snippet provided below
but with initial values for tileSize, numTiles_2, k, etc. i.e. something that
compiles) so that we can also give it a go?
Yes, it would be better if we can have a simplified code.
Actually, I have been trying to make a simple code that reproduce this problem
for several weeks. finally I managed to make it this morning.
Let me ask my advisor if we can show you the code.
Would you like to try also with these flags?:
-suseBulkTransferStride=true -suseBulkTransfer=false
I tried these flags, but I still get the error.
I'll keep you updated.
Best,
Akihiro
On Jan 22, 2014, at 5:23 AM, Rafael Asenjo Plaza wrote:
Hi Akihiro,
May we have a simplified copy of your code (kinda the snippet provided below
but with initial values for tileSize, numTiles_2, k, etc. i.e. something that
compiles) so that we can also give it a go?
Would you like to try also with these flags?:
-suseBulkTransferStride=true -suseBulkTransfer=false
Thank you,
Rafa.
El 21/01/2014, a las 18:33, Akihiro Hayashi <[email protected]> escribió:
Dear Chapel developers,
This is Akihiro Hayashi, postdoc at Rice University.
I'm writing this to ask array copy failure in chapel.
I'm trying to evaluate some chapel benchmark across multiple nodes but I get
strange error.
Please note that I'm using old version of chapel compiler (r21945) with
qthread-1.10 and GASNet-1.20.2(infiniband-conduit, mpi-spawner) because the
latest version does not work.
With the latest version of chapel compiler (r22568) with qthread-1.10 and
GASNet-1.22.0(infiniband-conduit, mpi-spawner), I get SEGV when running simple
program (coforall loc in Locales do on loc { writeln(loc); }) across multiple
nodes with mpi spawner.
This is another problem but I have not investigated this problem yet. I'll work
on this later.
The following problem might be fixed in the latest version, but any comments
and suggestions are appreciated.
Here is part of my code.
The main data structure is a 3-dimensional array, which is declared as a
distributed array that each of its element refers to a 2-dimension array.
You can see array copy statement (liBlock = lkji_tiles(k,k,k+1).tile_array;) in
Line 11. I want to use this copy statement because the Chapel compiler
generates bulk transfer code, which accelerates program execution.
// Code
1: const zero: int(32) = 0;
2: var tile_array_indices = {zero..tileSize-1,zero..tileSize-1};
3: class Tile {
4: var tile_array: [tile_array_indices] real;
5: }
6: var proto_ijk_space = {zero..numTiles_2-1, zero..numTiles_2,
zero..numTiles_2};
7: var ijk_space = proto_ijk_space dmapped Block(boundingBox=proto_ijk_space);
8: var lkji_tiles: [ijk_space] Tile;
...
9 :begin {
...
10: var liBlock: [tile_array_indices] real;
11: liBlock = lkji_tiles(k,k,k+1).tile_array;
12: for (m,n) in tile_array_indices {
13: if (liBlock(m,n) != lkji_tiles(k,k,k+1).tile_array(m,n)) {
14: invalid = true;
15: }
16: }
17: if (invalid) { writln("Copy Failed");}
18: ...
19: }
...
In my experiment, when running the program on 2 or more locales, the program prints "Copy
Failed" which means "liBlock = lkji_tiles(k,k,k+1).tile_array;" in Line 11 failed.
This happens sometime (not always). and I confirmed the copy is successfully
done if I replace the array copy in Line 11 with copy loop.
Additionally, I also see the same behavior when I replace the array copy in
Line 11 with liBlock._value.doiBulkTransfer(lkji_tiles(k,k,k+1).tile_array);.
Here is an output log at runtime when I compile the program with -s
debugBulkTransfer (tileSize=200):
-- Log starts here
In DefaultRectangularArr.doiBulkTransfer(): Alo=(0, 0), Blo=(0, 0), len=40000,
elemSize=8;
-- End of Log
In both cases, the runtime internally calls chpl_comm_get API(*) and the API
takes the above parameters.
I think it looks good.
(*) Please take a look at doiBulkTransfer function in
CHPL_HOME/modules/internal/DefaultRectangular.chpl
Any comments and suggestions are appreciated.
Best regards,
Akihiro
------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
Chapel-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/chapel-developers
__
Rafael Asenjo Plaza
Dept. Arquitectura de Computadores
Complejo Tecnologico Campus de Teatinos
E-29071 MALAGA (SPAIN)
Tel: +34 95 213 27 91
Fax: +34 95 213 27 90
http://www.ac.uma.es/~asenjo
__
Rafael Asenjo Plaza
Dept. Arquitectura de Computadores
Complejo Tecnologico Campus de Teatinos
E-29071 MALAGA (SPAIN)
Tel: +34 95 213 27 91
Fax: +34 95 213 27 90
http://www.ac.uma.es/~asenjo
------------------------------------------------------------------------------
WatchGuard Dimension instantly turns raw network data into actionable
security intelligence. It gives you real-time visual feedback on key
security issues and trends. Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
_______________________________________________
Chapel-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/chapel-developers
