Hi Rafael --
Thanks for your analysis and description.
I don't have any problem with changing the default segment for 'ibv' and
agree that it sounds like the right thing to do if it works around this
issue. It seems like 'fast' would be preferable to 'large' from a
performance perspective (based on a quick glance at the docs -- I'm mostly
unfamiliar with 'large').
In trying to use the 'fast' segment, do you know whether the memory
allocator used changed to either tcmalloc or dlmalloc (either
automatically or manually)? That is probably necessary for correctness,
but should've (hopefully) happened automatically. printchplenv can be used
to verify.
Note that I believe you should only need to set CHPL_GASNET_SEGMENT. The
CHPL_MAKE_ variables should follow automatically (and are not intended to
be set by the end-user).
Thanks again,
-Brad
On Thu, 30 Jan 2014, Rafael Larrosa Jiménez wrote:
El Jueves, 30 de enero de 2014 12:21:11 Akihiro Hayashi escribió:
Hi,
Thanks for your helping me to understand the problem.
I think now I understand the problem correctly.
Rafael Asenjo and Rafael Larrosa.
I'm pretty sure my assignment have involved a contiguous block and
useBulkTransfer is active and useBulkTransferStride is not active in my
chapel compiler. I just wanted to make both sure my code (useBulkTransfer)
and Rafaerl Larrosa's code (useBulkTransferStride) can fail due to the
ibv-conduit bug (https://upc-bugs.lbl.gov/bugzilla/show_bug.cgi?id=495).
First, tell that I'm not the developer, but a client of that code.
Second, I have been writting this email for several hours, as I changed my
vision on the problem, have been reading the ib verbs manuals, ib code, etc.
At the end I have found the solution, by default chapel puts all memory as
memory accesible by RDMA, which seems to be the problem, as it can be
"unpinned, if instead only "large" portions are used for RDMA then it works
fine, as explained in one of the messages:
---
In a GASNET_SEGMENT_FAST or _LARGE configuration the segment is obtained at
startup via mmap() and is never unmapped. So, sending from inside the GASNet
segment in these two cases will ensure this bug cannot occur.
---
So if you define :
export CHPL_MAKE_GASNET_SEGMENT=large
export CHPL_MAKE_COMM_SEGMENT=large
export CHPL_GASNET_SEGMENT=large
Then do a make clobber followed by a make and recompile your program, it
should work fine.
BTW, when using portals, gemini or aries, the fast segment is used, as
explained in :
doc/release/README.multilocale
---
3) Advanced GASNet users can set CHPL_GASNET_SEGMENT to choose a
memory segment to use with GASNet. Current defaults are:
When CHPL_COMM_SUBSTRATE is... Chapel will choose...
portals fast
gemini fast
(other) everything
---
But using fast with ibv gives this error:
*** FATAL ERROR: Unexpected error Bad address (rc=1 errno=14) when registering
the segment
But that is not a problem as large segments are the solution, hope they don't
break other things :-)
I have tested it with Akihiro code, and it seems to work fine now.
Perphaps is a good idea to change the default to use large segments with ibv.
Greets,
Rafael
--
Rafael Larrosa Jiménez
Centro de Supercomputación y Bioinformática - http://www.scbi.uma.es
Universidad de Málaga
EMAIL: [email protected] Edificio de Bioinnovación
TELEF: + 34951952788 C/ Severo Ochoa 34
FAX : +34951952792 Parque Tecnológico de Andalucía
29590 Málaga (SPAIN)
------------------------------------------------------------------------------
WatchGuard Dimension instantly turns raw network data into actionable
security intelligence. It gives you real-time visual feedback on key
security issues and trends. Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
_______________________________________________
Chapel-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/chapel-developers
