Hi, Rafeal, That's great to hear. Thank you so much!
I'll let you know when I confirm CHPL_GASNET_SEGMENT=large works for my benchmark. Best, Akihiro On Jan 30, 2014, at 3:17 PM, Rafael Larrosa Jiménez wrote: > Hi, > >> Hi Rafael -- >> >> Thanks for your analysis and description. > > Thanks to you and all that have help with this problem, I have been wrestling > with it a long time, and at the end left it for impossible at the beginning > of > the year. > > I have just checked that my code works now fine with ibv, it feels good to > know > it was not a bug in my code :-) > >> I don't have any problem with changing the default segment for 'ibv' and >> agree that it sounds like the right thing to do if it works around this >> issue. It seems like 'fast' would be preferable to 'large' from a >> performance perspective (based on a quick glance at the docs -- I'm mostly >> unfamiliar with 'large'). > > But fast doesn't work with ibv, it shows this error: > > *** FATAL ERROR: Unexpected error Bad address (rc=1 errno=14) when registering > the segment > > So I think we should stay with large on ibv, I have not benchmarked its speed > impact, but correct bulks comm should be enought to offset some impact on > speed. > >> In trying to use the 'fast' segment, do you know whether the memory >> allocator used changed to either tcmalloc or dlmalloc (either >> automatically or manually)? That is probably necessary for correctness, >> but should've (hopefully) happened automatically. printchplenv can be used >> to verify. > > Yes, it automatically uses dlmalloc as explained in > doc/release/README.chplenv, I have checked it. > >> Note that I believe you should only need to set CHPL_GASNET_SEGMENT. The >> CHPL_MAKE_ variables should follow automatically (and are not intended to >> be set by the end-user). > > Yes, you are right, it seems that using only : > export CHPL_GASNET_SEGMENT=large > > works fine, but I worked by reverse-engeneering :-) > > Thanks to all, > > Rafael > > >> Thanks again, >> -Brad >> >> On Thu, 30 Jan 2014, Rafael Larrosa Jiménez wrote: >>> El Jueves, 30 de enero de 2014 12:21:11 Akihiro Hayashi escribió: >>>> Hi, >>>> >>>> Thanks for your helping me to understand the problem. >>>> I think now I understand the problem correctly. >>>> >>>> Rafael Asenjo and Rafael Larrosa. >>>> I'm pretty sure my assignment have involved a contiguous block and >>>> useBulkTransfer is active and useBulkTransferStride is not active in my >>>> chapel compiler. I just wanted to make both sure my code >>>> (useBulkTransfer) >>>> and Rafaerl Larrosa's code (useBulkTransferStride) can fail due to the >>>> ibv-conduit bug (https://upc-bugs.lbl.gov/bugzilla/show_bug.cgi?id=495). >>> >>> First, tell that I'm not the developer, but a client of that code. >>> >>> Second, I have been writting this email for several hours, as I changed my >>> vision on the problem, have been reading the ib verbs manuals, ib code, >>> etc. >>> >>> At the end I have found the solution, by default chapel puts all memory as >>> memory accesible by RDMA, which seems to be the problem, as it can be >>> "unpinned, if instead only "large" portions are used for RDMA then it >>> works >>> fine, as explained in one of the messages: >>> >>> --- >>> In a GASNET_SEGMENT_FAST or _LARGE configuration the segment is obtained >>> at >>> startup via mmap() and is never unmapped. So, sending from inside the >>> GASNet segment in these two cases will ensure this bug cannot occur. >>> --- >>> >>> So if you define : >>> >>> export CHPL_MAKE_GASNET_SEGMENT=large >>> export CHPL_MAKE_COMM_SEGMENT=large >>> export CHPL_GASNET_SEGMENT=large >>> >>> Then do a make clobber followed by a make and recompile your program, it >>> should work fine. >>> >>> BTW, when using portals, gemini or aries, the fast segment is used, as >>> explained in : >>> >>> doc/release/README.multilocale >>> --- >>> 3) Advanced GASNet users can set CHPL_GASNET_SEGMENT to choose a >>> >>> memory segment to use with GASNet. Current defaults are: >>> When CHPL_COMM_SUBSTRATE is... Chapel will choose... >>> >>> portals fast >>> gemini fast >>> (other) everything >>> >>> --- >>> >>> But using fast with ibv gives this error: >>> *** FATAL ERROR: Unexpected error Bad address (rc=1 errno=14) when >>> registering the segment >>> >>> But that is not a problem as large segments are the solution, hope they >>> don't break other things :-) >>> >>> I have tested it with Akihiro code, and it seems to work fine now. >>> >>> Perphaps is a good idea to change the default to use large segments with >>> ibv. >>> >>> Greets, >>> >>> Rafael >>> >>> EMAIL: [email protected] Edificio de Bioinnovación >>> TELEF: + 34951952788 C/ Severo Ochoa 34 >>> FAX : +34951952792 Parque Tecnológico de Andalucía >>> >>> 29590 Málaga (SPAIN) > -- > Rafael Larrosa Jiménez > Centro de Supercomputación y Bioinformática - http://www.scbi.uma.es > Universidad de Málaga > > EMAIL: [email protected] Edificio de Bioinnovación > TELEF: + 34951952788 C/ Severo Ochoa 34 > FAX : +34951952792 Parque Tecnológico de Andalucía > 29590 Málaga (SPAIN) > > ------------------------------------------------------------------------------ WatchGuard Dimension instantly turns raw network data into actionable security intelligence. It gives you real-time visual feedback on key security issues and trends. Skip the complicated setup - simply import a virtual appliance and go from zero to informed in seconds. http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk _______________________________________________ Chapel-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/chapel-developers
