In your listing, I did not see that you are using a 64-bits version of linux. EVen on 64 bits hardware, this will fail without a 64 bit kernel and a 64 bit glibc. Are you sure you have those installed? I know SUSE has such a configuration available but, generally, if the distro does not explicitly state that it is 64-bit, it isn't.
- michael dykman On Mon, Mar 31, 2008 at 10:06 AM, Eric Iverson <[EMAIL PROTECTED]> wrote: > I am interested in the problem you report but haven't had a chance to work > on it yet. With luck I will be able to do so in a few days. > > Thanks for taking the initial steps of fixing the SetFilePointerR vs 64bit > etc problems and we'll see that this work gets included in a future J libary > release. > > The first thing I wouold do is a web search on Linux and large mapped files > and wonder if you have already done that. > > (I replied to your original message to programming and not the one to chat > as this is quite properly a programming issue) > > > > ----- Original Message ----- > From: "Jorge Antonio Arredondo Vargas" <[EMAIL PROTECTED]> > To: <[email protected]> > Sent: Thursday, March 27, 2008 11:36 PM > Subject: [Jprogramming] Memory-map a large file on Linux > > > > I'm trying to memory-map a large file (more than 6 GB) from J64 on > > Linux, but I'm getting a ENOMEM error which I haven't been able to > > solve. > > I can successfully map such kind of file, (indeed bigger: 200GB) on > > Windows XP x64 and use it as a J noun, however. > > > > Have any of you done this already? > > > > My system has the following specifications: > > OS= OpenSuse 10.2 > > RAM= 8GB > > CPU= 2 Dual Core Opteron64 @2GHz > > 9!:12'' > > 5 > > 9!:14'' > > j601/2006-09-04/09:20 > > > > I start getting problems after testing with files bigger than 6GB, > > which is close to the physical amount of RAM in my machine, so I'm > > thinking the mmap system call might be trying to actually load the > > file into memory, perhaps this problem can be solved by using a > > different flag on the mmap call. > > > > > > When I first tried to create the file I want to map, I discovered > > jmf.ijs didn't worked with big files: > > ======================================================================== > > require 'jmf files' > > FN=: '/usr/test.jmf' > > SIZE_6GB=: 6 * 1024(<[EMAIL PROTECTED])3 > > HS=: 568 > > > > createjmf_jmf_ FN;SIZE_6GB > > assert (SIZE_6GB+HS)-:fsize FN > > |assertion failure: assert > > | assert (SIZE_6GB+HS)-:fsize FN > > |[-3] > > fsize FN > > 64 > > ======================================================================== > > > > > > I solved this by defining c_open and c_lseek with large file support > > (LFS) function equivalents and then running the relevant code of > > createjmf_jmf_, here is the code: > > ======================================================================== > > cd=: 15!:0 > > c_open=: 'libc.so.6 open64 i *c i i'&cd > > c_close=: 'libc.so.6 close i i'&cd > > c_lseek=: 'libc.so.6 lseek64 i i l i'&cd > > c_write=: 'libc.so.6 write i i * i'&cd > > O_RDWR=: 2 > > O_CREAT=: 64 > > O_TRUNC=: 512 > > JINT=: 4 > > SZI=: 8 > > AFNJA=: 2 > > SEEK_SET=: 0 > > > > ]fn=: FN > > /usr/test.jmf > > ]ms=: SIZE_6GB > > 6442450944 > > ]ts=: HS+ms > > 6442451512 > > > > ]fh=: 0 pick c_open fn; ((23 b.)/O_RDWR,O_CREAT,O_TRUNC); 8b666 > > 19 > > c_lseek fh;(<:ts);SEEK_SET > > +-----------+--+----------+-+ > > |_2147483081|19|6442451511|0| > > +-----------+--+----------+-+ > > c_write fh; (,0{a.); 0+1 > > +-+--+-+-+ > > |1|19| |1| > > +-+--+-+-+ > > c_lseek fh;0 ;SEEK_SET > > +-+--+-+-+ > > |0|19|0|0| > > +-+--+-+-+ > > ]d=: HS,AFNJA,ms,JINT,0,0,1,0 NB. integer empty list > > 568 2 6442450944 4 0 0 1 0 > > c_write fh;d;(SZI*#d) > > +--+--+--------------------------+--+ > > |64|19|568 2 6442450944 4 0 0 1 0|64| > > +--+--+--------------------------+--+ > > c_close fh > > +-+--+ > > |0|19| > > +-+--+ > > assert (SIZE_6GB+HS)-:fsize FN > > ======================================================================== > > > > > > After creating the file, I tried using map_jmf_, but J crashed with a > > segmentation fault: > > ======================================================================== > > map_jmf_ 'test';FN > > 9183 Segmentation fault > > ======================================================================== > > > > > > So I defined c_mmap with LFS, and I could finally perform the mapping > > by running the relevant map_jmf_ code: > > ======================================================================== > > cderx=: 15!:11 > > c_mmap=: 'libc.so.6 mmap64 * * x i i i l'&cd > > c_munmap=: 'libc.so.6 munmap i * x'&cd > > O_RDONLY=: 0 > > PROT_READ=: 1 > > MAP_SHARED=: 1 > > ENOMEM=: 12 > > > > ]ts=: 1!:4 <fn > > 6442451512 > > ]fh=: >0 { c_open fn;O_RDONLY;0 > > 19 > > mh=: ts > > ]fad=: >0{ c_mmap (<0);ts;PROT_READ;MAP_SHARED;fh;0 > > 46913860665344 > > assert ENOMEM ~: >0{cderx'' > > c_munmap (<fad);mh > > +-+----------------+----------+ > > |0|+--------------+|6442451512| > > | ||46913860665344|| | > > | |+--------------+| | > > +-+----------------+----------+ > > c_close fh > > +-+--+ > > |0|19| > > +-+--+ > > ======================================================================== > > > > > > The problem is that the c_mmap call fails when trying with larger > > files, for instance, here is the result of working with an 8GB file: > > ======================================================================== > > SIZE_8GB=: 8 * 1024(<[EMAIL PROTECTED])3 > > > > ]fn=: FN > > /usr/test.jmf > > ]ms=: SIZE_8GB > > 8589934592 > > ]ts=: HS+ms > > 8589935160 > > > > NB. create the file > > ]fh=: 0 pick c_open fn; ((23 b.)/O_RDWR,O_CREAT,O_TRUNC); 8b666 > > 20 > > c_lseek fh;(<:ts);SEEK_SET > > +---+--+----------+-+ > > |567|20|8589935159|0| > > +---+--+----------+-+ > > c_write fh; (,0{a.); 0+1 > > +-+--+-+-+ > > |1|20| |1| > > +-+--+-+-+ > > c_lseek fh;0 ;SEEK_SET > > +-+--+-+-+ > > |0|20|0|0| > > +-+--+-+-+ > > ]d=: HS,AFNJA,ms,JINT,0,0,1,0 NB. integer empty list > > 568 2 8589934592 4 0 0 1 0 > > c_write fh;d;(SZI*#d) > > +--+--+--------------------------+--+ > > |64|20|568 2 8589934592 4 0 0 1 0|64| > > +--+--+--------------------------+--+ > > c_close fh > > +-+--+ > > |0|20| > > +-+--+ > > > > NB. no errors so far, now.try to map the file > > ]ts=: 1!:4 <fn > > 8589935160 > > ]fh=: >0 { c_open fn;O_RDONLY;0 > > 20 > > > > mh=: ts > > ]fad=: >0{ c_mmap (<0);ts;PROT_READ;MAP_SHARED;fh;0 > > _1 > > assert ENOMEM ~: >0{cderx'' > > |assertion failure: assert > > | assert ENOMEM~:>0{cderx'' > > |[-21] > > c_close fh > > +-+--+ > > |0|20| > > +-+--+ > > ======================================================================== > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > -- - michael dykman - [EMAIL PROTECTED] - All models are wrong. Some models are useful. ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
