Roger,
I'm trying to memory-map a large file (more than 6 GB) from J64 on
Linux, but I'm getting a ENOMEM error which I haven't been able to
solve.
After modifying SetFilePointerR and GetFileSizeR definitions in
jmf.ijs, I was able to successfully map such kind of file, (indeed
bigger: 200GB) on Windows XP x64 and use it as a J noun.
My system has the following specifications:
OS= OpenSuse 10.2
RAM= 8GB
CPU= 2 Dual Core Opteron64 @2GHz
9!:12''
5
9!:14''
j601/2006-09-04/09:20
I start getting problems after testing with files bigger than 6GB,
which is close to the physical amount of RAM in my machine, so I'm
thinking the mmap system call might be trying to actually load the
file into memory, perhaps this problem can be solved by using a
different flag on the mmap call.
When I first tried to create the file I want to map, I discovered
jmf.ijs didn't worked with big files:
========================================================================
require 'jmf files'
FN=: '/usr/test.jmf'
SIZE_6GB=: 6 * 1024(<[EMAIL PROTECTED])3
HS=: 568
createjmf_jmf_ FN;SIZE_6GB
assert (SIZE_6GB+HS)-:fsize FN
|assertion failure: assert
| assert (SIZE_6GB+HS)-:fsize FN
|[-3]
fsize FN
64
========================================================================
I solved this by defining c_open and c_lseek with large file support
(LFS) function equivalents and then running the relevant code of
createjmf_jmf_, here is the code:
========================================================================
cd=: 15!:0
c_open=: 'libc.so.6 open64 i *c i i'&cd
c_close=: 'libc.so.6 close i i'&cd
c_lseek=: 'libc.so.6 lseek64 i i l i'&cd
c_write=: 'libc.so.6 write i i * i'&cd
O_RDWR=: 2
O_CREAT=: 64
O_TRUNC=: 512
JINT=: 4
SZI=: 8
AFNJA=: 2
SEEK_SET=: 0
]fn=: FN
/usr/test.jmf
]ms=: SIZE_6GB
6442450944
]ts=: HS+ms
6442451512
]fh=: 0 pick c_open fn; ((23 b.)/O_RDWR,O_CREAT,O_TRUNC); 8b666
19
c_lseek fh;(<:ts);SEEK_SET
+-----------+--+----------+-+
|_2147483081|19|6442451511|0|
+-----------+--+----------+-+
c_write fh; (,0{a.); 0+1
+-+--+-+-+
|1|19| |1|
+-+--+-+-+
c_lseek fh;0 ;SEEK_SET
+-+--+-+-+
|0|19|0|0|
+-+--+-+-+
]d=: HS,AFNJA,ms,JINT,0,0,1,0 NB. integer empty list
568 2 6442450944 4 0 0 1 0
c_write fh;d;(SZI*#d)
+--+--+--------------------------+--+
|64|19|568 2 6442450944 4 0 0 1 0|64|
+--+--+--------------------------+--+
c_close fh
+-+--+
|0|19|
+-+--+
assert (SIZE_6GB+HS)-:fsize FN
========================================================================
After creating the file, I tried using map_jmf_, but J crashed with a
segmentation fault:
========================================================================
map_jmf_ 'test';FN
9183 Segmentation fault
========================================================================
So I defined c_mmap with LFS, and I could finally perform the mapping
by running the relevant map_jmf_ code:
========================================================================
cderx=: 15!:11
c_mmap=: 'libc.so.6 mmap64 * * x i i i l'&cd
c_munmap=: 'libc.so.6 munmap i * x'&cd
O_RDONLY=: 0
PROT_READ=: 1
MAP_SHARED=: 1
ENOMEM=: 12
]ts=: 1!:4 <fn
6442451512
]fh=: >0 { c_open fn;O_RDONLY;0
19
mh=: ts
]fad=: >0{ c_mmap (<0);ts;PROT_READ;MAP_SHARED;fh;0
46913860665344
assert ENOMEM ~: >0{cderx''
c_munmap (<fad);mh
+-+----------------+----------+
|0|+--------------+|6442451512|
| ||46913860665344|| |
| |+--------------+| |
+-+----------------+----------+
c_close fh
+-+--+
|0|19|
+-+--+
========================================================================
The problem is that the c_mmap call fails when trying with larger
files, for instance, here is the result of working with an 8GB file:
========================================================================
SIZE_8GB=: 8 * 1024(<[EMAIL PROTECTED])3
]fn=: FN
/usr/test.jmf
]ms=: SIZE_8GB
8589934592
]ts=: HS+ms
8589935160
NB. create the file
]fh=: 0 pick c_open fn; ((23 b.)/O_RDWR,O_CREAT,O_TRUNC); 8b666
20
c_lseek fh;(<:ts);SEEK_SET
+---+--+----------+-+
|567|20|8589935159|0|
+---+--+----------+-+
c_write fh; (,0{a.); 0+1
+-+--+-+-+
|1|20| |1|
+-+--+-+-+
c_lseek fh;0 ;SEEK_SET
+-+--+-+-+
|0|20|0|0|
+-+--+-+-+
]d=: HS,AFNJA,ms,JINT,0,0,1,0 NB. integer empty list
568 2 8589934592 4 0 0 1 0
c_write fh;d;(SZI*#d)
+--+--+--------------------------+--+
|64|20|568 2 8589934592 4 0 0 1 0|64|
+--+--+--------------------------+--+
c_close fh
+-+--+
|0|20|
+-+--+
NB. no errors so far, now.try to map the file
]ts=: 1!:4 <fn
8589935160
]fh=: >0 { c_open fn;O_RDONLY;0
20
mh=: ts
]fad=: >0{ c_mmap (<0);ts;PROT_READ;MAP_SHARED;fh;0
_1
assert ENOMEM ~: >0{cderx''
|assertion failure: assert
| assert ENOMEM~:>0{cderx''
|[-21]
c_close fh
+-+--+
|0|20|
+-+--+
========================================================================
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
