Thank you Eric.
I was not sure where to post the message, sorry for using the wrong forum.
Here is the code I added to make the windows version work:
===
K32=: 2^32
K31=: 2^31
MAXLONG32=: <._1+K31
long64to32=: 3 : 0
y=. (2$K32)#:y
({.;(}.))|.<.((y>K31)*-K32)+y
)
long32to64=: <.@(K32&#.)@|:@(K32&|)@;@|.
SetFilePointerDLLCALL=: >@{.@('kernel32 SetFilePointer i i i *i i'&(15!:0))
GetFileSizeDLLCALL=: >@{.@('kernel32 GetFileSize i i *i'&(15!:0))
SetFilePointerR=: verb define
'hFile lDistanceToMove lpDistanceToMoveHigh dwMoveMethod'=. y
if. (lpDistanceToMoveHigh-:NULLPTR) *. (lDistanceToMove>MAXLONG32) do.
'lDistanceToMove lpDistanceToMoveHigh'=. long64to32
lDistanceToMove
end.
SetFilePointerDLLCALL
hFile;lDistanceToMove;lpDistanceToMoveHigh;dwMoveMethod
)
GetFileSizeR=: verb define
hFile=. >{.y
lpFileSizeHigh=. ,2
lpFileSize=. GetFileSizeDLLCALL hFile;lpFileSizeHigh
long32to64 lpFileSize,lpFileSizeHigh
)
===
I already have searched for Linux discussions about this topic but
haven't found any solution so far.
On Mon, Mar 31, 2008 at 8:06 AM, Eric Iverson <[EMAIL PROTECTED]> wrote:
> I am interested in the problem you report but haven't had a chance to work
> on it yet. With luck I will be able to do so in a few days.
>
> Thanks for taking the initial steps of fixing the SetFilePointerR vs 64bit
> etc problems and we'll see that this work gets included in a future J libary
> release.
>
> The first thing I wouold do is a web search on Linux and large mapped files
> and wonder if you have already done that.
>
> (I replied to your original message to programming and not the one to chat
> as this is quite properly a programming issue)
>
>
>
> ----- Original Message -----
> From: "Jorge Antonio Arredondo Vargas" <[EMAIL PROTECTED]>
> To: <[email protected]>
> Sent: Thursday, March 27, 2008 11:36 PM
> Subject: [Jprogramming] Memory-map a large file on Linux
>
>
> > I'm trying to memory-map a large file (more than 6 GB) from J64 on
> > Linux, but I'm getting a ENOMEM error which I haven't been able to
> > solve.
> > I can successfully map such kind of file, (indeed bigger: 200GB) on
> > Windows XP x64 and use it as a J noun, however.
> >
> > Have any of you done this already?
> >
> > My system has the following specifications:
> > OS= OpenSuse 10.2
> > RAM= 8GB
> > CPU= 2 Dual Core Opteron64 @2GHz
> > 9!:12''
> > 5
> > 9!:14''
> > j601/2006-09-04/09:20
> >
> > I start getting problems after testing with files bigger than 6GB,
> > which is close to the physical amount of RAM in my machine, so I'm
> > thinking the mmap system call might be trying to actually load the
> > file into memory, perhaps this problem can be solved by using a
> > different flag on the mmap call.
> >
> >
> > When I first tried to create the file I want to map, I discovered
> > jmf.ijs didn't worked with big files:
> > ========================================================================
> > require 'jmf files'
> > FN=: '/usr/test.jmf'
> > SIZE_6GB=: 6 * 1024(<[EMAIL PROTECTED])3
> > HS=: 568
> >
> > createjmf_jmf_ FN;SIZE_6GB
> > assert (SIZE_6GB+HS)-:fsize FN
> > |assertion failure: assert
> > | assert (SIZE_6GB+HS)-:fsize FN
> > |[-3]
> > fsize FN
> > 64
> > ========================================================================
> >
> >
> > I solved this by defining c_open and c_lseek with large file support
> > (LFS) function equivalents and then running the relevant code of
> > createjmf_jmf_, here is the code:
> > ========================================================================
> > cd=: 15!:0
> > c_open=: 'libc.so.6 open64 i *c i i'&cd
> > c_close=: 'libc.so.6 close i i'&cd
> > c_lseek=: 'libc.so.6 lseek64 i i l i'&cd
> > c_write=: 'libc.so.6 write i i * i'&cd
> > O_RDWR=: 2
> > O_CREAT=: 64
> > O_TRUNC=: 512
> > JINT=: 4
> > SZI=: 8
> > AFNJA=: 2
> > SEEK_SET=: 0
> >
> > ]fn=: FN
> > /usr/test.jmf
> > ]ms=: SIZE_6GB
> > 6442450944
> > ]ts=: HS+ms
> > 6442451512
> >
> > ]fh=: 0 pick c_open fn; ((23 b.)/O_RDWR,O_CREAT,O_TRUNC); 8b666
> > 19
> > c_lseek fh;(<:ts);SEEK_SET
> > +-----------+--+----------+-+
> > |_2147483081|19|6442451511|0|
> > +-----------+--+----------+-+
> > c_write fh; (,0{a.); 0+1
> > +-+--+-+-+
> > |1|19| |1|
> > +-+--+-+-+
> > c_lseek fh;0 ;SEEK_SET
> > +-+--+-+-+
> > |0|19|0|0|
> > +-+--+-+-+
> > ]d=: HS,AFNJA,ms,JINT,0,0,1,0 NB. integer empty list
> > 568 2 6442450944 4 0 0 1 0
> > c_write fh;d;(SZI*#d)
> > +--+--+--------------------------+--+
> > |64|19|568 2 6442450944 4 0 0 1 0|64|
> > +--+--+--------------------------+--+
> > c_close fh
> > +-+--+
> > |0|19|
> > +-+--+
> > assert (SIZE_6GB+HS)-:fsize FN
> > ========================================================================
> >
> >
> > After creating the file, I tried using map_jmf_, but J crashed with a
> > segmentation fault:
> > ========================================================================
> > map_jmf_ 'test';FN
> > 9183 Segmentation fault
> > ========================================================================
> >
> >
> > So I defined c_mmap with LFS, and I could finally perform the mapping
> > by running the relevant map_jmf_ code:
> > ========================================================================
> > cderx=: 15!:11
> > c_mmap=: 'libc.so.6 mmap64 * * x i i i l'&cd
> > c_munmap=: 'libc.so.6 munmap i * x'&cd
> > O_RDONLY=: 0
> > PROT_READ=: 1
> > MAP_SHARED=: 1
> > ENOMEM=: 12
> >
> > ]ts=: 1!:4 <fn
> > 6442451512
> > ]fh=: >0 { c_open fn;O_RDONLY;0
> > 19
> > mh=: ts
> > ]fad=: >0{ c_mmap (<0);ts;PROT_READ;MAP_SHARED;fh;0
> > 46913860665344
> > assert ENOMEM ~: >0{cderx''
> > c_munmap (<fad);mh
> > +-+----------------+----------+
> > |0|+--------------+|6442451512|
> > | ||46913860665344|| |
> > | |+--------------+| |
> > +-+----------------+----------+
> > c_close fh
> > +-+--+
> > |0|19|
> > +-+--+
> > ========================================================================
> >
> >
> > The problem is that the c_mmap call fails when trying with larger
> > files, for instance, here is the result of working with an 8GB file:
> > ========================================================================
> > SIZE_8GB=: 8 * 1024(<[EMAIL PROTECTED])3
> >
> > ]fn=: FN
> > /usr/test.jmf
> > ]ms=: SIZE_8GB
> > 8589934592
> > ]ts=: HS+ms
> > 8589935160
> >
> > NB. create the file
> > ]fh=: 0 pick c_open fn; ((23 b.)/O_RDWR,O_CREAT,O_TRUNC); 8b666
> > 20
> > c_lseek fh;(<:ts);SEEK_SET
> > +---+--+----------+-+
> > |567|20|8589935159|0|
> > +---+--+----------+-+
> > c_write fh; (,0{a.); 0+1
> > +-+--+-+-+
> > |1|20| |1|
> > +-+--+-+-+
> > c_lseek fh;0 ;SEEK_SET
> > +-+--+-+-+
> > |0|20|0|0|
> > +-+--+-+-+
> > ]d=: HS,AFNJA,ms,JINT,0,0,1,0 NB. integer empty list
> > 568 2 8589934592 4 0 0 1 0
> > c_write fh;d;(SZI*#d)
> > +--+--+--------------------------+--+
> > |64|20|568 2 8589934592 4 0 0 1 0|64|
> > +--+--+--------------------------+--+
> > c_close fh
> > +-+--+
> > |0|20|
> > +-+--+
> >
> > NB. no errors so far, now.try to map the file
> > ]ts=: 1!:4 <fn
> > 8589935160
> > ]fh=: >0 { c_open fn;O_RDONLY;0
> > 20
> >
> > mh=: ts
> > ]fad=: >0{ c_mmap (<0);ts;PROT_READ;MAP_SHARED;fh;0
> > _1
> > assert ENOMEM ~: >0{cderx''
> > |assertion failure: assert
> > | assert ENOMEM~:>0{cderx''
> > |[-21]
> > c_close fh
> > +-+--+
> > |0|20|
> > +-+--+
> > ========================================================================
> > ----------------------------------------------------------------------
> > For information about J forums see http://www.jsoftware.com/forums.htm
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
>
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
