This is a very important matter for us, I already searched this topic
in many sites but there is no mention of this mmap limitation on
Linux. Any help would be very appreciated.
On Mon, Mar 31, 2008 at 12:05 PM, Jorge Arredondo
<[EMAIL PROTECTED]> wrote:
> Thank you Eric.
>
> I was not sure where to post the message, sorry for using the wrong forum.
>
> Here is the code I added to make the windows version work:
> ===
> K32=: 2^32
> K31=: 2^31
> MAXLONG32=: <._1+K31
>
> long64to32=: 3 : 0
> y=. (2$K32)#:y
> ({.;(}.))|.<.((y>K31)*-K32)+y
> )
> long32to64=: <.@(K32&#.)@|:@(K32&|)@;@|.
>
> SetFilePointerDLLCALL=: >@{.@('kernel32 SetFilePointer i i i *i i'&(15!:0))
> GetFileSizeDLLCALL=: >@{.@('kernel32 GetFileSize i i *i'&(15!:0))
>
> SetFilePointerR=: verb define
> 'hFile lDistanceToMove lpDistanceToMoveHigh dwMoveMethod'=. y
> if. (lpDistanceToMoveHigh-:NULLPTR) *. (lDistanceToMove>MAXLONG32) do.
> 'lDistanceToMove lpDistanceToMoveHigh'=. long64to32
> lDistanceToMove
> end.
> SetFilePointerDLLCALL
> hFile;lDistanceToMove;lpDistanceToMoveHigh;dwMoveMethod
> )
>
> GetFileSizeR=: verb define
> hFile=. >{.y
> lpFileSizeHigh=. ,2
> lpFileSize=. GetFileSizeDLLCALL hFile;lpFileSizeHigh
> long32to64 lpFileSize,lpFileSizeHigh
> )
> ===
> I already have searched for Linux discussions about this topic but
> haven't found any solution so far.
>
>
>
>
> On Mon, Mar 31, 2008 at 8:06 AM, Eric Iverson <[EMAIL PROTECTED]> wrote:
> > I am interested in the problem you report but haven't had a chance to work
> > on it yet. With luck I will be able to do so in a few days.
> >
> > Thanks for taking the initial steps of fixing the SetFilePointerR vs
> 64bit
> > etc problems and we'll see that this work gets included in a future J
> libary
> > release.
> >
> > The first thing I wouold do is a web search on Linux and large mapped
> files
> > and wonder if you have already done that.
> >
> > (I replied to your original message to programming and not the one to chat
> > as this is quite properly a programming issue)
> >
> >
> >
> > ----- Original Message -----
> > From: "Jorge Antonio Arredondo Vargas" <[EMAIL PROTECTED]>
> > To: <[email protected]>
> > Sent: Thursday, March 27, 2008 11:36 PM
> > Subject: [Jprogramming] Memory-map a large file on Linux
> >
> >
> > > I'm trying to memory-map a large file (more than 6 GB) from J64 on
> > > Linux, but I'm getting a ENOMEM error which I haven't been able to
> > > solve.
> > > I can successfully map such kind of file, (indeed bigger: 200GB) on
> > > Windows XP x64 and use it as a J noun, however.
> > >
> > > Have any of you done this already?
> > >
> > > My system has the following specifications:
> > > OS= OpenSuse 10.2
> > > RAM= 8GB
> > > CPU= 2 Dual Core Opteron64 @2GHz
> > > 9!:12''
> > > 5
> > > 9!:14''
> > > j601/2006-09-04/09:20
> > >
> > > I start getting problems after testing with files bigger than 6GB,
> > > which is close to the physical amount of RAM in my machine, so I'm
> > > thinking the mmap system call might be trying to actually load the
> > > file into memory, perhaps this problem can be solved by using a
> > > different flag on the mmap call.
> > >
> > >
> > > When I first tried to create the file I want to map, I discovered
> > > jmf.ijs didn't worked with big files:
> > > ========================================================================
> > > require 'jmf files'
> > > FN=: '/usr/test.jmf'
> > > SIZE_6GB=: 6 * 1024(<[EMAIL PROTECTED])3
> > > HS=: 568
> > >
> > > createjmf_jmf_ FN;SIZE_6GB
> > > assert (SIZE_6GB+HS)-:fsize FN
> > > |assertion failure: assert
> > > | assert (SIZE_6GB+HS)-:fsize FN
> > > |[-3]
> > > fsize FN
> > > 64
> > > ========================================================================
> > >
> > >
> > > I solved this by defining c_open and c_lseek with large file support
> > > (LFS) function equivalents and then running the relevant code of
> > > createjmf_jmf_, here is the code:
> > > ========================================================================
> > > cd=: 15!:0
> > > c_open=: 'libc.so.6 open64 i *c i i'&cd
> > > c_close=: 'libc.so.6 close i i'&cd
> > > c_lseek=: 'libc.so.6 lseek64 i i l i'&cd
> > > c_write=: 'libc.so.6 write i i * i'&cd
> > > O_RDWR=: 2
> > > O_CREAT=: 64
> > > O_TRUNC=: 512
> > > JINT=: 4
> > > SZI=: 8
> > > AFNJA=: 2
> > > SEEK_SET=: 0
> > >
> > > ]fn=: FN
> > > /usr/test.jmf
> > > ]ms=: SIZE_6GB
> > > 6442450944
> > > ]ts=: HS+ms
> > > 6442451512
> > >
> > > ]fh=: 0 pick c_open fn; ((23 b.)/O_RDWR,O_CREAT,O_TRUNC); 8b666
> > > 19
> > > c_lseek fh;(<:ts);SEEK_SET
> > > +-----------+--+----------+-+
> > > |_2147483081|19|6442451511|0|
> > > +-----------+--+----------+-+
> > > c_write fh; (,0{a.); 0+1
> > > +-+--+-+-+
> > > |1|19| |1|
> > > +-+--+-+-+
> > > c_lseek fh;0 ;SEEK_SET
> > > +-+--+-+-+
> > > |0|19|0|0|
> > > +-+--+-+-+
> > > ]d=: HS,AFNJA,ms,JINT,0,0,1,0 NB. integer empty list
> > > 568 2 6442450944 4 0 0 1 0
> > > c_write fh;d;(SZI*#d)
> > > +--+--+--------------------------+--+
> > > |64|19|568 2 6442450944 4 0 0 1 0|64|
> > > +--+--+--------------------------+--+
> > > c_close fh
> > > +-+--+
> > > |0|19|
> > > +-+--+
> > > assert (SIZE_6GB+HS)-:fsize FN
> > > ========================================================================
> > >
> > >
> > > After creating the file, I tried using map_jmf_, but J crashed with a
> > > segmentation fault:
> > > ========================================================================
> > > map_jmf_ 'test';FN
> > > 9183 Segmentation fault
> > > ========================================================================
> > >
> > >
> > > So I defined c_mmap with LFS, and I could finally perform the mapping
> > > by running the relevant map_jmf_ code:
> > > ========================================================================
> > > cderx=: 15!:11
> > > c_mmap=: 'libc.so.6 mmap64 * * x i i i l'&cd
> > > c_munmap=: 'libc.so.6 munmap i * x'&cd
> > > O_RDONLY=: 0
> > > PROT_READ=: 1
> > > MAP_SHARED=: 1
> > > ENOMEM=: 12
> > >
> > > ]ts=: 1!:4 <fn
> > > 6442451512
> > > ]fh=: >0 { c_open fn;O_RDONLY;0
> > > 19
> > > mh=: ts
> > > ]fad=: >0{ c_mmap (<0);ts;PROT_READ;MAP_SHARED;fh;0
> > > 46913860665344
> > > assert ENOMEM ~: >0{cderx''
> > > c_munmap (<fad);mh
> > > +-+----------------+----------+
> > > |0|+--------------+|6442451512|
> > > | ||46913860665344|| |
> > > | |+--------------+| |
> > > +-+----------------+----------+
> > > c_close fh
> > > +-+--+
> > > |0|19|
> > > +-+--+
> > > ========================================================================
> > >
> > >
> > > The problem is that the c_mmap call fails when trying with larger
> > > files, for instance, here is the result of working with an 8GB file:
> > > ========================================================================
> > > SIZE_8GB=: 8 * 1024(<[EMAIL PROTECTED])3
> > >
> > > ]fn=: FN
> > > /usr/test.jmf
> > > ]ms=: SIZE_8GB
> > > 8589934592
> > > ]ts=: HS+ms
> > > 8589935160
> > >
> > > NB. create the file
> > > ]fh=: 0 pick c_open fn; ((23 b.)/O_RDWR,O_CREAT,O_TRUNC); 8b666
> > > 20
> > > c_lseek fh;(<:ts);SEEK_SET
> > > +---+--+----------+-+
> > > |567|20|8589935159|0|
> > > +---+--+----------+-+
> > > c_write fh; (,0{a.); 0+1
> > > +-+--+-+-+
> > > |1|20| |1|
> > > +-+--+-+-+
> > > c_lseek fh;0 ;SEEK_SET
> > > +-+--+-+-+
> > > |0|20|0|0|
> > > +-+--+-+-+
> > > ]d=: HS,AFNJA,ms,JINT,0,0,1,0 NB. integer empty list
> > > 568 2 8589934592 4 0 0 1 0
> > > c_write fh;d;(SZI*#d)
> > > +--+--+--------------------------+--+
> > > |64|20|568 2 8589934592 4 0 0 1 0|64|
> > > +--+--+--------------------------+--+
> > > c_close fh
> > > +-+--+
> > > |0|20|
> > > +-+--+
> > >
> > > NB. no errors so far, now.try to map the file
> > > ]ts=: 1!:4 <fn
> > > 8589935160
> > > ]fh=: >0 { c_open fn;O_RDONLY;0
> > > 20
> > >
> > > mh=: ts
> > > ]fad=: >0{ c_mmap (<0);ts;PROT_READ;MAP_SHARED;fh;0
> > > _1
> > > assert ENOMEM ~: >0{cderx''
> > > |assertion failure: assert
> > > | assert ENOMEM~:>0{cderx''
> > > |[-21]
> > > c_close fh
> > > +-+--+
> > > |0|20|
> > > +-+--+
> > > ========================================================================
> > > ----------------------------------------------------------------------
> > > For information about J forums see http://www.jsoftware.com/forums.htm
> > ----------------------------------------------------------------------
> > For information about J forums see http://www.jsoftware.com/forums.htm
> >
>
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
