Re: mremap help ? or no support for FreeBSD ? so do what ?

2000-05-13 Thread nathan

Alfred Perlstein wrote:

 * [EMAIL PROTECTED] [EMAIL PROTECTED] [000513 00:41] wrote:
  Alfred Perlstein wrote:
 
   * [EMAIL PROTECTED] [EMAIL PROTECTED] [000512 21:54] wrote:
I know that this was discussed in the past but I can't find out what to
do ?
   
In Linux if I have to resize a mmap 'ed object I can just use mremap
but in FreeBSD if,  I want to resize it what do I do ?
   
I have tried writing past where I know the end is and it kinda works ?
but why ?
   
Is their a better solution besides just writing to the file and then
calling msync ?
   
Is their new plans to make a mremap call for FreeBSD 4.x ?
  
   no.
  
   
Or am I just  sh%t  out of luck ?
  
   Possibly, but if you describe what you are trying to accomplish
   there may be some advice available.  Your misuse of msync makes me
   think that a rethinking of what you are trying to accomplish may
   be a good idea.  Please explain what makes you need mremap which
   is not portable to any version of unix.  I'm assuming you want your
   app to work on Solaris and other commercial systems.
  
   --
   -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]]
   "I have the heart of a child; I keep it in a jar on my desk."
  
   To Unsubscribe: send mail to [EMAIL PROTECTED]
   with "unsubscribe freebsd-hackers" in the body of the message
 
  Sorry !
 
  Ok here is a snipit of what I am trying to do
 
  fstat(fd, st);
  if( (base = (caddr_t *) mmap(0, st.st_size, ( PROT_READ | PROT_WRITE ) ,
  MAP_SHARED, fd, 0)) == MAP_FAILED)
   {
 MSG_ERR("cannot mmap file, exiting ! ");
 _exit(1);
   }
 
  /* say st.st_size = 200, and I want to add to the end of the file more data
  */
  base += 200; /* we are now at the end */
 
  rec = (stuct rec_t *) base;
  rec-len = 200;
  /* we are writing to the buff, but we are past */
  memmove(rec-data, data, 100);
  msync(base, 0, (st.st_len + rec-len),  MS_ASYNC);
 
  the struct rec_t is an example of a stuct that I want to append to the buf
  and make another reccord.
  If base was a regular pointer we would segfault cause we are past the
  pointers buf
  but  it kinda works.. meaning it will write and add and sometimes and not
  segfault... sometimes...
 
  or I can do this:
 
  rec = (stuct rec_t *) base;
  rec-len = 200;
  memmove(rec-data, data, 100);
  write(fd, (char *) rec, rec-len);
  msync(base, 0, (st.st_len + rec-len),  MS_ASYNC);
 
  After I append to the file and then msync.  I should be able to read the
  reccord that I just added from base.
  base  should reflect the file right ?
 
  I want to make things simple and only append to the buffer "base" and not
  have to re-open the file and write to it then msync it back to mem. I know
  that some of this under FreeBSD "automatic" meaning that the changes made to
  the file  are seen by "base" even though I do not call msync. but in
  order to keep things clear, I will msync.
 
  I hope that this wasn't too confusing... I am not good at explanations or
  spelling

 The first code snippet is completely incorrect, this is _not_ how
 you use the mmap() interface and I'm suprised it works at all.
 Writing after the mmap'd area should cause a SEGV of BUS signal to
 be sent to your process.

 You really haven't provided the code you're using on Linux to use mremap()
 which I'm assuming looks something like:

  rec = (stuct rec_t *) base;
  rec-len = 200;
  memmove(rec-data, data, 100);
  write(fd, (char *) rec, rec-len);
 - mremap(arguments to extend the map);
  msync(base, 0, (st.st_len + rec-len),  MS_ASYNC);

 (derived from the second snippet)

 Now i'm confused, why are you mmap()'ing like this?  I know mremap()
 offers a slightly easier interface, but you could simulate with some
 code like so:

 (you've already mmap()'d a page aligned and multiple of page sized
 region)
 if (!mmap(base + offset, additional length,
 PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, fd, offset)) {

 unmap file and remap it.
 };

 which would tell the OS to map in the end of the file at the end
 of the map you already have.

 problems which can then happen is that if you mmap() multiple files
 without specifying a fixed address then they may be mmap'd into
 your address space right next to each other so that the fixed
 mapping will fail.

 A more flexible way would be to maintain a structure for each file
 that keeps track of all the base and length addresses that you've
 mmap'd in the file.  Then before writing to the memory address you
 can do a quick lookup to figure out where the OS has mapped in that
 particular offset into your address space.  You could use a hash
 on the fd and a binary tree to store that infomation pretty easily.

 Basically implement a simple mmap() manager and you should be set.

 best of luck,
 --
 -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]]
 "I have the heart of a child; I keep it in a jar on my desk."

 To Unsubscribe: send mail to 

Re: mremap help ? or no support for FreeBSD ? so do what ?

2000-05-13 Thread Alfred Perlstein

* [EMAIL PROTECTED] [EMAIL PROTECTED] [000513 01:53] wrote:
 
 When you  wrote
 
 if (!mmap(base + offset, additional length,
 PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, fd, offset)) {
 
 This may sound silly but after this is done then the file will
 reflect the changes?

 and now the file will extend beyond the original size ? why do
 you need to unmap it ? if the file was mmap 'ed with MAP_SHARED
 and not MAP_FIXED wouldn't the changes be made to all objects?
 then you only need to msync back the diff if any ? (or at all)

 See I want to get around using the fd at all. I just want to open
 the file then close it and just reference it from mem only. With
 Linux I think that you can do this by calling mremap  Linux man :
 "mremap expands (or shrinks) an existing memory mapping" So it
 would be kinda like realloc but the changes would be seen by all
 objects.. ? and then I can close the fd and only keep track of
 1 object. If I need to add to it ... mremap on it.
 
 Or am I just way off in my understanding ? I know that I did
 misuse the mmap in the top snipit but I was just playing.

No, you _must_ write() or ftruncate() the file to extend it before
writing to the mmap'd region, whoever made that actually work (if
it even does) under Linux really tossed the whole interface out
which encourages code that will not work on any other operating
system.

In fact the way you are ab^H^Husing mremap() on Linux may wind up
biting you later when they change the interface to be more like
other mmap() interfaces.

-- 
-Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]]
"I have the heart of a child; I keep it in a jar on my desk."


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message



Re: mremap help ? or no support for FreeBSD ? so do what ?

2000-05-13 Thread Arun Sharma

On Sat, May 13, 2000 at 12:08:35PM -0700, Matthew Dillon wrote:
 The linux mremap() is an idiotic system call.  Just unmap the file and
 re-mmap it.

If you are just appending to the file, you can skip the munmap. mmap deletes
the old mappings.

-Arun



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message



Re: mremap help ? or no support for FreeBSD ? so do what ?

2000-05-13 Thread Alfred Perlstein

* [EMAIL PROTECTED] [EMAIL PROTECTED] [000513 00:41] wrote:
 Alfred Perlstein wrote:
 
  * [EMAIL PROTECTED] [EMAIL PROTECTED] [000512 21:54] wrote:
   I know that this was discussed in the past but I can't find out what to
   do ?
  
   In Linux if I have to resize a mmap 'ed object I can just use mremap
   but in FreeBSD if,  I want to resize it what do I do ?
  
   I have tried writing past where I know the end is and it kinda works ?
   but why ?
  
   Is their a better solution besides just writing to the file and then
   calling msync ?
  
   Is their new plans to make a mremap call for FreeBSD 4.x ?
 
  no.
 
  
   Or am I just  sh%t  out of luck ?
 
  Possibly, but if you describe what you are trying to accomplish
  there may be some advice available.  Your misuse of msync makes me
  think that a rethinking of what you are trying to accomplish may
  be a good idea.  Please explain what makes you need mremap which
  is not portable to any version of unix.  I'm assuming you want your
  app to work on Solaris and other commercial systems.
 
  --
  -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]]
  "I have the heart of a child; I keep it in a jar on my desk."
 
  To Unsubscribe: send mail to [EMAIL PROTECTED]
  with "unsubscribe freebsd-hackers" in the body of the message
 
 Sorry !
 
 Ok here is a snipit of what I am trying to do
 
 fstat(fd, st);
 if( (base = (caddr_t *) mmap(0, st.st_size, ( PROT_READ | PROT_WRITE ) ,
 MAP_SHARED, fd, 0)) == MAP_FAILED)
  {
MSG_ERR("cannot mmap file, exiting ! ");
_exit(1);
  }
 
 /* say st.st_size = 200, and I want to add to the end of the file more data
 */
 base += 200; /* we are now at the end */
 
 rec = (stuct rec_t *) base;
 rec-len = 200;
 /* we are writing to the buff, but we are past */
 memmove(rec-data, data, 100);
 msync(base, 0, (st.st_len + rec-len),  MS_ASYNC);
 
 the struct rec_t is an example of a stuct that I want to append to the buf
 and make another reccord.
 If base was a regular pointer we would segfault cause we are past the
 pointers buf
 but  it kinda works.. meaning it will write and add and sometimes and not
 segfault... sometimes...
 
 or I can do this:
 
 rec = (stuct rec_t *) base;
 rec-len = 200;
 memmove(rec-data, data, 100);
 write(fd, (char *) rec, rec-len);
 msync(base, 0, (st.st_len + rec-len),  MS_ASYNC);
 
 After I append to the file and then msync.  I should be able to read the
 reccord that I just added from base.
 base  should reflect the file right ?
 
 I want to make things simple and only append to the buffer "base" and not
 have to re-open the file and write to it then msync it back to mem. I know
 that some of this under FreeBSD "automatic" meaning that the changes made to
 the file  are seen by "base" even though I do not call msync. but in
 order to keep things clear, I will msync.
 
 I hope that this wasn't too confusing... I am not good at explanations or
 spelling

The first code snippet is completely incorrect, this is _not_ how
you use the mmap() interface and I'm suprised it works at all.
Writing after the mmap'd area should cause a SEGV of BUS signal to
be sent to your process.

You really haven't provided the code you're using on Linux to use mremap()
which I'm assuming looks something like:

 rec = (stuct rec_t *) base;
 rec-len = 200;
 memmove(rec-data, data, 100);
 write(fd, (char *) rec, rec-len);
- mremap(arguments to extend the map);
 msync(base, 0, (st.st_len + rec-len),  MS_ASYNC);

(derived from the second snippet)

Now i'm confused, why are you mmap()'ing like this?  I know mremap()
offers a slightly easier interface, but you could simulate with some
code like so:

(you've already mmap()'d a page aligned and multiple of page sized
region)
if (!mmap(base + offset, additional length,
PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, fd, offset)) {

unmap file and remap it.
};

which would tell the OS to map in the end of the file at the end
of the map you already have.

problems which can then happen is that if you mmap() multiple files
without specifying a fixed address then they may be mmap'd into
your address space right next to each other so that the fixed
mapping will fail.

A more flexible way would be to maintain a structure for each file
that keeps track of all the base and length addresses that you've
mmap'd in the file.  Then before writing to the memory address you
can do a quick lookup to figure out where the OS has mapped in that
particular offset into your address space.  You could use a hash
on the fd and a binary tree to store that infomation pretty easily.

Basically implement a simple mmap() manager and you should be set.

best of luck,
-- 
-Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]]
"I have the heart of a child; I keep it in a jar on my desk."


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message



Re: mremap help ? or no support for FreeBSD ? so do what ?

2000-05-13 Thread Matthew Dillon


:In muc.lists.freebsd.hackers, you wrote:
: I know that this was discussed in the past but I can't find out what to
: do ?
: 
: In Linux if I have to resize a mmap 'ed object I can just use mremap
: but in FreeBSD if,  I want to resize it what do I do ?
:
:Have you tried mmap'ing the file again at the same address, but with a
:larger length ?
:
:As I see it, the only advantage of using mmap instead of read/write interface
:is to avoid the extra system calls. However, by calling mremap or mmap
:repeatedly, you're losing that advantage. I think you're better off by
:extending your map in larger chunks, reducing the number of mmap/mremap 
:calls. If you have a big enough address space, just use a large mmap region
:to begin with.
:
:   -Arun

The linux mremap() is an idiotic system call.  Just unmap the file and
re-mmap it.

-Matt
Matthew Dillon 
[EMAIL PROTECTED]


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message



Re: mremap help ? or no support for FreeBSD ? so do what ?

2000-05-13 Thread Matthew Dillon


:On Sat, May 13, 2000 at 12:08:35PM -0700, Matthew Dillon wrote:
: The linux mremap() is an idiotic system call.  Just unmap the file and
: re-mmap it.
:
:If you are just appending to the file, you can skip the munmap. mmap deletes
:the old mappings.
:
:   -Arun

There are a thousand ways to do it, which is why linux's mremap() 
syscall is stupid.

* simply mmap() a larger block in the first place.  For example,
  if you have a 16K file mmap() 1MB.  You will seg fault on pages
  that are beyond the file EOF, but those pages will become valid
  the moment the file is extended into them without having to lift
  a finger.

* mmap() the tail end of the newly extended file without removing or
  overwriting the previous mmap, by specifying an absolute address.

* munmap() and re-mmap() the file.

* Don't depend on a single monolithic mmap(), it won't work for files
  larger then 2-3GB anyway (on intel architecture), instead mmap the
  file in chunks on an as-needed basis.

-Matt
Matthew Dillon 
[EMAIL PROTECTED]


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message



Re: mremap help ? or no support for FreeBSD ? so do what ?

2000-05-13 Thread nathan

Alfred Perlstein wrote:

 * [EMAIL PROTECTED] [EMAIL PROTECTED] [000512 21:54] wrote:
  I know that this was discussed in the past but I can't find out what to
  do ?
 
  In Linux if I have to resize a mmap 'ed object I can just use mremap
  but in FreeBSD if,  I want to resize it what do I do ?
 
  I have tried writing past where I know the end is and it kinda works ?
  but why ?
 
  Is their a better solution besides just writing to the file and then
  calling msync ?
 
  Is their new plans to make a mremap call for FreeBSD 4.x ?

 no.

 
  Or am I just  sh%t  out of luck ?

 Possibly, but if you describe what you are trying to accomplish
 there may be some advice available.  Your misuse of msync makes me
 think that a rethinking of what you are trying to accomplish may
 be a good idea.  Please explain what makes you need mremap which
 is not portable to any version of unix.  I'm assuming you want your
 app to work on Solaris and other commercial systems.

 --
 -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]]
 "I have the heart of a child; I keep it in a jar on my desk."

 To Unsubscribe: send mail to [EMAIL PROTECTED]
 with "unsubscribe freebsd-hackers" in the body of the message

Sorry !

Ok here is a snipit of what I am trying to do

fstat(fd, st);
if( (base = (caddr_t *) mmap(0, st.st_size, ( PROT_READ | PROT_WRITE ) ,
MAP_SHARED, fd, 0)) == MAP_FAILED)
 {
   MSG_ERR("cannot mmap file, exiting ! ");
   _exit(1);
 }

/* say st.st_size = 200, and I want to add to the end of the file more data
*/
base += 200; /* we are now at the end */

rec = (stuct rec_t *) base;
rec-len = 200;
/* we are writing to the buff, but we are past */
memmove(rec-data, data, 100);
msync(base, 0, (st.st_len + rec-len),  MS_ASYNC);

the struct rec_t is an example of a stuct that I want to append to the buf
and make another reccord.
If base was a regular pointer we would segfault cause we are past the
pointers buf
but  it kinda works.. meaning it will write and add and sometimes and not
segfault... sometimes...

or I can do this:

rec = (stuct rec_t *) base;
rec-len = 200;
memmove(rec-data, data, 100);
write(fd, (char *) rec, rec-len);
msync(base, 0, (st.st_len + rec-len),  MS_ASYNC);

After I append to the file and then msync.  I should be able to read the
reccord that I just added from base.
base  should reflect the file right ?

I want to make things simple and only append to the buffer "base" and not
have to re-open the file and write to it then msync it back to mem. I know
that some of this under FreeBSD "automatic" meaning that the changes made to
the file  are seen by "base" even though I do not call msync. but in
order to keep things clear, I will msync.

I hope that this wasn't too confusing... I am not good at explanations or
spelling

thank you

nathan



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message



Re: mremap help ? or no support for FreeBSD ? so do what ?

2000-05-13 Thread Alfred Perlstein

* [EMAIL PROTECTED] [EMAIL PROTECTED] [000512 21:54] wrote:
 I know that this was discussed in the past but I can't find out what to
 do ?
 
 In Linux if I have to resize a mmap 'ed object I can just use mremap
 but in FreeBSD if,  I want to resize it what do I do ?
 
 I have tried writing past where I know the end is and it kinda works ?
 but why ?
 
 Is their a better solution besides just writing to the file and then
 calling msync ?
 
 Is their new plans to make a mremap call for FreeBSD 4.x ?

no.

 
 Or am I just  sh%t  out of luck ?

Possibly, but if you describe what you are trying to accomplish
there may be some advice available.  Your misuse of msync makes me
think that a rethinking of what you are trying to accomplish may
be a good idea.  Please explain what makes you need mremap which
is not portable to any version of unix.  I'm assuming you want your
app to work on Solaris and other commercial systems.

-- 
-Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]]
"I have the heart of a child; I keep it in a jar on my desk."


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message