Re: mremap help ? or no support for FreeBSD ? so do what ?
:On Sat, May 13, 2000 at 12:08:35PM -0700, Matthew Dillon wrote: :> The linux mremap() is an idiotic system call. Just unmap the file and :> re-mmap it. : :If you are just appending to the file, you can skip the munmap. mmap deletes :the old mappings. : : -Arun There are a thousand ways to do it, which is why linux's mremap() syscall is stupid. * simply mmap() a larger block in the first place. For example, if you have a 16K file mmap() 1MB. You will seg fault on pages that are beyond the file EOF, but those pages will become valid the moment the file is extended into them without having to lift a finger. * mmap() the tail end of the newly extended file without removing or overwriting the previous mmap, by specifying an absolute address. * munmap() and re-mmap() the file. * Don't depend on a single monolithic mmap(), it won't work for files larger then 2-3GB anyway (on intel architecture), instead mmap the file in chunks on an as-needed basis. -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mremap help ? or no support for FreeBSD ? so do what ?
On Sat, May 13, 2000 at 12:08:35PM -0700, Matthew Dillon wrote: > The linux mremap() is an idiotic system call. Just unmap the file and > re-mmap it. If you are just appending to the file, you can skip the munmap. mmap deletes the old mappings. -Arun To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mremap help ? or no support for FreeBSD ? so do what ?
:In muc.lists.freebsd.hackers, you wrote: :> I know that this was discussed in the past but I can't find out what to :> do ? :> :> In Linux if I have to resize a mmap 'ed object I can just use mremap :> but in FreeBSD if, I want to resize it what do I do ? : :Have you tried mmap'ing the file again at the same address, but with a :larger length ? : :As I see it, the only advantage of using mmap instead of read/write interface :is to avoid the extra system calls. However, by calling mremap or mmap :repeatedly, you're losing that advantage. I think you're better off by :extending your map in larger chunks, reducing the number of mmap/mremap :calls. If you have a big enough address space, just use a large mmap region :to begin with. : : -Arun The linux mremap() is an idiotic system call. Just unmap the file and re-mmap it. -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mremap help ? or no support for FreeBSD ? so do what ?
In muc.lists.freebsd.hackers, you wrote: > I know that this was discussed in the past but I can't find out what to > do ? > > In Linux if I have to resize a mmap 'ed object I can just use mremap > but in FreeBSD if, I want to resize it what do I do ? Have you tried mmap'ing the file again at the same address, but with a larger length ? As I see it, the only advantage of using mmap instead of read/write interface is to avoid the extra system calls. However, by calling mremap or mmap repeatedly, you're losing that advantage. I think you're better off by extending your map in larger chunks, reducing the number of mmap/mremap calls. If you have a big enough address space, just use a large mmap region to begin with. -Arun To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mremap help ? or no support for FreeBSD ? so do what ?
* [EMAIL PROTECTED] <[EMAIL PROTECTED]> [000513 01:53] wrote: > > When you wrote > > if (!mmap(base + offset, additional length, > PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, fd, offset)) { > > This may sound silly but after this is done then the file will > reflect the changes? > > and now the file will extend beyond the original size ? why do > you need to unmap it ? if the file was mmap 'ed with MAP_SHARED > and not MAP_FIXED wouldn't the changes be made to all objects? > then you only need to msync back the diff if any ? (or at all) > > See I want to get around using the fd at all. I just want to open > the file then close it and just reference it from mem only. With > Linux I think that you can do this by calling mremap Linux man : > "mremap expands (or shrinks) an existing memory mapping" So it > would be kinda like realloc but the changes would be seen by all > objects.. ? and then I can close the fd and only keep track of > 1 object. If I need to add to it ... mremap on it. > > Or am I just way off in my understanding ? I know that I did > misuse the mmap in the top snipit but I was just playing. No, you _must_ write() or ftruncate() the file to extend it before writing to the mmap'd region, whoever made that actually work (if it even does) under Linux really tossed the whole interface out which encourages code that will not work on any other operating system. In fact the way you are ab^H^Husing mremap() on Linux may wind up biting you later when they change the interface to be more like other mmap() interfaces. -- -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] "I have the heart of a child; I keep it in a jar on my desk." To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mremap help ? or no support for FreeBSD ? so do what ?
Alfred Perlstein wrote: > * [EMAIL PROTECTED] <[EMAIL PROTECTED]> [000513 00:41] wrote: > > Alfred Perlstein wrote: > > > > > * [EMAIL PROTECTED] <[EMAIL PROTECTED]> [000512 21:54] wrote: > > > > I know that this was discussed in the past but I can't find out what to > > > > do ? > > > > > > > > In Linux if I have to resize a mmap 'ed object I can just use mremap > > > > but in FreeBSD if, I want to resize it what do I do ? > > > > > > > > I have tried writing past where I know the end is and it kinda works ? > > > > but why ? > > > > > > > > Is their a better solution besides just writing to the file and then > > > > calling msync ? > > > > > > > > Is their new plans to make a mremap call for FreeBSD 4.x ? > > > > > > no. > > > > > > > > > > > Or am I just sh%t out of luck ? > > > > > > Possibly, but if you describe what you are trying to accomplish > > > there may be some advice available. Your misuse of msync makes me > > > think that a rethinking of what you are trying to accomplish may > > > be a good idea. Please explain what makes you need mremap which > > > is not portable to any version of unix. I'm assuming you want your > > > app to work on Solaris and other commercial systems. > > > > > > -- > > > -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] > > > "I have the heart of a child; I keep it in a jar on my desk." > > > > > > To Unsubscribe: send mail to [EMAIL PROTECTED] > > > with "unsubscribe freebsd-hackers" in the body of the message > > > > Sorry ! > > > > Ok here is a snipit of what I am trying to do > > > > fstat(fd, &st); > > if( (base = (caddr_t *) mmap(0, st.st_size, ( PROT_READ | PROT_WRITE ) , > > MAP_SHARED, fd, 0)) == MAP_FAILED) > > { > >MSG_ERR("cannot mmap file, exiting ! "); > >_exit(1); > > } > > > > /* say st.st_size = 200, and I want to add to the end of the file more data > > */ > > base += 200; /* we are now at the end */ > > > > rec = (stuct rec_t *) base; > > rec->len = 200; > > /* we are writing to the buff, but we are past */ > > memmove(rec->data, data, 100); > > msync(base, 0, (st.st_len + rec->len), MS_ASYNC); > > > > the struct rec_t is an example of a stuct that I want to append to the buf > > and make another reccord. > > If base was a regular pointer we would segfault cause we are past the > > pointers buf > > but it kinda works.. meaning it will write and add and sometimes and not > > segfault... sometimes... > > > > or I can do this: > > > > rec = (stuct rec_t *) base; > > rec->len = 200; > > memmove(rec->data, data, 100); > > write(fd, (char *) rec, rec->len); > > msync(base, 0, (st.st_len + rec->len), MS_ASYNC); > > > > After I append to the file and then msync. I should be able to read the > > reccord that I just added from base. > > base should reflect the file right ? > > > > I want to make things simple and only append to the buffer "base" and not > > have to re-open the file and write to it then msync it back to mem. I know > > that some of this under FreeBSD "automatic" meaning that the changes made to > > the file are seen by "base" even though I do not call msync. but in > > order to keep things clear, I will msync. > > > > I hope that this wasn't too confusing... I am not good at explanations or > > spelling > > The first code snippet is completely incorrect, this is _not_ how > you use the mmap() interface and I'm suprised it works at all. > Writing after the mmap'd area should cause a SEGV of BUS signal to > be sent to your process. > > You really haven't provided the code you're using on Linux to use mremap() > which I'm assuming looks something like: > > > rec = (stuct rec_t *) base; > > rec->len = 200; > > memmove(rec->data, data, 100); > > write(fd, (char *) rec, rec->len); > ->>> mremap(arguments to extend the map); > > msync(base, 0, (st.st_len + rec->len), MS_ASYNC); > > (derived from the second snippet) > > Now i'm confused, why are you mmap()'ing like this? I know mremap() > offers a slightly easier interface, but you could simulate with some > code like so: > > (you've already mmap()'d a page aligned and multiple of page sized > region) > if (!mmap(base + offset, additional length, > PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, fd, offset)) { > > unmap file and remap it. > }; > > which would tell the OS to map in the end of the file at the end > of the map you already have. > > problems which can then happen is that if you mmap() multiple files > without specifying a fixed address then they may be mmap'd into > your address space right next to each other so that the fixed > mapping will fail. > > A more flexible way would be to maintain a structure for each file > that keeps track of all the base and length addresses that you've > mmap'd in the file. Then before writing to the memory address you > can do a quick lookup to figure out where the OS has mapped in that > particular offset into your address space. You could use a hash > on the fd and a binary t
Re: mremap help ? or no support for FreeBSD ? so do what ?
* [EMAIL PROTECTED] <[EMAIL PROTECTED]> [000513 00:41] wrote: > Alfred Perlstein wrote: > > > * [EMAIL PROTECTED] <[EMAIL PROTECTED]> [000512 21:54] wrote: > > > I know that this was discussed in the past but I can't find out what to > > > do ? > > > > > > In Linux if I have to resize a mmap 'ed object I can just use mremap > > > but in FreeBSD if, I want to resize it what do I do ? > > > > > > I have tried writing past where I know the end is and it kinda works ? > > > but why ? > > > > > > Is their a better solution besides just writing to the file and then > > > calling msync ? > > > > > > Is their new plans to make a mremap call for FreeBSD 4.x ? > > > > no. > > > > > > > > Or am I just sh%t out of luck ? > > > > Possibly, but if you describe what you are trying to accomplish > > there may be some advice available. Your misuse of msync makes me > > think that a rethinking of what you are trying to accomplish may > > be a good idea. Please explain what makes you need mremap which > > is not portable to any version of unix. I'm assuming you want your > > app to work on Solaris and other commercial systems. > > > > -- > > -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] > > "I have the heart of a child; I keep it in a jar on my desk." > > > > To Unsubscribe: send mail to [EMAIL PROTECTED] > > with "unsubscribe freebsd-hackers" in the body of the message > > Sorry ! > > Ok here is a snipit of what I am trying to do > > fstat(fd, &st); > if( (base = (caddr_t *) mmap(0, st.st_size, ( PROT_READ | PROT_WRITE ) , > MAP_SHARED, fd, 0)) == MAP_FAILED) > { >MSG_ERR("cannot mmap file, exiting ! "); >_exit(1); > } > > /* say st.st_size = 200, and I want to add to the end of the file more data > */ > base += 200; /* we are now at the end */ > > rec = (stuct rec_t *) base; > rec->len = 200; > /* we are writing to the buff, but we are past */ > memmove(rec->data, data, 100); > msync(base, 0, (st.st_len + rec->len), MS_ASYNC); > > the struct rec_t is an example of a stuct that I want to append to the buf > and make another reccord. > If base was a regular pointer we would segfault cause we are past the > pointers buf > but it kinda works.. meaning it will write and add and sometimes and not > segfault... sometimes... > > or I can do this: > > rec = (stuct rec_t *) base; > rec->len = 200; > memmove(rec->data, data, 100); > write(fd, (char *) rec, rec->len); > msync(base, 0, (st.st_len + rec->len), MS_ASYNC); > > After I append to the file and then msync. I should be able to read the > reccord that I just added from base. > base should reflect the file right ? > > I want to make things simple and only append to the buffer "base" and not > have to re-open the file and write to it then msync it back to mem. I know > that some of this under FreeBSD "automatic" meaning that the changes made to > the file are seen by "base" even though I do not call msync. but in > order to keep things clear, I will msync. > > I hope that this wasn't too confusing... I am not good at explanations or > spelling The first code snippet is completely incorrect, this is _not_ how you use the mmap() interface and I'm suprised it works at all. Writing after the mmap'd area should cause a SEGV of BUS signal to be sent to your process. You really haven't provided the code you're using on Linux to use mremap() which I'm assuming looks something like: > rec = (stuct rec_t *) base; > rec->len = 200; > memmove(rec->data, data, 100); > write(fd, (char *) rec, rec->len); ->>> mremap(arguments to extend the map); > msync(base, 0, (st.st_len + rec->len), MS_ASYNC); (derived from the second snippet) Now i'm confused, why are you mmap()'ing like this? I know mremap() offers a slightly easier interface, but you could simulate with some code like so: (you've already mmap()'d a page aligned and multiple of page sized region) if (!mmap(base + offset, additional length, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, fd, offset)) { unmap file and remap it. }; which would tell the OS to map in the end of the file at the end of the map you already have. problems which can then happen is that if you mmap() multiple files without specifying a fixed address then they may be mmap'd into your address space right next to each other so that the fixed mapping will fail. A more flexible way would be to maintain a structure for each file that keeps track of all the base and length addresses that you've mmap'd in the file. Then before writing to the memory address you can do a quick lookup to figure out where the OS has mapped in that particular offset into your address space. You could use a hash on the fd and a binary tree to store that infomation pretty easily. Basically implement a simple mmap() manager and you should be set. best of luck, -- -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] "I have the heart of a child; I keep it in a jar on my desk." To Unsubsc
Re: mremap help ? or no support for FreeBSD ? so do what ?
Alfred Perlstein wrote: > * [EMAIL PROTECTED] <[EMAIL PROTECTED]> [000512 21:54] wrote: > > I know that this was discussed in the past but I can't find out what to > > do ? > > > > In Linux if I have to resize a mmap 'ed object I can just use mremap > > but in FreeBSD if, I want to resize it what do I do ? > > > > I have tried writing past where I know the end is and it kinda works ? > > but why ? > > > > Is their a better solution besides just writing to the file and then > > calling msync ? > > > > Is their new plans to make a mremap call for FreeBSD 4.x ? > > no. > > > > > Or am I just sh%t out of luck ? > > Possibly, but if you describe what you are trying to accomplish > there may be some advice available. Your misuse of msync makes me > think that a rethinking of what you are trying to accomplish may > be a good idea. Please explain what makes you need mremap which > is not portable to any version of unix. I'm assuming you want your > app to work on Solaris and other commercial systems. > > -- > -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] > "I have the heart of a child; I keep it in a jar on my desk." > > To Unsubscribe: send mail to [EMAIL PROTECTED] > with "unsubscribe freebsd-hackers" in the body of the message Sorry ! Ok here is a snipit of what I am trying to do fstat(fd, &st); if( (base = (caddr_t *) mmap(0, st.st_size, ( PROT_READ | PROT_WRITE ) , MAP_SHARED, fd, 0)) == MAP_FAILED) { MSG_ERR("cannot mmap file, exiting ! "); _exit(1); } /* say st.st_size = 200, and I want to add to the end of the file more data */ base += 200; /* we are now at the end */ rec = (stuct rec_t *) base; rec->len = 200; /* we are writing to the buff, but we are past */ memmove(rec->data, data, 100); msync(base, 0, (st.st_len + rec->len), MS_ASYNC); the struct rec_t is an example of a stuct that I want to append to the buf and make another reccord. If base was a regular pointer we would segfault cause we are past the pointers buf but it kinda works.. meaning it will write and add and sometimes and not segfault... sometimes... or I can do this: rec = (stuct rec_t *) base; rec->len = 200; memmove(rec->data, data, 100); write(fd, (char *) rec, rec->len); msync(base, 0, (st.st_len + rec->len), MS_ASYNC); After I append to the file and then msync. I should be able to read the reccord that I just added from base. base should reflect the file right ? I want to make things simple and only append to the buffer "base" and not have to re-open the file and write to it then msync it back to mem. I know that some of this under FreeBSD "automatic" meaning that the changes made to the file are seen by "base" even though I do not call msync. but in order to keep things clear, I will msync. I hope that this wasn't too confusing... I am not good at explanations or spelling thank you nathan To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mremap help ? or no support for FreeBSD ? so do what ?
* [EMAIL PROTECTED] <[EMAIL PROTECTED]> [000512 21:54] wrote: > I know that this was discussed in the past but I can't find out what to > do ? > > In Linux if I have to resize a mmap 'ed object I can just use mremap > but in FreeBSD if, I want to resize it what do I do ? > > I have tried writing past where I know the end is and it kinda works ? > but why ? > > Is their a better solution besides just writing to the file and then > calling msync ? > > Is their new plans to make a mremap call for FreeBSD 4.x ? no. > > Or am I just sh%t out of luck ? Possibly, but if you describe what you are trying to accomplish there may be some advice available. Your misuse of msync makes me think that a rethinking of what you are trying to accomplish may be a good idea. Please explain what makes you need mremap which is not portable to any version of unix. I'm assuming you want your app to work on Solaris and other commercial systems. -- -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] "I have the heart of a child; I keep it in a jar on my desk." To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
mremap help ? or no support for FreeBSD ? so do what ?
I know that this was discussed in the past but I can't find out what to do ? In Linux if I have to resize a mmap 'ed object I can just use mremap but in FreeBSD if, I want to resize it what do I do ? I have tried writing past where I know the end is and it kinda works ? but why ? Is their a better solution besides just writing to the file and then calling msync ? Is their new plans to make a mremap call for FreeBSD 4.x ? Or am I just sh%t out of luck ? thank you in advance nathan To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message