Re: Concerning a post that you made about expandable anonymous shared mappings
Stas Sergeev wrote: Hi. William Tambe wrote: I understand your concern. But since I am working on a dynamic memory management code that I wish to use with other projects that I have, I didn't find appropriate to use shm_open. Could you please provide a detailed list of the problems you have with shm_open? If they are valid, then I can bet the patch will be applied, no matter what. :) In fact there is a name associated with the shared memory requested with shm_open, so that it can be mmap(ed) in another process. And I do not wish to have it accessible by any other process, unless I choose to do so. In this case you need to use shm_unlink() right after shm_open(). Then this shm will be accessable only to your process and its children, via an fd, and not to anyone else. And you still can do anything with it (ftruncate/mmap/mremap whatever). Ok, now I find myself without any other arguments :-) shm_unlink() right after shm_open() is a solution. And I think remap(ing) ANONYMOUS memory kind of make a lot of things easier. In what way, exactly? I wrote the above not knowing that I could use shm_unlink() right after shm_open(). But still, I have lost a considerable amount of time trying to figure that out. It appeared all natural to me that I could just remap ANONYMOUS and get what I wanted. And the worst thing here is that the man pages do not let you know about that. Sincerely, William Tambe - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Concerning a post that you made about expandable anonymous shared mappings
Stas Sergeev wrote: Hi. William Tambe wrote: I understand your concern. But since I am working on a dynamic memory management code that I wish to use with other projects that I have, I didn't find appropriate to use shm_open. Could you please provide a detailed list of the problems you have with shm_open? If they are valid, then I can bet the patch will be applied, no matter what. :) In fact there is a name associated with the shared memory requested with shm_open, so that it can be mmap(ed) in another process. And I do not wish to have it accessible by any other process, unless I choose to do so. In this case you need to use shm_unlink() right after shm_open(). Then this shm will be accessable only to your process and its children, via an fd, and not to anyone else. And you still can do anything with it (ftruncate/mmap/mremap whatever). Ok, now I find myself without any other arguments :-) shm_unlink() right after shm_open() is a solution. And I think remap(ing) ANONYMOUS memory kind of make a lot of things easier. In what way, exactly? I wrote the above not knowing that I could use shm_unlink() right after shm_open(). But still, I have lost a considerable amount of time trying to figure that out. It appeared all natural to me that I could just remap ANONYMOUS and get what I wanted. And the worst thing here is that the man pages do not let you know about that. Sincerely, William Tambe - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Concerning a post that you made about expandable anonymous shared mappings
Hugh Dickins wrote: On Mon, 9 Jul 2007, William Tambe wrote: Hugh Dickins wrote: I've come right around to your original view, Stas, and William's: if that mmap creates such an object, then the expanding mremap really ought to be useful, and allow the underlying object to be expanded. The shared anonymous object is already anomalous: expanding it on fault makes it more consistent with its own nature, not less. ... Here's a patch against 2.6.22-rc7: would you, Stas, put your Signed-off-by into this, and accept authorship - although I'm sending this back to you, it's very much your idea, and only trivially modified from your three-year-old patch by me. If you're agreeable, I can then forward it or its shmem_zero_fault equivalent to Andrew when we see which way 2.6.23 is going. ... Will this patch be added to stable versions of the linux kernel? Please let me know. I confess that the lukewarm response from Stas cooled my enthusiasm, and left me feeling that perhaps I'm an idiot to be adding such a feature so many years too late; and my old caution about the way a child could use up memory not freed on child's exit, unknown to parent, returned to haunt me. That could be documented for new usages, but I just don't know what usages are already out there, and fear I'd be introducing an exploit. It most certainly will not be added to a stable version of the linux kernel, if by that you mean 2.6.22.N or 2.6.21.N etc. Though it can be viewed as a bugfix, the patch as it stands seems in danger of introducing its own bug, and it's just too much of a feature to be suitable for a -stable release. But more probably you meant, will it be in 2.6.23 or 2.6.24? Sorry to be such a vacillatiing wimp, but I don't know. How well are you managing with the shm_open approach? Hugh I understand your concern. But since I am working on a dynamic memory management code that I wish to use with other projects that I have, I didn't find appropriate to use shm_open. In fact there is a name associated with the shared memory requested with shm_open, so that it can be mmap(ed) in another process. And I do not wish to have it accessible by any other process, unless I choose to do so. shm_open is great, but it doesn't quite fit my needs. And I think remap(ing) ANONYMOUS memory kind of make a lot of things easier. Can that feature be added at least to release candidates series so that everybody can try it and see how well it does? Sincerely, William Tambe - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Concerning a post that you made about expandable anonymous shared mappings
Hugh Dickins wrote: On Mon, 9 Jul 2007, William Tambe wrote: Hugh Dickins wrote: I've come right around to your original view, Stas, and William's: if that mmap creates such an object, then the expanding mremap really ought to be useful, and allow the underlying object to be expanded. The shared anonymous object is already anomalous: expanding it on fault makes it more consistent with its own nature, not less. ... Here's a patch against 2.6.22-rc7: would you, Stas, put your Signed-off-by into this, and accept authorship - although I'm sending this back to you, it's very much your idea, and only trivially modified from your three-year-old patch by me. If you're agreeable, I can then forward it or its shmem_zero_fault equivalent to Andrew when we see which way 2.6.23 is going. ... Will this patch be added to stable versions of the linux kernel? Please let me know. I confess that the lukewarm response from Stas cooled my enthusiasm, and left me feeling that perhaps I'm an idiot to be adding such a feature so many years too late; and my old caution about the way a child could use up memory not freed on child's exit, unknown to parent, returned to haunt me. That could be documented for new usages, but I just don't know what usages are already out there, and fear I'd be introducing an exploit. It most certainly will not be added to a stable version of the linux kernel, if by that you mean 2.6.22.N or 2.6.21.N etc. Though it can be viewed as a bugfix, the patch as it stands seems in danger of introducing its own bug, and it's just too much of a feature to be suitable for a -stable release. But more probably you meant, will it be in 2.6.23 or 2.6.24? Sorry to be such a vacillatiing wimp, but I don't know. How well are you managing with the shm_open approach? Hugh I understand your concern. But since I am working on a dynamic memory management code that I wish to use with other projects that I have, I didn't find appropriate to use shm_open. In fact there is a name associated with the shared memory requested with shm_open, so that it can be mmap(ed) in another process. And I do not wish to have it accessible by any other process, unless I choose to do so. shm_open is great, but it doesn't quite fit my needs. And I think remap(ing) ANONYMOUS memory kind of make a lot of things easier. Can that feature be added at least to release candidates series so that everybody can try it and see how well it does? Sincerely, William Tambe - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Concerning a post that you made about expandable anonymous shared mappings
Hugh Dickins wrote:
On Mon, 2 Jul 2007, Stas Sergeev wrote:
Hugh Dickins wrote:
You've answered your own question: we did not make the change Stas
suggested, IIRC because I remained a little uneasy with that change
in behaviour, and nobody else spoke up for it.
IIRC your argument, that made sense to me,
was that with such an approach, you can only
expand the backing-store, but never shrink.
I agree this is a problem from some point of
view. I have no idea whether it is important
or not, but it definitely _looks_ not very good.
You were gracious enough to accept my arguments back then, but after
mulling this over overnight, I've come to think I was just too timid
back then, and gave too much weight to the issue of there being no
shrink, and to the issue that child might expand the object without
parent knowing (that surplus remaining allocated until both exited).
I've come right around to your original view, Stas, and William's:
if that mmap creates such an object, then the expanding mremap really
ought to be useful, and allow the underlying object to be expanded.
The shared anonymous object is already anomalous: expanding it on
fault makes it more consistent with its own nature, not less.
//why does this failed. I am well in the interval [4096, 8192]
*(unsigned int *)(ptr + 4096 + 8)= 10;
}
Well, this generates a bus error, while, for
example, doing the same trick with having a
/dev/mem as a backing-store, simply maps the
"enlarged" space with the anonymous memory,
and so does not generate a SIGBUS (not producing
a desired effect either, of course).
Why do we have it both ways? Shouldn't they
(i.e. /dev/zero and /dev/mem mapping) behave
the same after expanding with mremap?
They've very different: mapping /dev/mem maps an existing object;
whereas mapping /dev/zero is a convention by which a new object
is created - and we can't afford the memory to make it of infinite
extent, so have limited it to the mapped size.
I haven't given it any thought since then:
I was thinking about it a bit, and I imagined
we need something like
int mopen(void *addr, size_t len, unsigned int flags);
which will give you an fd for the memory area,
which you can then ftruncate() and mmap() (and
mremap).
Ah, if we added an mopen(), there's no end to the discussions
we could have about what it can do. It may be a great idea:
but it's really not needed to solve this particular little
problem. As last time around, you were suggesting .mremap
callouts; but I much prefer your original shmem_zero_nopage,
which is a solution of the scale appropriate to the problem.
Such a thing could solve that mremap() problem
that me and William Tambe have.
If you're thinking of going that way, for this problem it
would be better just to stick with POSIX shm_open, as Christoph
suggested before, and you suggest to William in other mail.
But I now accept your view, that the shared anonymous object
is not behaving consistently in response to mremap, and would
like to put in a patch for that. This isn't a particularly good
time for such a patch - it's unclear right now whether 2.6.23 will
have shmem_nopage like 2.6.22 or shmem_fault like 2.6.22-rc-mm;
but we can easily adjust to whichever.
Here's a patch against 2.6.22-rc7: would you, Stas, put your
Signed-off-by into this, and accept authorship - although I'm
sending this back to you, it's very much your idea, and only
trivially modified from your three-year-old patch by me. If
you're agreeable, I can then forward it or its shmem_zero_fault
equivalent to Andrew when we see which way 2.6.23 is going.
[I'll fill in the patch comment later]
Signed-off-by: Hugh Dickins <[EMAIL PROTECTED]>
mm/shmem.c | 41 -
1 file changed, 40 insertions(+), 1 deletion(-)
--- 2.6.22-rc7/mm/shmem.c 2007-06-17 10:51:02.0 +0100
+++ linux/mm/shmem.c2007-07-03 15:35:32.0 +0100
@@ -1320,6 +1320,36 @@ static struct page *shmem_nopage(struct
return page;
}
+struct page *shmem_zero_nopage(struct vm_area_struct *vma,
+ unsigned long address, int *type)
+{
+ struct page *page;
+
+ page = shmem_nopage(vma, address, type);
+ if (unlikely(page == NOPAGE_SIGBUS)) {
+ struct inode *inode = vma->vm_file->f_dentry->d_inode;
+ struct shmem_inode_info *info = SHMEM_I(inode);
+ loff_t vm_size, i_size;
+
+ /*
+* If mremap has been used to extend the vma,
+* now extend the underlying object to include this page.
+*/
+ vm_size = (PAGE_ALIGN(address) - vma->vm_start) +
+ ((loff_t)vma->vm_pgoff << PAGE_SHIFT);
+ spin_lock(>lock);
+ i_size = i_size_read(inode);
+ if (i_size < vm_size && vm_size <= SHMEM_MAX_BYTES &&
+
Re: Concerning a post that you made about expandable anonymous shared mappings
Hugh Dickins wrote:
On Mon, 2 Jul 2007, Stas Sergeev wrote:
Hugh Dickins wrote:
You've answered your own question: we did not make the change Stas
suggested, IIRC because I remained a little uneasy with that change
in behaviour, and nobody else spoke up for it.
IIRC your argument, that made sense to me,
was that with such an approach, you can only
expand the backing-store, but never shrink.
I agree this is a problem from some point of
view. I have no idea whether it is important
or not, but it definitely _looks_ not very good.
You were gracious enough to accept my arguments back then, but after
mulling this over overnight, I've come to think I was just too timid
back then, and gave too much weight to the issue of there being no
shrink, and to the issue that child might expand the object without
parent knowing (that surplus remaining allocated until both exited).
I've come right around to your original view, Stas, and William's:
if that mmap creates such an object, then the expanding mremap really
ought to be useful, and allow the underlying object to be expanded.
The shared anonymous object is already anomalous: expanding it on
fault makes it more consistent with its own nature, not less.
//why does this failed. I am well in the interval [4096, 8192]
*(unsigned int *)(ptr + 4096 + 8)= 10;
}
Well, this generates a bus error, while, for
example, doing the same trick with having a
/dev/mem as a backing-store, simply maps the
enlarged space with the anonymous memory,
and so does not generate a SIGBUS (not producing
a desired effect either, of course).
Why do we have it both ways? Shouldn't they
(i.e. /dev/zero and /dev/mem mapping) behave
the same after expanding with mremap?
They've very different: mapping /dev/mem maps an existing object;
whereas mapping /dev/zero is a convention by which a new object
is created - and we can't afford the memory to make it of infinite
extent, so have limited it to the mapped size.
I haven't given it any thought since then:
I was thinking about it a bit, and I imagined
we need something like
int mopen(void *addr, size_t len, unsigned int flags);
which will give you an fd for the memory area,
which you can then ftruncate() and mmap() (and
mremap).
Ah, if we added an mopen(), there's no end to the discussions
we could have about what it can do. It may be a great idea:
but it's really not needed to solve this particular little
problem. As last time around, you were suggesting .mremap
callouts; but I much prefer your original shmem_zero_nopage,
which is a solution of the scale appropriate to the problem.
Such a thing could solve that mremap() problem
that me and William Tambe have.
If you're thinking of going that way, for this problem it
would be better just to stick with POSIX shm_open, as Christoph
suggested before, and you suggest to William in other mail.
But I now accept your view, that the shared anonymous object
is not behaving consistently in response to mremap, and would
like to put in a patch for that. This isn't a particularly good
time for such a patch - it's unclear right now whether 2.6.23 will
have shmem_nopage like 2.6.22 or shmem_fault like 2.6.22-rc-mm;
but we can easily adjust to whichever.
Here's a patch against 2.6.22-rc7: would you, Stas, put your
Signed-off-by into this, and accept authorship - although I'm
sending this back to you, it's very much your idea, and only
trivially modified from your three-year-old patch by me. If
you're agreeable, I can then forward it or its shmem_zero_fault
equivalent to Andrew when we see which way 2.6.23 is going.
[I'll fill in the patch comment later]
Signed-off-by: Hugh Dickins [EMAIL PROTECTED]
mm/shmem.c | 41 -
1 file changed, 40 insertions(+), 1 deletion(-)
--- 2.6.22-rc7/mm/shmem.c 2007-06-17 10:51:02.0 +0100
+++ linux/mm/shmem.c2007-07-03 15:35:32.0 +0100
@@ -1320,6 +1320,36 @@ static struct page *shmem_nopage(struct
return page;
}
+struct page *shmem_zero_nopage(struct vm_area_struct *vma,
+ unsigned long address, int *type)
+{
+ struct page *page;
+
+ page = shmem_nopage(vma, address, type);
+ if (unlikely(page == NOPAGE_SIGBUS)) {
+ struct inode *inode = vma-vm_file-f_dentry-d_inode;
+ struct shmem_inode_info *info = SHMEM_I(inode);
+ loff_t vm_size, i_size;
+
+ /*
+* If mremap has been used to extend the vma,
+* now extend the underlying object to include this page.
+*/
+ vm_size = (PAGE_ALIGN(address) - vma-vm_start) +
+ ((loff_t)vma-vm_pgoff PAGE_SHIFT);
+ spin_lock(info-lock);
+ i_size = i_size_read(inode);
+ if (i_size vm_size vm_size = SHMEM_MAX_BYTES
+ !shmem_acct_size(info-flags, vm_size - i_size
Re: Concerning a post that you made about expandable anonymous shared mappings
Yes, I have a good case, but my case may not sound interesting until you
see it working.
Ok, I am developing a dynamic memory allocation routine which takes
direct advantage of the ability of a machine to use Virtual Memory to
make everything look contiguous and fast.
And it just doesn't make sens to have mmap() map ANONYMOUS shared memory
and mremap() not to expand it and make the expanded area available.
I am not a 100% sure, but I don't think correcting its behavior would
cause a problem.
Would you happen to know how I can work around that issue for now, and
make writing in an expended area not to generate a Bus error?
Sincerely,
William Tambe
Hugh Dickins wrote:
On Fri, 29 Jun 2007, William Tambe wrote:
I read a post that you made about not being able to expand anonymous shared
mapping with mremap(). And I am actually having that issue now.
I guess you're referring to the thread at
http://lkml.org/lkml/2004/6/16/155
and you're asking either Stas or me.
You made the post in 2004 and we are now in 2007. I would like to know if that
feature was added because the code below always fail with bus error on my
machine. I use glibc 2.5
You've answered your own question: we did not make the change Stas
suggested, IIRC because I remained a little uneasy with that change
in behaviour, and nobody else spoke up for it.
I haven't given it any thought since then:
do you have a good case for us to reconsider it?
Hugh
Thank you for helping.
#define _GNU_SOURCE
#include
#include
#include
main() {
void *ptr;
if ((ptr=mmap(0, 4096, PROT_READ|PROT_WRITE,
MAP_ANONYMOUS|MAP_SHARED|MAP_GROWSDOWN, 0, 0)) == -1) {
printf("failed to mmap\n");
return;
}
if ((ptr=mremap(ptr, 4096, 8192, MREMAP_MAYMOVE)) == -1) {
printf("failed to mremap\n");
return;
}
//why does this failed. I am well in the interval [4096, 8192]
*(unsigned int *)(ptr + 4096 + 8)= 10;
}
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Concerning a post that you made about expandable anonymous shared mappings
Yes, I have a good case, but my case may not sound interesting until you
see it working.
Ok, I am developing a dynamic memory allocation routine which takes
direct advantage of the ability of a machine to use Virtual Memory to
make everything look contiguous and fast.
And it just doesn't make sens to have mmap() map ANONYMOUS shared memory
and mremap() not to expand it and make the expanded area available.
I am not a 100% sure, but I don't think correcting its behavior would
cause a problem.
Would you happen to know how I can work around that issue for now, and
make writing in an expended area not to generate a Bus error?
Sincerely,
William Tambe
Hugh Dickins wrote:
On Fri, 29 Jun 2007, William Tambe wrote:
I read a post that you made about not being able to expand anonymous shared
mapping with mremap(). And I am actually having that issue now.
I guess you're referring to the thread at
http://lkml.org/lkml/2004/6/16/155
and you're asking either Stas or me.
You made the post in 2004 and we are now in 2007. I would like to know if that
feature was added because the code below always fail with bus error on my
machine. I use glibc 2.5
You've answered your own question: we did not make the change Stas
suggested, IIRC because I remained a little uneasy with that change
in behaviour, and nobody else spoke up for it.
I haven't given it any thought since then:
do you have a good case for us to reconsider it?
Hugh
Thank you for helping.
#define _GNU_SOURCE
#include sys/mman.h
#include unistd.h
#include stdio.h
main() {
void *ptr;
if ((ptr=mmap(0, 4096, PROT_READ|PROT_WRITE,
MAP_ANONYMOUS|MAP_SHARED|MAP_GROWSDOWN, 0, 0)) == -1) {
printf(failed to mmap\n);
return;
}
if ((ptr=mremap(ptr, 4096, 8192, MREMAP_MAYMOVE)) == -1) {
printf(failed to mremap\n);
return;
}
//why does this failed. I am well in the interval [4096, 8192]
*(unsigned int *)(ptr + 4096 + 8)= 10;
}
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Concerning a post that you made about expandable anonymous shared mappings
I read a post that you made about not being able to expand anonymous
shared mapping with mremap(). And I am actually having that issue now.
You made the post in 2004 and we are now in 2007. I would like to know
if that feature was added because the code below always fail with bus
error on my machine. I use glibc 2.5
Thank you for helping.
#define _GNU_SOURCE
#include
#include
#include
main() {
void *ptr;
if ((ptr=mmap(0, 4096, PROT_READ|PROT_WRITE,
MAP_ANONYMOUS|MAP_SHARED|MAP_GROWSDOWN, 0, 0)) == -1) {
printf("failed to mmap\n");
return;
}
if ((ptr=mremap(ptr, 4096, 8192, MREMAP_MAYMOVE)) == -1) {
printf("failed to mremap\n");
return;
}
//why does this failed. I am well in the interval [4096, 8192]
*(unsigned int *)(ptr + 4096 + 8)= 10;
}
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Concerning a post that you made about expandable anonymous shared mappings
I read a post that you made about not being able to expand anonymous
shared mapping with mremap(). And I am actually having that issue now.
You made the post in 2004 and we are now in 2007. I would like to know
if that feature was added because the code below always fail with bus
error on my machine. I use glibc 2.5
Thank you for helping.
#define _GNU_SOURCE
#include sys/mman.h
#include unistd.h
#include stdio.h
main() {
void *ptr;
if ((ptr=mmap(0, 4096, PROT_READ|PROT_WRITE,
MAP_ANONYMOUS|MAP_SHARED|MAP_GROWSDOWN, 0, 0)) == -1) {
printf(failed to mmap\n);
return;
}
if ((ptr=mremap(ptr, 4096, 8192, MREMAP_MAYMOVE)) == -1) {
printf(failed to mremap\n);
return;
}
//why does this failed. I am well in the interval [4096, 8192]
*(unsigned int *)(ptr + 4096 + 8)= 10;
}
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
