Re: possible problem with memory allocation using calloc/mmap/munmap
> > > > > > > > It seems that when mmap() is called with length argument > > > > > > > > exceeding > > > > > > > > size of file, only memory to fit that file is allocated. > > > > > > > > munmap() > > > > > > > > however frees the full specified length. [...] > > > > > > > [...] > > > > > > > I know this situation is unsatisfying, but I have no easy > > > > > > > workaround > > > > > > > to allow this. Cygwin could add the anonymous mapping on the next > > > > > > > 64K boundary on 64 bit, but that would result in a hole in the > > > > > > > mapping > > > > > > > which seemed like a rather bad idea when porting mmap to 64 bit. > > > > > > > > > > > > > > Ken's also right that munmap is doing the right thing here. If > > > > > > > anything's wrong, it's mmap's workaround for mappings beyond the > > > > > > > file > > > > > > > length. If only 64 bit would allow 4K-aligned mappings :( > > > > > > > > > > > > Thanks for the answer. It is appreciated. > > > > > > I understand the problem and difficulty to resolve it. Maybe > > > > > > returning > > > > > > an error from mmap (and putting a comment to code for its reason) > > > > > > would be sufficient. mmap caller could just adjust requested > > > > > > allocation size to file size. Without error, caller has no way of > > > > > > knowing memory was not allocated and segfault is then thrown in an > > > > > > unrelated memory segment which makes the root cause hard to track > > > > > > down. But, I do not know all the implication that could result from > > > > > > that, so evaluation of this approach is up to you. > > > > > [...] > > > > > Eventually Cygwin adds another mapping to fullfill the entire mapping > > > > > request: > > > > > > > > > > |-- file 4K --|-- filler 60K --|-- filler 192K --| > > > > > > > > > > The problem on WOW64 and real 64 bit is that it's impossible to map > > > > > the first filler. However, this area in the VM will *never* be > > > > > allocated by other application functions due to the allocation > > > > > granularity of 64K! > > > > > > > > > > So my workaround for 64 bit and WOW64 is to just skip allocating the > > > > > first filler: > > > > > > > > > > |-- file 4K --|-- THE VOID 60K --|-- filler 192K --| > > > > > > > > > > The advantage is now that the following munmap of 256K will only > > > > > unmap the map for the file and the filler, but not the region you > > > > > calloced before, which formerly was accidentally mapped to the > > > > > filler region. This just can't happen anymore now. > > > > > > > > > > Would that be feasible? If so I can push my patch and create a > > > > > developer snapshot for testing. > > > > > > > > Two questions arise when I'm thinking about workaround solution: > > > > - what happens if caller tries to write to |-- THE VOID 60K --|. Since > > > > this is unallocated, would there be a segfault? > > > > > > Accessing the VOID would raise SIGSEGV, while accessing the filler > > > raises SIGBUS. The latter is also used to implement MAP_NORESERVE, > > > which the VOID can't support. > > > > I played around a bit and I can confirm it would be consistent with > > current behavior: > > memwrite <0 - filesize) - no error, written to file > > memwrite > memwrite <4k, 64k) - SIGSEGV > > memwrite <64k, mmap alloc size) - SIGSEGV or another mem alloc > > overwrite (depending on whether there is another allocation) > > With workaround last line would be fixed to SIGBUS (along with proper > > allocation length). I believe this is completely OK. > > > > > > > > > - is it possible that some subsequent mem alloc request would return > > > > region from |-- THE VOID 60K --| which could again cause segfault > > > > after munmap? > > > > > > No, as stated above. Allocations are restricted to Windows' 64K > > > allocation granularity. > > > > I apologize. I missed that sentence. So, your workaround seems fine. > > Please try the latest snapshot from https://cygwin.com/snapshots/ > Just replacing the Cygwin DLL is sufficient. My test programs work fine and also openldap crash that led me to this issue is gone (openldap tests still fail, but on a different place after the one above and now with a SIGBUS which already gives a good hint as to where to look). I think your patch works quite well. Thanks a lot. Stanislav Kascak -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: possible problem with memory allocation using calloc/mmap/munmap
> > > > > > It seems that when mmap() is called with length argument exceeding > > > > > > size of file, only memory to fit that file is allocated. munmap() > > > > > > however frees the full specified length. [...] > > > > > [...] > > > > > I know this situation is unsatisfying, but I have no easy workaround > > > > > to allow this. Cygwin could add the anonymous mapping on the next > > > > > 64K boundary on 64 bit, but that would result in a hole in the mapping > > > > > which seemed like a rather bad idea when porting mmap to 64 bit. > > > > > > > > > > Ken's also right that munmap is doing the right thing here. If > > > > > anything's wrong, it's mmap's workaround for mappings beyond the file > > > > > length. If only 64 bit would allow 4K-aligned mappings :( > > > > > > > > Thanks for the answer. It is appreciated. > > > > I understand the problem and difficulty to resolve it. Maybe returning > > > > an error from mmap (and putting a comment to code for its reason) > > > > would be sufficient. mmap caller could just adjust requested > > > > allocation size to file size. Without error, caller has no way of > > > > knowing memory was not allocated and segfault is then thrown in an > > > > unrelated memory segment which makes the root cause hard to track > > > > down. But, I do not know all the implication that could result from > > > > that, so evaluation of this approach is up to you. > > > [...] > > > Eventually Cygwin adds another mapping to fullfill the entire mapping > > > request: > > > > > > |-- file 4K --|-- filler 60K --|-- filler 192K --| > > > > > > The problem on WOW64 and real 64 bit is that it's impossible to map > > > the first filler. However, this area in the VM will *never* be > > > allocated by other application functions due to the allocation > > > granularity of 64K! > > > > > > So my workaround for 64 bit and WOW64 is to just skip allocating the > > > first filler: > > > > > > |-- file 4K --|-- THE VOID 60K --|-- filler 192K --| > > > > > > The advantage is now that the following munmap of 256K will only > > > unmap the map for the file and the filler, but not the region you > > > calloced before, which formerly was accidentally mapped to the > > > filler region. This just can't happen anymore now. > > > > > > Would that be feasible? If so I can push my patch and create a > > > developer snapshot for testing. > > > > Two questions arise when I'm thinking about workaround solution: > > - what happens if caller tries to write to |-- THE VOID 60K --|. Since > > this is unallocated, would there be a segfault? > > Accessing the VOID would raise SIGSEGV, while accessing the filler > raises SIGBUS. The latter is also used to implement MAP_NORESERVE, > which the VOID can't support. I played around a bit and I can confirm it would be consistent with current behavior: memwrite <0 - filesize) - no error, written to file memwrite > > - is it possible that some subsequent mem alloc request would return > > region from |-- THE VOID 60K --| which could again cause segfault > > after munmap? > > No, as stated above. Allocations are restricted to Windows' 64K > allocation granularity. I apologize. I missed that sentence. So, your workaround seems fine. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: possible problem with memory allocation using calloc/mmap/munmap
> > > > It seems that when mmap() is called with length argument exceeding > > > > size of file, only memory to fit that file is allocated. munmap() > > > > however frees the full specified length. Since (at least on my > > > > computer) big chunk of memory allocated by calloc() is located after > > > > mmap() allocation, munmap() frees even memory of that calloc(). > > > > > > Ken's right. Due to the differences between mapping files on Windows > > > vs. Unix, Cygwin can't map beyond the file size + the remainder of the > > > last page. Cygwin tries to workaround that on 32 bit by allocating > > > an anonymous mapping following the file mapping to keep the range free > > > from other mappings. But on 64 bit this workaround doesn't work anymore > > > because the OS is missing an (undocumented) flag which allows to > > > create mappings on 4K boundaries, rather than just on 64K boundaries. > > > > > > I know this situation is unsatisfying, but I have no easy workaround > > > to allow this. Cygwin could add the anonymous mapping on the next > > > 64K boundary on 64 bit, but that would result in a hole in the mapping > > > which seemed like a rather bad idea when porting mmap to 64 bit. > > > > > > Ken's also right that munmap is doing the right thing here. If > > > anything's wrong, it's mmap's workaround for mappings beyond the file > > > length. If only 64 bit would allow 4K-aligned mappings :( > > > > Thanks for the answer. It is appreciated. > > I understand the problem and difficulty to resolve it. Maybe returning > > an error from mmap (and putting a comment to code for its reason) > > would be sufficient. mmap caller could just adjust requested > > allocation size to file size. Without error, caller has no way of > > knowing memory was not allocated and segfault is then thrown in an > > unrelated memory segment which makes the root cause hard to track > > down. But, I do not know all the implication that could result from > > that, so evaluation of this approach is up to you. > > Given that most of the required code already exists for 32 bit systems > (except under WOW64, suffering the same problem as the 64 bit WIndows > environment), I hacked a bit on this code this morning and I got your > testcase running fine. The idea being that after a successful mmap the > expectation that a matching munmap does *not* unmap unrelated mappings > is valid. > > In more depth, here's what Cygwin does on 32 bit, assuming a file size > of 100 bytes and a mapping request of 256K: > > First Cygwin mmaps the file. This results in a 4K mapping in Windows: > > file:|-- 100b --| > > mapping: |-- 4K ----| > > Next Cygwin adds another mapping to fill up the range up to the next > 64K allocation granularity boundary: > > |-- file 4K --|-- filler 60K --| > > Eventually Cygwin adds another mapping to fullfill the entire mapping > request: > > |-- file 4K --|-- filler 60K --|-- filler 192K --| > > The problem on WOW64 and real 64 bit is that it's impossible to map > the first filler. However, this area in the VM will *never* be > allocated by other application functions due to the allocation > granularity of 64K! > > So my workaround for 64 bit and WOW64 is to just skip allocating the > first filler: > > |-- file 4K --|-- THE VOID 60K --|-- filler 192K --| > > The advantage is now that the following munmap of 256K will only > unmap the map for the file and the filler, but not the region you > calloced before, which formerly was accidentally mapped to the > filler region. This just can't happen anymore now. > > Would that be feasible? If so I can push my patch and create a > developer snapshot for testing. Two questions arise when I'm thinking about workaround solution: - what happens if caller tries to write to |-- THE VOID 60K --|. Since this is unallocated, would there be a segfault? - is it possible that some subsequent mem alloc request would return region from |-- THE VOID 60K --| which could again cause segfault after munmap? Stanislav Kascak -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: possible problem with memory allocation using calloc/mmap/munmap
> > It seems that when mmap() is called with length argument exceeding > > size of file, only memory to fit that file is allocated. munmap() > > however frees the full specified length. Since (at least on my > > computer) big chunk of memory allocated by calloc() is located after > > mmap() allocation, munmap() frees even memory of that calloc(). > > Ken's right. Due to the differences between mapping files on Windows > vs. Unix, Cygwin can't map beyond the file size + the remainder of the > last page. Cygwin tries to workaround that on 32 bit by allocating > an anonymous mapping following the file mapping to keep the range free > from other mappings. But on 64 bit this workaround doesn't work anymore > because the OS is missing an (undocumented) flag which allows to > create mappings on 4K boundaries, rather than just on 64K boundaries. > > I know this situation is unsatisfying, but I have no easy workaround > to allow this. Cygwin could add the anonymous mapping on the next > 64K boundary on 64 bit, but that would result in a hole in the mapping > which seemed like a rather bad idea when porting mmap to 64 bit. > > Ken's also right that munmap is doing the right thing here. If > anything's wrong, it's mmap's workaround for mappings beyond the file > length. If only 64 bit would allow 4K-aligned mappings :( Thanks for the answer. It is appreciated. I understand the problem and difficulty to resolve it. Maybe returning an error from mmap (and putting a comment to code for its reason) would be sufficient. mmap caller could just adjust requested allocation size to file size. Without error, caller has no way of knowing memory was not allocated and segfault is then thrown in an unrelated memory segment which makes the root cause hard to track down. But, I do not know all the implication that could result from that, so evaluation of this approach is up to you. Thanks again. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
possible problem with memory allocation using calloc/mmap/munmap
Hello cygwin team,
I came across a problem with memory allocation/deallocation when
trying to compile and run tests of openldap under cygwin.
I created a test program to simulate sequence of actions. First a
bigger chunk of memory (>~262kB) is allocated using calloc(), then
mmap() is called with requested memory size much bigger than actual
file size. Subsequently, munmap() to free previous mmap(), and at the
end an attempt to access memory allocated with calloc(). That last
memory access is causing Segmentation fault.
It seems that when mmap() is called with length argument exceeding
size of file, only memory to fit that file is allocated. munmap()
however frees the full specified length. Since (at least on my
computer) big chunk of memory allocated by calloc() is located after
mmap() allocation, munmap() frees even memory of that calloc().
I'm not sure if this is only a problem of my environment, or there is
something I'm missing.
Below is the test (notice that memory allocated by mmap() is only 64k
bytes in front of the memory allocated by calloc(), not 32MB as
requested).
My system is Windows 7 64 bit, cygwin64 3.0.7.
Best regards,
Stanislav Kascak
$ cat a.c
#include
#include
#include
#include
#include
#include
#include
int main(int argc, char** argv){
char* x;
char* y;
int fd;
size_t s1 = 0x4;
size_t s2 = 0x200;
x = (char*)calloc(1, s1);
if(x == NULL){
printf("calloc failed\n");
}
x[0] = 'a';
fd = open("testfile", O_RDONLY);
if(fd == -1){
printf("open failed: %d\n", errno);
}
y = mmap(NULL, s2, PROT_READ, MAP_SHARED, fd, 0);
if(y == MAP_FAILED){
printf("mmap failed: %d\n", errno);
}
close(fd);
printf("%lx -> %c\n", x, x[0]);
printf("%lx\n", y);
if(munmap(y, s2)){
printf("munmap failed: %d\n", errno);
}
printf("%lx -> %c\n", x, x[0]);
free(x);
}
$ gcc a.c
$ echo "testdata" > testfile
$ ./a.exe
6fb0010 -> a
6fa
Segmentation fault (core dumped)
$ uname -a
CYGWIN_NT-6.1 myhost 3.0.7(0.338/5/3) 2019-04-30 18:08 x86_64 Cygwin
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
