On Oct 15, 2010, at 10:24 PM, Black, Michael (IS) wrote: > I'm not sure but I suspect sqlite is not calling unmap before > extending the area. > That would explain why it still gets zeroed out even with the flags. > > Put a break point in the unixShmUnmap call and see if it gets called > before mmap.
We're doing this: ftruncate(fd, 32*1024); mmap(0, 32*1024, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0); <write into the shared memory we just mapped> ftruncate(fd, 64*1024); mmap(0, 32*1024, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 32*1024); No unmap calls. Praveen's investigations suggest that the second mmap() call is zeroing the memory mapped by the first mmap() call. Which is, as you might expect, confusing SQLite. I guess it could also be the second ftruncate() call that is zeroing our mapped memory. That would be even odder though... Dan. > May just need some QNX logic that says "if we're extending an area > unmap it first with flags". > > I think the mmap should honor the NOINIT flag when extending an area > but apparently it doesn't -- only unmap can set the flag to make it > honor it. QNX must be about the only one that does this. > > QNX mmap claims to be POSIX 1003.1 compliant but I don't see this > behavior defined in there. > > The only zero-fill reference I see in POSIX is > "The system shall always zero-fill any partial page at the end of an > object" > which doesn't fit this behavior at all. > > Perhaps you should report this as a bug or non-desirable/non- > compliant behavior compared to every other OS in the world (and the > POSIX standard which doesn't call for this behavior). > > Michael D. Black > Senior Scientist > Advanced Analytics Directorate > Northrop Grumman Information Systems > > > ________________________________ > > From: [email protected] on behalf of Raj, Praveen > Sent: Fri 10/15/2010 9:55 AM > To: General Discussion of SQLite Database > Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests > fail on QNX OS > > > > Hi Michael, > > Yes I added the "MAP_NOINIT" to mmap() and "UNMAP_INIT_OPTIONAL" > flag to munmap_flags() call. Don't know where i might be going wrong > in SQLite. > > As you suggested, I wrote a small application to check if this > works. Fortunately it worked as desired (as given below). > > MAP_NOINIT > When specified, the POSIX requirement that the memory be zeroed is > relaxed. The physical memory being used for this allocation must > have been previously freed with UNMAP_INIT_OPTIONAL for this flag to > have any effect. > > _________________________________________________________________________________ > int main(int argc, char *argv[]) { > printf("Welcome to the QNX Momentics IDE\n"); > > //Open a file > int fd = open("/tmp/mmaptest",O_RDWR|O_CREAT, 0777); > > //Truncate the file > int size = ftruncate(fd, 2048); > > const char buff[]="Testing the mmap API"; > write(fd,buff,sizeof(buff)); > > //Mapping the disk file to memory > void *pMem = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED, fd, 0); > > //Check the mmapped memory contents > printf("Memory Mapped Data: %s\n", pMem); > > //Set all mmapped locations to 'A' > memset(pMem,'A',2048); > > //Unmap the memory > //munmap(pMem, 2048); > munmap_flags(pMem, 2048, UNMAP_INIT_OPTIONAL); > > //Buffer to read disk data > char disk_read[2048]; > > //Read the data from the file to check if its synced > read(fd,disk_read,sizeof(disk_read)); > printf("Reading disk file data:%s\n",disk_read); > > //Resetting the contents of disk file > size = ftruncate(fd, 0); > close(fd); > > //Open the file again to check to re-map the memory > fd = open("/tmp/mmaptest",O_RDWR, 0777); > size = ftruncate(fd, 2048); > > //Map the memory region again > //int pMem1 = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED, fd, > 0); > int pMem1 = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED| > MAP_NOINIT, fd, 0); > > //Print the re-mapped region > printf("New Memory Mapped Data: %s\n", pMem1); > > //Read the disk contents again to check if "MAP_NOINIT" works > read(fd,disk_read,sizeof(disk_read)); > printf("Disk file data during re-map: %s\n", disk_read); > > //Close the disk file > close(fd); > > return EXIT_SUCCESS; > } > > --------------------------------------------------------------- > Output: > > Welcome to the QNX Momentics IDE > Memory Mapped Data: Testing the mmap API > Reading disk file data:AAAAAAAAAAAAAAAAAAAAAA [2048 times] > New Memory Mapped Data: AAAAAAAAAAAAAAAAAAAAA [2048 times] > Disk file data during re-map: > --------------------------------------------------------------- > > Also found that the mmapped data is written back to disk file on > munmap()/munmap_flags() call. > > Maybe i need to look into Sqlite amalgamation file in detail to > analyze what's happening here during unmapping of memory. > I believe we can use these flags under all scenarios without any pre- > conditions. Not sure if POSIX (on QNX) have some limitations on > their usage. > > Thanks, > Praveen > ______________________________________________________________________________________________________________________ > > -----Original Message----- > From: [email protected] [mailto:[email protected] > ] On Behalf Of Black, Michael (IS) > Sent: Thursday, October 14, 2010 9:22 PM > To: General Discussion of SQLite Database > Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX > OS > > Did you also add the MAP_NOINIT to the mmap() call? > > It sounds like exactly the behavior you're seeing. > > Did you try writing a stand-alone app to test this idea? > > Michael D. Black > Senior Scientist > Advanced Analytics Directorate > Northrop Grumman Information Systems > > > ________________________________ > > From: [email protected] on behalf of Raj, Praveen > Sent: Thu 10/14/2010 10:44 AM > To: General Discussion of SQLite Database > Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests > fail on QNX OS > > > > Hi Micheal, > > Thanks Dan and Michael for all your inputs. > > I tried this approach as well, but didn't find any success. > During unmapping i used the API munmap_flags() with > "UNMAP_INIT_OPTIONAL" flag to avoid the zero initialization during > the next mmaping. > > Another thought I have here is that the old mmapped regions may not > be initialized with zeros, but instead the regions are getting > synced with the data in disk file (which is full of zeros). Not sure > if can happen with mmap() API though? > > > Thanks, > Praveen > > -----Original Message----- > From: [email protected] [mailto:[email protected] > ] On Behalf Of Black, Michael (IS) > Sent: Thursday, October 14, 2010 5:21 PM > To: General Discussion of SQLite Database > Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX > OS > > I sent this before...have you tried this? > > According to the QNX mmap page > http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/lib_ref/m/mmap.html > > <https://owa1.ngc.com/exchweb/bin/redir.asp?URL=http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/lib_ref/m/mmap.html > > > > > MAP_NOINIT > When specified, the POSIX requirement that the memory be zeroed is > relaxed. The physical memory being used for this allocation must > have been previously freed with UNMAP_INIT_OPTIONAL for this flag to > have any effect. > ________________________________ > > This flag was added in the QNX Neutrino Core OS 6.3.2. > > Interesting that this claims it's a POSIX requirement but I don't > think most any others do this. > > > Michael D. Black > Senior Scientist > Advanced Analytics Directorate > Northrop Grumman Information Systems > > > ________________________________ > > From: [email protected] on behalf of Raj, Praveen > Sent: Thu 10/14/2010 5:53 AM > To: General Discussion of SQLite Database > Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests > fail on QNX OS > > > > Hi Dan, > > I did some more investigation on the issue and i feel there is > synchronization problem happening here. > > After mmapping the shm (wal index) file to process memory, the WAL > indexes are written into the mmapped area, and this data is not > getting synchronized with physical (shm) file. As a result when the > mmap() function is called the second time to map the 32k-64k memory > region, it is synchronizing the complete mmapped region (previous > 32k regions) with physical file, even though a valid offset is > passed. Not sure if this is the actual behaviour of mmap() call. > > While debugging, before the mmap() call i checked mmapped region and > it had valid indexes, whereas after the call all became 0's. Also i > found that the shm file is always filled with 0's even after commits. > > When i added the msync() statement (to sync the shm file) before > mmap call as shown below, the problem is not seen. In this case the > shm file has valid 32-bit indexes, as data is synchronized before > next mmap call is executed. > > while(pShmNode->nRegion<=iRegion){ > int ret = msync( apNew[0], iRegion*szRegion, MS_SYNC); > void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE, > MAP_SHARED, pShmNode->h, pShmNode->nRegion*szRegion > ); > > With the above msync() call all my failed test cases are passing. > > I don't see any msync() call in the SQLite amalgamation/wal.c file. > I believe the data in mapped region and physical file are not > synched automatically. We need to explicitly do it using msync() call. > Don't know if there is any other mechanism in SQLite through which > the data is synchronized. Does the call to sqlite3OsSync() sync the > shm file as well? or is the shm file not syned purposefully? > > This is all my understanding and not sure if this is causing the > actual issue. Please guide me if my approach/understanding is > incorrect. > > > Thanks, > Praveen > > -----Original Message----- > From: [email protected] [mailto:[email protected] > ] On Behalf Of Dan Kennedy > Sent: Friday, October 08, 2010 9:33 PM > To: General Discussion of SQLite Database > Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX > OS > > > On Oct 8, 2010, at 9:44 PM, Raj, Praveen wrote: > >> Hello, >> >> I debugged the SQLite functions and here is my finding: >> >> The call to "mmap" in the function "unixShmMap" is causing the issue. >> void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE, >> MAP_SHARED, pShmNode->h, iRegion*szRegion); >> >> It is setting the previous memory region/regions to zero while >> mapping >> the new ones. Mmap call internally uses the QNX API mmap64() to map >> the required memory region. Not sure on what is happening here. Just >> need to dig into memory mapping to find whats happening and hopefully >> find a solution. >> >> Dan - Do you have any idea on why this could be happening? > > Sounds like a bug in QNX to me. > > _______________________________________________ > sqlite-users mailing list > [email protected] > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > The information contained in this message may be confidential and > legally protected under applicable law. The message is intended > solely for the addressee(s). If you are not the intended recipient, > you are hereby notified that any use, forwarding, dissemination, or > reproduction of this message is strictly prohibited and may be > unlawful. If you are not the intended recipient, please contact the > sender by return e-mail and destroy all copies of the original > message. > > _______________________________________________ > sqlite-users mailing list > [email protected] > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > > > _______________________________________________ > sqlite-users mailing list > [email protected] > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > > > _______________________________________________ > sqlite-users mailing list > [email protected] > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > > _______________________________________________ > sqlite-users mailing list > [email protected] > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list [email protected] http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
