Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Raj, Praveen]

Hello Dan,

To answer your question, the 2nd ftruncate() is not zeroing the 1st mmapped 
region. I had checked it using debugger, and it happens only during
2nd mmap call.
But from my investigation, i found that the 2nd ftruncate()(and further calls) 
is the one from where the problem is arising.

The ftruncate() call is setting some flag (it could be time field of file) 
which will make the 1st region to remap with disk file contents, when 2nd 
mmap() is called.
I avoided the second ftruncate call (hardcoded size to high value) and 2nd 
mmap() call worked as desired - mapped only the 2nd 32k region retaining the 
1st 32k region data intact.

---
int ftruncate( int fildes,
   off_t length );

int ftruncate64( int fildes,
 off64_t length );

Arguments:
fildes
The descriptor for the file that you want to truncate.
length
The length that you want the file to be, in bytes.
Library:
libc

Use the -l c option to qcc to link against this library. This library is 
usually included automatically.

Description:
These functions cause the file referenced by fildes to have a size of length 
bytes. If the size of the file previously exceeded length, the extra data is 
discarded (this is similar to using the F_FREESP option with fcntl()). If the 
size of the file was previously shorter than length, the file size is extended 
with NUL characters (similar to the F_ALLOCSP option to fcntl()).

The value of the seek pointer isn't modified by a call to ftruncate().

Upon successful completion, the ftruncate() function marks the st_ctime and 
st_mtime fields of the file for update. If the ftruncate() function is 
unsuccessful, the file is unaffected.


The API tells that the time fields of file are updated. This could be the 
triggering point for our issue, not sure though.

Michael/Dan,
Any inputs from your side?

Thanks for all your valuable inputs till now.

Thanks,
Praveen


-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Dan Kennedy
Sent: Friday, October 15, 2010 9:08 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS


On Oct 15, 2010, at 10:24 PM, Black, Michael (IS) wrote:

> I'm not sure but I suspect sqlite is not calling unmap before
> extending the area.
> That would explain why it still gets zeroed out even with the flags.
>
> Put a break point in the unixShmUnmap call and see if it gets called
> before mmap.


We're doing this:

   ftruncate(fd, 32*1024);
   mmap(0, 32*1024, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
   
   ftruncate(fd, 64*1024);
   mmap(0, 32*1024, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 32*1024);

No unmap calls.

Praveen's investigations suggest that the second mmap() call is
zeroing the memory mapped by the first mmap() call. Which is,
as you might expect, confusing SQLite.

I guess it could also be the second ftruncate() call that is
zeroing our mapped memory. That would be even odder though...

Dan.



> May just need some QNX logic that says "if we're extending an area
> unmap it first with flags".
>
> I think the mmap should honor the NOINIT flag when extending an area
> but apparently it doesn't -- only unmap can set the flag to make it
> honor it.  QNX must be about the only one that does this.
>
> QNX mmap claims to be POSIX 1003.1 compliant but I don't see this
> behavior defined in there.
>
> The only zero-fill reference I see in POSIX is
> "The system shall always zero-fill any partial page at the end of an
> object"
> which doesn't fit this behavior at all.
>
> Perhaps you should report this as a bug or non-desirable/non-
> compliant behavior compared to every other OS in the world (and the
> POSIX standard which doesn't call for this behavior).
>
> Michael D. Black
> Senior Scientist
> Advanced Analytics Directorate
> Northrop Grumman Information Systems
>
>
> 
>
> From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
> Sent: Fri 10/15/2010 9:55 AM
> To: General Discussion of SQLite Database
> Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests
> fail on QNX OS
>
>
>
> Hi Michael,
>
> Yes I added the "MAP_NOINIT" to mmap() and "UNMAP_INIT_OPTIONAL"
> flag to munmap_flags() call. Don't know where i might be going wrong
> in SQLite.
>
> As you suggested, I wrote a small application to check if this
> works. Fortunately it worked as desired (as given below).
>
> MAP_NOINIT
> When specified, the POSIX requirement that the memory be zeroed is
> relaxed. The physical memory being used for this allocation must
&g

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Black, Michael (IS)]

I also see where you can set the behavior using procnto -- I'll bet "procnto 
~i" will make sqlite behave correctly.  Though this is a global change.  
Anybody who depends on this zeroing though is nuts...
 
http://www.qnx.com/developers/docs/6.4.0/neutrino/lib_ref/m/munmap_flags.html
 

There are some interactions of the flags argument with the MAP_NOINIT flag of 
the mmap()   
function as well as procnto 
 , as 
detailed below: 

*   The mmap() function has a new flag, MAP_NOINIT. When specified, the 
POSIX requirement that the memory be zeroed is relaxed. The physical memory 
being used for this allocation must have been previously freed with 
UNMAP_INIT_OPTIONAL for this flag to have any effect. 
*   The procnto command line now has an -mmemmgr_configuration option. The 
memmgr_configuration string has a sequence of configuration options to enable 
(or if preceeded with a ~ character, disable) memory-manager aspects. The 
configuation options are: 

i 
munmap_flags() (with flags parameter as zero) acts as if 
UNMAP_INIT_REQUIRED were specified (the default). 
~i 
munmap_flags() (with flags parameter as zero) acts as if 
UNMAP_INIT_OPTIONAL were specified. 

Michael D. Black
Senior Scientist
Advanced Analytics Directorate
Northrop Grumman Information Systems
 

 
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Black, Michael (IS)]

The problem is the lack of the unmap call before the 2nd mmap.
It's redundant on most systems but apparently needed to make QNX happy 
(including the flags we discussed before on both unmap and mmap).
 
So we need
 
   ftruncate(fd, 32*1024);
   mmap(0, 32*1024, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
   
   ftruncate(fd, 64*1024);
#if QNX0..
   munmap_flags(0,32*1024,UNMAP_INIT_OPTIONAL);
   munmap(0,32*1024);
   mmap(0, 32*1024, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_NOINIT, fd, 32*1024);
#else
   map(0, 32*1024, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 32*1024);
#endif


Michael D. Black
Senior Scientist
Advanced Analytics Directorate
Northrop Grumman Information Systems
 



From: sqlite-users-boun...@sqlite.org on behalf of Dan Kennedy
Sent: Fri 10/15/2010 10:37 AM
To: General Discussion of SQLite Database
Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS




On Oct 15, 2010, at 10:24 PM, Black, Michael (IS) wrote:

> I'm not sure but I suspect sqlite is not calling unmap before 
> extending the area.
> That would explain why it still gets zeroed out even with the flags.
>
> Put a break point in the unixShmUnmap call and see if it gets called 
> before mmap.


We're doing this:

   ftruncate(fd, 32*1024);
   mmap(0, 32*1024, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
   
   ftruncate(fd, 64*1024);
   mmap(0, 32*1024, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 32*1024);

No unmap calls.

Praveen's investigations suggest that the second mmap() call is
zeroing the memory mapped by the first mmap() call. Which is,
as you might expect, confusing SQLite.

I guess it could also be the second ftruncate() call that is
zeroing our mapped memory. That would be even odder though...

Dan.



> May just need some QNX logic that says "if we're extending an area 
> unmap it first with flags".
>
> I think the mmap should honor the NOINIT flag when extending an area 
> but apparently it doesn't -- only unmap can set the flag to make it 
> honor it.  QNX must be about the only one that does this.
>
> QNX mmap claims to be POSIX 1003.1 compliant but I don't see this 
> behavior defined in there.
>
> The only zero-fill reference I see in POSIX is
> "The system shall always zero-fill any partial page at the end of an 
> object"
> which doesn't fit this behavior at all.
>
> Perhaps you should report this as a bug or non-desirable/non-
> compliant behavior compared to every other OS in the world (and the 
> POSIX standard which doesn't call for this behavior).
>
> Michael D. Black
> Senior Scientist
> Advanced Analytics Directorate
> Northrop Grumman Information Systems
>
>
> 
>
> From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
> Sent: Fri 10/15/2010 9:55 AM
> To: General Discussion of SQLite Database
> Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests 
> fail on QNX OS
>
>
>
> Hi Michael,
>
> Yes I added the "MAP_NOINIT" to mmap() and "UNMAP_INIT_OPTIONAL" 
> flag to munmap_flags() call. Don't know where i might be going wrong 
> in SQLite.
>
> As you suggested, I wrote a small application to check if this 
> works. Fortunately it worked as desired (as given below).
>
> MAP_NOINIT
> When specified, the POSIX requirement that the memory be zeroed is 
> relaxed. The physical memory being used for this allocation must 
> have been previously freed with UNMAP_INIT_OPTIONAL for this flag to 
> have any effect.
>
> _
> int main(int argc, char *argv[]) {
>printf("Welcome to the QNX Momentics IDE\n");
>
>//Open a file
>int fd = open("/tmp/mmaptest",O_RDWR|O_CREAT, 0777);
>
>//Truncate the file
>int size = ftruncate(fd, 2048);
>
>const char buff[]="Testing the mmap API";
>write(fd,buff,sizeof(buff));
>
>//Mapping the disk file to memory
>void *pMem = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED, fd, 0);
>
>//Check the mmapped memory contents
>printf("Memory Mapped Data: %s\n", pMem);
>
>//Set all mmapped locations to 'A'
>memset(pMem,'A',2048);
>
>//Unmap the memory
>//munmap(pMem, 2048);
>munmap_flags(pMem, 2048, UNMAP_INIT_OPTIONAL);
>
>//Buffer to read disk data
>char disk_read[2048];
>
>//Read the data from the file to check if its synced
>read(fd,disk_read,sizeof(disk_read));
>printf("Reading disk file data:%s\n",disk_read);
>
>//Resetting the contents of disk file
>size = ftruncate(fd, 0);
>close(fd);
>
>//Open the file again to check

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Dan Kennedy]

On Oct 15, 2010, at 10:24 PM, Black, Michael (IS) wrote:

> I'm not sure but I suspect sqlite is not calling unmap before  
> extending the area.
> That would explain why it still gets zeroed out even with the flags.
>
> Put a break point in the unixShmUnmap call and see if it gets called  
> before mmap.


We're doing this:

   ftruncate(fd, 32*1024);
   mmap(0, 32*1024, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
   
   ftruncate(fd, 64*1024);
   mmap(0, 32*1024, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 32*1024);

No unmap calls.

Praveen's investigations suggest that the second mmap() call is
zeroing the memory mapped by the first mmap() call. Which is,
as you might expect, confusing SQLite.

I guess it could also be the second ftruncate() call that is
zeroing our mapped memory. That would be even odder though...

Dan.



> May just need some QNX logic that says "if we're extending an area  
> unmap it first with flags".
>
> I think the mmap should honor the NOINIT flag when extending an area  
> but apparently it doesn't -- only unmap can set the flag to make it  
> honor it.  QNX must be about the only one that does this.
>
> QNX mmap claims to be POSIX 1003.1 compliant but I don't see this  
> behavior defined in there.
>
> The only zero-fill reference I see in POSIX is
> "The system shall always zero-fill any partial page at the end of an  
> object"
> which doesn't fit this behavior at all.
>
> Perhaps you should report this as a bug or non-desirable/non- 
> compliant behavior compared to every other OS in the world (and the  
> POSIX standard which doesn't call for this behavior).
>
> Michael D. Black
> Senior Scientist
> Advanced Analytics Directorate
> Northrop Grumman Information Systems
>
>
> 
>
> From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
> Sent: Fri 10/15/2010 9:55 AM
> To: General Discussion of SQLite Database
> Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests  
> fail on QNX OS
>
>
>
> Hi Michael,
>
> Yes I added the "MAP_NOINIT" to mmap() and "UNMAP_INIT_OPTIONAL"  
> flag to munmap_flags() call. Don't know where i might be going wrong  
> in SQLite.
>
> As you suggested, I wrote a small application to check if this  
> works. Fortunately it worked as desired (as given below).
>
> MAP_NOINIT
> When specified, the POSIX requirement that the memory be zeroed is  
> relaxed. The physical memory being used for this allocation must  
> have been previously freed with UNMAP_INIT_OPTIONAL for this flag to  
> have any effect.
>
> _
> int main(int argc, char *argv[]) {
>printf("Welcome to the QNX Momentics IDE\n");
>
>//Open a file
>int fd = open("/tmp/mmaptest",O_RDWR|O_CREAT, 0777);
>
>//Truncate the file
>int size = ftruncate(fd, 2048);
>
>const char buff[]="Testing the mmap API";
>write(fd,buff,sizeof(buff));
>
>//Mapping the disk file to memory
>void *pMem = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED, fd, 0);
>
>//Check the mmapped memory contents
>printf("Memory Mapped Data: %s\n", pMem);
>
>//Set all mmapped locations to 'A'
>memset(pMem,'A',2048);
>
>//Unmap the memory
>//munmap(pMem, 2048);
>munmap_flags(pMem, 2048, UNMAP_INIT_OPTIONAL);
>
>//Buffer to read disk data
>char disk_read[2048];
>
>//Read the data from the file to check if its synced
>read(fd,disk_read,sizeof(disk_read));
>printf("Reading disk file data:%s\n",disk_read);
>
>//Resetting the contents of disk file
>size = ftruncate(fd, 0);
>close(fd);
>
>//Open the file again to check to re-map the memory
>fd = open("/tmp/mmaptest",O_RDWR, 0777);
>size = ftruncate(fd, 2048);
>
>//Map the memory region again
>//int pMem1 = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED, fd,  
> 0);
>int pMem1 = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED| 
> MAP_NOINIT, fd, 0);
>
>//Print the re-mapped region
>printf("New Memory Mapped Data: %s\n", pMem1);
>
>//Read the disk contents again to check if "MAP_NOINIT" works
>read(fd,disk_read,sizeof(disk_read));
>printf("Disk file data during re-map: %s\n", disk_read);
>
>//Close the disk file
>close(fd);
>
>return EXIT_SUCCESS;
> }
>
> ---
> Output:
>
> Welcome to the QNX Momentics IDE
> Memory Mapped Data: Testing the mma

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Black, Michael (IS)]

I'm not sure but I suspect sqlite is not calling unmap before extending the 
area.
That would explain why it still gets zeroed out even with the flags.
 
Put a break point in the unixShmUnmap call and see if it gets called before 
mmap.
 
May just need some QNX logic that says "if we're extending an area unmap it 
first with flags".
 
I think the mmap should honor the NOINIT flag when extending an area but 
apparently it doesn't -- only unmap can set the flag to make it honor it.  QNX 
must be about the only one that does this.
 
QNX mmap claims to be POSIX 1003.1 compliant but I don't see this behavior 
defined in there.
 
The only zero-fill reference I see in POSIX is 
"The system shall always zero-fill any partial page at the end of an object"
which doesn't fit this behavior at all.
 
Perhaps you should report this as a bug or non-desirable/non-compliant behavior 
compared to every other OS in the world (and the POSIX standard which doesn't 
call for this behavior).
 
Michael D. Black
Senior Scientist
Advanced Analytics Directorate
Northrop Grumman Information Systems
 



From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
Sent: Fri 10/15/2010 9:55 AM
To: General Discussion of SQLite Database
Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS



Hi Michael,

Yes I added the "MAP_NOINIT" to mmap() and "UNMAP_INIT_OPTIONAL" flag to 
munmap_flags() call. Don't know where i might be going wrong in SQLite.

As you suggested, I wrote a small application to check if this works. 
Fortunately it worked as desired (as given below).

MAP_NOINIT
When specified, the POSIX requirement that the memory be zeroed is relaxed. The 
physical memory being used for this allocation must have been previously freed 
with UNMAP_INIT_OPTIONAL for this flag to have any effect.

_
int main(int argc, char *argv[]) {
printf("Welcome to the QNX Momentics IDE\n");

//Open a file
int fd = open("/tmp/mmaptest",O_RDWR|O_CREAT, 0777);

//Truncate the file
int size = ftruncate(fd, 2048);

const char buff[]="Testing the mmap API";
write(fd,buff,sizeof(buff));

//Mapping the disk file to memory
void *pMem = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED, fd, 0);

//Check the mmapped memory contents
printf("Memory Mapped Data: %s\n", pMem);

//Set all mmapped locations to 'A'
memset(pMem,'A',2048);

//Unmap the memory
//munmap(pMem, 2048);
munmap_flags(pMem, 2048, UNMAP_INIT_OPTIONAL);

//Buffer to read disk data
char disk_read[2048];

//Read the data from the file to check if its synced
read(fd,disk_read,sizeof(disk_read));
printf("Reading disk file data:%s\n",disk_read);

//Resetting the contents of disk file
size = ftruncate(fd, 0);
close(fd);

//Open the file again to check to re-map the memory
fd = open("/tmp/mmaptest",O_RDWR, 0777);
size = ftruncate(fd, 2048);

//Map the memory region again
//int pMem1 = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED, fd, 0);
int pMem1 = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED|MAP_NOINIT, fd, 
0);

//Print the re-mapped region
printf("New Memory Mapped Data: %s\n", pMem1);

//Read the disk contents again to check if "MAP_NOINIT" works
read(fd,disk_read,sizeof(disk_read));
printf("Disk file data during re-map: %s\n", disk_read);

//Close the disk file
close(fd);

return EXIT_SUCCESS;
}

---
Output:

Welcome to the QNX Momentics IDE
Memory Mapped Data: Testing the mmap API
Reading disk file data:AA [2048 times]
New Memory Mapped Data: A [2048 times]
Disk file data during re-map:
---

Also found that the mmapped data is written back to disk file on 
munmap()/munmap_flags() call.

Maybe i need to look into Sqlite amalgamation file in detail to analyze what's 
happening here during unmapping of memory.
I believe we can use these flags under all scenarios without any 
pre-conditions. Not sure if POSIX (on QNX) have some limitations on their usage.

Thanks,
Praveen
__

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Black, Michael (IS)
Sent: Thursday, October 14, 2010 9:22 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

Did you also add the MAP_NOINIT to the mmap() call?

It sounds like exactly the behavior you're seeing.

Di

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Raj, Praveen]

Hi Michael,

Yes I added the "MAP_NOINIT" to mmap() and "UNMAP_INIT_OPTIONAL" flag to 
munmap_flags() call. Don't know where i might be going wrong in SQLite.

As you suggested, I wrote a small application to check if this works. 
Fortunately it worked as desired (as given below).

MAP_NOINIT
When specified, the POSIX requirement that the memory be zeroed is relaxed. The 
physical memory being used for this allocation must have been previously freed 
with UNMAP_INIT_OPTIONAL for this flag to have any effect.

_
int main(int argc, char *argv[]) {
printf("Welcome to the QNX Momentics IDE\n");

//Open a file
int fd = open("/tmp/mmaptest",O_RDWR|O_CREAT, 0777);

//Truncate the file
int size = ftruncate(fd, 2048);

const char buff[]="Testing the mmap API";
write(fd,buff,sizeof(buff));

//Mapping the disk file to memory
void *pMem = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED, fd, 0);

//Check the mmapped memory contents
printf("Memory Mapped Data: %s\n", pMem);

//Set all mmapped locations to 'A'
memset(pMem,'A',2048);

//Unmap the memory
//munmap(pMem, 2048);
munmap_flags(pMem, 2048, UNMAP_INIT_OPTIONAL);

//Buffer to read disk data
char disk_read[2048];

//Read the data from the file to check if its synced
read(fd,disk_read,sizeof(disk_read));
printf("Reading disk file data:%s\n",disk_read);

//Resetting the contents of disk file
size = ftruncate(fd, 0);
close(fd);

//Open the file again to check to re-map the memory 
fd = open("/tmp/mmaptest",O_RDWR, 0777);
size = ftruncate(fd, 2048);

//Map the memory region again
//int pMem1 = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED, fd, 0);
int pMem1 = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED|MAP_NOINIT, fd, 
0);

//Print the re-mapped region
printf("New Memory Mapped Data: %s\n", pMem1);

//Read the disk contents again to check if "MAP_NOINIT" works
read(fd,disk_read,sizeof(disk_read));
printf("Disk file data during re-map: %s\n", disk_read);

//Close the disk file
close(fd);

return EXIT_SUCCESS;
}

---
Output:

Welcome to the QNX Momentics IDE
Memory Mapped Data: Testing the mmap API
Reading disk file data:AA [2048 times]
New Memory Mapped Data: A [2048 times]
Disk file data during re-map:
---

Also found that the mmapped data is written back to disk file on 
munmap()/munmap_flags() call.

Maybe i need to look into Sqlite amalgamation file in detail to analyze what's 
happening here during unmapping of memory. 
I believe we can use these flags under all scenarios without any 
pre-conditions. Not sure if POSIX (on QNX) have some limitations on their usage.

Thanks,
Praveen
__

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Black, Michael (IS)
Sent: Thursday, October 14, 2010 9:22 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

Did you also add the MAP_NOINIT to the mmap() call?
 
It sounds like exactly the behavior you're seeing. 
 
Did you try writing  a stand-alone app to test this idea?
 
Michael D. Black
Senior Scientist
Advanced Analytics Directorate
Northrop Grumman Information Systems
 



From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
Sent: Thu 10/14/2010 10:44 AM
To: General Discussion of SQLite Database
Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS



Hi Micheal,

Thanks Dan and Michael for all your inputs.

I tried this approach as well, but didn't find any success.
During unmapping i used the API munmap_flags() with "UNMAP_INIT_OPTIONAL" flag 
to avoid the zero initialization during the next mmaping.

Another thought I have here is that the old mmapped regions may not be 
initialized with zeros, but instead the regions are getting synced with the 
data in disk file (which is full of zeros). Not sure if can happen with mmap() 
API though?


Thanks,
Praveen

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Black, Michael (IS)
Sent: Thursday, October 14, 2010 5:21 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

I sent this before...have you tried this?

According to the QNX mmap page
http://www.qnx.com/developers/docs/6.3.0SP3/n

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Black, Michael (IS)]

Did you also add the MAP_NOINIT to the mmap() call?
 
It sounds like exactly the behavior you're seeing. 
 
Did you try writing  a stand-alone app to test this idea?
 
Michael D. Black
Senior Scientist
Advanced Analytics Directorate
Northrop Grumman Information Systems
 



From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
Sent: Thu 10/14/2010 10:44 AM
To: General Discussion of SQLite Database
Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS



Hi Micheal,

Thanks Dan and Michael for all your inputs.

I tried this approach as well, but didn't find any success.
During unmapping i used the API munmap_flags() with "UNMAP_INIT_OPTIONAL" flag 
to avoid the zero initialization during the next mmaping.

Another thought I have here is that the old mmapped regions may not be 
initialized with zeros, but instead the regions are getting synced with
the data in disk file (which is full of zeros). Not sure if can happen with 
mmap() API though?


Thanks,
Praveen

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Black, Michael (IS)
Sent: Thursday, October 14, 2010 5:21 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

I sent this before...have you tried this?

According to the QNX mmap page
http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/lib_ref/m/mmap.html 
<https://owa1.ngc.com/exchweb/bin/redir.asp?URL=http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/lib_ref/m/mmap.html>

MAP_NOINIT
When specified, the POSIX requirement that the memory be zeroed is relaxed. The 
physical memory being used for this allocation must have been previously freed 
with UNMAP_INIT_OPTIONAL for this flag to have any effect.


This flag was added in the QNX Neutrino Core OS 6.3.2. 

Interesting that this claims it's a POSIX requirement but I don't think most 
any others do this.


Michael D. Black
Senior Scientist
Advanced Analytics Directorate
Northrop Grumman Information Systems




From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
Sent: Thu 10/14/2010 5:53 AM
To: General Discussion of SQLite Database
Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS



Hi Dan,

I did some more investigation on the issue and i feel there is synchronization 
problem happening here.

After mmapping the shm (wal index) file to process memory, the WAL indexes are 
written into the mmapped area, and this data is not getting synchronized with 
physical (shm) file. As a result when the mmap() function is called the second 
time to map the 32k-64k memory region, it is synchronizing the complete mmapped 
region (previous 32k regions) with physical file, even though a valid offset is 
passed. Not sure if this is the actual behaviour of mmap() call.

While debugging, before the mmap() call i checked mmapped region and it had 
valid indexes, whereas after the call all became 0's. Also i found that the shm 
file is always filled with 0's even after commits.

When i added the msync() statement (to sync the shm file) before mmap call as 
shown below, the problem is not seen. In this case the shm file has valid 
32-bit indexes, as data is synchronized before next mmap call is executed.

while(pShmNode->nRegion<=iRegion){
  int ret = msync( apNew[0], iRegion*szRegion, MS_SYNC);
  void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
  MAP_SHARED, pShmNode->h, pShmNode->nRegion*szRegion
  );

With the above msync() call all my failed test cases are passing.

I don't see any msync() call in the SQLite amalgamation/wal.c file.
I believe the data in mapped region and physical file are not synched 
automatically. We need to explicitly do it using msync() call.
Don't know if there is any other mechanism in SQLite through which the data is 
synchronized. Does the call to sqlite3OsSync() sync the shm file as well? or is 
the shm file not syned purposefully?

This is all my understanding and not sure if this is causing the actual issue. 
Please guide me if my approach/understanding is incorrect.


Thanks,
Praveen

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Dan Kennedy
Sent: Friday, October 08, 2010 9:33 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS


On Oct 8, 2010, at 9:44 PM, Raj, Praveen wrote:

> Hello,
>
> I debugged the SQLite functions and here is my finding:
>
> The call to "mmap" in the function "unixShmMap" is causing the issue.
> void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
>  MAP_SHARED, pShmNode->h, iRegion*szRegion);
>
> It is setting the previous memory region/regio

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Raj, Praveen]

Hi Micheal,

Thanks Dan and Michael for all your inputs.

I tried this approach as well, but didn't find any success.
During unmapping i used the API munmap_flags() with "UNMAP_INIT_OPTIONAL" flag 
to avoid the zero initialization during the next mmaping.

Another thought I have here is that the old mmapped regions may not be 
initialized with zeros, but instead the regions are getting synced with
the data in disk file (which is full of zeros). Not sure if can happen with 
mmap() API though?


Thanks,
Praveen

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Black, Michael (IS)
Sent: Thursday, October 14, 2010 5:21 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

I sent this before...have you tried this?
 
According to the QNX mmap page
http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/lib_ref/m/mmap.html 
<https://owa1.ngc.com/exchweb/bin/redir.asp?URL=http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/lib_ref/m/mmap.html>
 
 
MAP_NOINIT
When specified, the POSIX requirement that the memory be zeroed is relaxed. The 
physical memory being used for this allocation must have been previously freed 
with UNMAP_INIT_OPTIONAL for this flag to have any effect. 


This flag was added in the QNX Neutrino Core OS 6.3.2.  

Interesting that this claims it's a POSIX requirement but I don't think most 
any others do this.

 
Michael D. Black
Senior Scientist
Advanced Analytics Directorate
Northrop Grumman Information Systems
 



From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
Sent: Thu 10/14/2010 5:53 AM
To: General Discussion of SQLite Database
Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS



Hi Dan,

I did some more investigation on the issue and i feel there is synchronization 
problem happening here.

After mmapping the shm (wal index) file to process memory, the WAL indexes are 
written into the mmapped area, and this data is not getting synchronized with 
physical (shm) file. As a result when the mmap() function is called the second 
time to map the 32k-64k memory region, it is synchronizing the complete mmapped 
region (previous 32k regions) with physical file, even though a valid offset is 
passed. Not sure if this is the actual behaviour of mmap() call.

While debugging, before the mmap() call i checked mmapped region and it had 
valid indexes, whereas after the call all became 0's. Also i found that the shm 
file is always filled with 0's even after commits.

When i added the msync() statement (to sync the shm file) before mmap call as 
shown below, the problem is not seen. In this case the shm file has valid 
32-bit indexes, as data is synchronized before next mmap call is executed.

while(pShmNode->nRegion<=iRegion){
  int ret = msync( apNew[0], iRegion*szRegion, MS_SYNC);
  void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
  MAP_SHARED, pShmNode->h, pShmNode->nRegion*szRegion
  );

With the above msync() call all my failed test cases are passing.

I don't see any msync() call in the SQLite amalgamation/wal.c file.
I believe the data in mapped region and physical file are not synched 
automatically. We need to explicitly do it using msync() call.
Don't know if there is any other mechanism in SQLite through which the data is 
synchronized. Does the call to sqlite3OsSync() sync the shm file as well? or is 
the shm file not syned purposefully?

This is all my understanding and not sure if this is causing the actual issue. 
Please guide me if my approach/understanding is incorrect.


Thanks,
Praveen

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Dan Kennedy
Sent: Friday, October 08, 2010 9:33 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS


On Oct 8, 2010, at 9:44 PM, Raj, Praveen wrote:

> Hello,
>
> I debugged the SQLite functions and here is my finding:
>
> The call to "mmap" in the function "unixShmMap" is causing the issue.
> void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
>  MAP_SHARED, pShmNode->h, iRegion*szRegion);
>
> It is setting the previous memory region/regions to zero while mapping 
> the new ones. Mmap call internally uses the QNX API mmap64() to map 
> the required memory region. Not sure on what is happening here. Just 
> need to dig into memory mapping to find whats happening and hopefully 
> find a solution.
>
> Dan - Do you have any idea on why this could be happening?

Sounds like a bug in QNX to me.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sq

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Black, Michael (IS)]

I sent this before...have you tried this?
 
According to the QNX mmap page
http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/lib_ref/m/mmap.html 
<https://owa1.ngc.com/exchweb/bin/redir.asp?URL=http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/lib_ref/m/mmap.html>
 
 
MAP_NOINIT 
When specified, the POSIX requirement that the memory be zeroed is relaxed. The 
physical memory being used for this allocation must have been previously freed 
with UNMAP_INIT_OPTIONAL for this flag to have any effect. 


This flag was added in the QNX Neutrino Core OS 6.3.2.  

Interesting that this claims it's a POSIX requirement but I don't think most 
any others do this.

 
Michael D. Black
Senior Scientist
Advanced Analytics Directorate
Northrop Grumman Information Systems
 



From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
Sent: Thu 10/14/2010 5:53 AM
To: General Discussion of SQLite Database
Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS



Hi Dan,

I did some more investigation on the issue and i feel there is synchronization 
problem happening here.

After mmapping the shm (wal index) file to process memory, the WAL indexes are 
written into the mmapped area, and this data is not getting synchronized with 
physical (shm) file. As a result when the mmap() function is called the second 
time to map the 32k-64k memory region, it is synchronizing the complete mmapped 
region (previous 32k regions) with physical file, even though a valid offset is 
passed. Not sure if this is the actual behaviour of mmap() call.

While debugging, before the mmap() call i checked mmapped region and it had 
valid indexes, whereas after the call all became 0's. Also i found that the shm 
file is always filled with 0's even after commits.

When i added the msync() statement (to sync the shm file) before mmap call as 
shown below, the problem is not seen. In this case the shm file has valid 
32-bit indexes, as data is synchronized before next mmap call is executed.

while(pShmNode->nRegion<=iRegion){
  int ret = msync( apNew[0], iRegion*szRegion, MS_SYNC);
  void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
  MAP_SHARED, pShmNode->h, pShmNode->nRegion*szRegion
  );

With the above msync() call all my failed test cases are passing.

I don't see any msync() call in the SQLite amalgamation/wal.c file.
I believe the data in mapped region and physical file are not synched 
automatically. We need to explicitly do it using msync() call.
Don't know if there is any other mechanism in SQLite through which the data is 
synchronized. Does the call to sqlite3OsSync() sync
the shm file as well? or is the shm file not syned purposefully?

This is all my understanding and not sure if this is causing the actual issue. 
Please guide me if my approach/understanding is incorrect.


Thanks,
Praveen

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Dan Kennedy
Sent: Friday, October 08, 2010 9:33 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS


On Oct 8, 2010, at 9:44 PM, Raj, Praveen wrote:

> Hello,
>
> I debugged the SQLite functions and here is my finding:
>
> The call to "mmap" in the function "unixShmMap" is causing the issue.
> void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
>  MAP_SHARED, pShmNode->h, iRegion*szRegion);
>
> It is setting the previous memory region/regions to zero while
> mapping the new ones. Mmap call internally uses the QNX API mmap64()
> to map the required memory region. Not sure on what is happening
> here. Just need to dig into memory mapping to find whats happening
> and hopefully find a solution.
>
> Dan - Do you have any idea on why this could be happening?

Sounds like a bug in QNX to me.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

The information contained in this message may be confidential and legally 
protected under applicable law. The message is intended solely for the 
addressee(s). If you are not the intended recipient, you are hereby notified 
that any use, forwarding, dissemination, or reproduction of this message is 
strictly prohibited and may be unlawful. If you are not the intended recipient, 
please contact the sender by return e-mail and destroy all copies of the 
original message.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Dan Kennedy]

On Oct 14, 2010, at 5:53 PM, Raj, Praveen wrote:

> Hi Dan,
>
> I did some more investigation on the issue and i feel there is  
> synchronization problem happening here.
>
> After mmapping the shm (wal index) file to process memory, the WAL  
> indexes are written into the mmapped area, and this data is not  
> getting synchronized with physical (shm) file. As a result when the  
> mmap() function is called the second time to map the 32k-64k memory  
> region, it is synchronizing the complete mmapped region (previous  
> 32k regions) with physical file, even though a valid offset is  
> passed. Not sure if this is the actual behaviour of mmap() call.
>
> While debugging, before the mmap() call i checked mmapped region and  
> it had valid indexes, whereas after the call all became 0's. Also i  
> found that the shm file is always filled with 0's even after commits.
>
> When i added the msync() statement (to sync the shm file) before  
> mmap call as shown below, the problem is not seen. In this case the  
> shm file has valid 32-bit indexes, as data is synchronized before  
> next mmap call is executed.
>
>while(pShmNode->nRegion<=iRegion){
>  int ret = msync( apNew[0], iRegion*szRegion, MS_SYNC);
>  void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
>  MAP_SHARED, pShmNode->h, pShmNode->nRegion*szRegion
>  );
>
> With the above msync() call all my failed test cases are passing.
>
> I don't see any msync() call in the SQLite amalgamation/wal.c file.
> I believe the data in mapped region and physical file are not  
> synched automatically. We need to explicitly do it using msync() call.
> Don't know if there is any other mechanism in SQLite through which  
> the data is synchronized. Does the call to sqlite3OsSync() sync
> the shm file as well? or is the shm file not syned purposefully?

It's true that the mapped region and physical file are not
synced automatically. But mmap() still should not be zeroing
regions that have already been mapped. This is a bug in mmap().

We don't sync it because we don't care if that file is
written to persistent storage or not. The only reason
we use a file located next to the database in the file-system
instead of in /tmp or something is because it happens to be
a convenient way to make sure all clients access the same
shared memory. See the section entitled "Implementation Of
Shared-Memory For The WAL-Index" here for more details:

   http://www.sqlite.org/wal.html

It seems plausible that adding the msync() might work around
the mmap() problem. Hard to be really confident though - there
may be race conditions lurking...

Thanks for looking into this.

Dan.





>
> This is all my understanding and not sure if this is causing the  
> actual issue. Please guide me if my approach/understanding is  
> incorrect.
>
>
> Thanks,
> Praveen
>
> -Original Message-
> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org 
> ] On Behalf Of Dan Kennedy
> Sent: Friday, October 08, 2010 9:33 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX  
> OS
>
>
> On Oct 8, 2010, at 9:44 PM, Raj, Praveen wrote:
>
>> Hello,
>>
>> I debugged the SQLite functions and here is my finding:
>>
>> The call to "mmap" in the function "unixShmMap" is causing the issue.
>> void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
>> MAP_SHARED, pShmNode->h, iRegion*szRegion);
>>
>> It is setting the previous memory region/regions to zero while
>> mapping the new ones. Mmap call internally uses the QNX API mmap64()
>> to map the required memory region. Not sure on what is happening
>> here. Just need to dig into memory mapping to find whats happening
>> and hopefully find a solution.
>>
>> Dan - Do you have any idea on why this could be happening?
>
> Sounds like a bug in QNX to me.
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
> The information contained in this message may be confidential and  
> legally protected under applicable law. The message is intended  
> solely for the addressee(s). If you are not the intended recipient,  
> you are hereby notified that any use, forwarding, dissemination, or  
> reproduction of this message is strictly prohibited and may be  
> unlawful. If you are not the intended recipient, please contact the  
> sender by return e-mail and destroy all copies of the original  
> message.
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Raj, Praveen]

Hi Dan,

I did some more investigation on the issue and i feel there is synchronization 
problem happening here.

After mmapping the shm (wal index) file to process memory, the WAL indexes are 
written into the mmapped area, and this data is not getting synchronized with 
physical (shm) file. As a result when the mmap() function is called the second 
time to map the 32k-64k memory region, it is synchronizing the complete mmapped 
region (previous 32k regions) with physical file, even though a valid offset is 
passed. Not sure if this is the actual behaviour of mmap() call.

While debugging, before the mmap() call i checked mmapped region and it had 
valid indexes, whereas after the call all became 0's. Also i found that the shm 
file is always filled with 0's even after commits.

When i added the msync() statement (to sync the shm file) before mmap call as 
shown below, the problem is not seen. In this case the shm file has valid 
32-bit indexes, as data is synchronized before next mmap call is executed.

while(pShmNode->nRegion<=iRegion){
  int ret = msync( apNew[0], iRegion*szRegion, MS_SYNC);
  void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
  MAP_SHARED, pShmNode->h, pShmNode->nRegion*szRegion
  );

With the above msync() call all my failed test cases are passing.

I don't see any msync() call in the SQLite amalgamation/wal.c file.
I believe the data in mapped region and physical file are not synched 
automatically. We need to explicitly do it using msync() call.
Don't know if there is any other mechanism in SQLite through which the data is 
synchronized. Does the call to sqlite3OsSync() sync
the shm file as well? or is the shm file not syned purposefully?

This is all my understanding and not sure if this is causing the actual issue. 
Please guide me if my approach/understanding is incorrect.


Thanks,
Praveen

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Dan Kennedy
Sent: Friday, October 08, 2010 9:33 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS


On Oct 8, 2010, at 9:44 PM, Raj, Praveen wrote:

> Hello,
>
> I debugged the SQLite functions and here is my finding:
>
> The call to "mmap" in the function "unixShmMap" is causing the issue.
> void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
>  MAP_SHARED, pShmNode->h, iRegion*szRegion);
>
> It is setting the previous memory region/regions to zero while
> mapping the new ones. Mmap call internally uses the QNX API mmap64()
> to map the required memory region. Not sure on what is happening
> here. Just need to dig into memory mapping to find whats happening
> and hopefully find a solution.
>
> Dan - Do you have any idea on why this could be happening?

Sounds like a bug in QNX to me.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

The information contained in this message may be confidential and legally 
protected under applicable law. The message is intended solely for the 
addressee(s). If you are not the intended recipient, you are hereby notified 
that any use, forwarding, dissemination, or reproduction of this message is 
strictly prohibited and may be unlawful. If you are not the intended recipient, 
please contact the sender by return e-mail and destroy all copies of the 
original message.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Dan Kennedy]

On Oct 8, 2010, at 9:44 PM, Raj, Praveen wrote:

> Hello,
>
> I debugged the SQLite functions and here is my finding:
>
> The call to "mmap" in the function "unixShmMap" is causing the issue.
> void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
>  MAP_SHARED, pShmNode->h, iRegion*szRegion);
>
> It is setting the previous memory region/regions to zero while  
> mapping the new ones. Mmap call internally uses the QNX API mmap64()  
> to map the required memory region. Not sure on what is happening  
> here. Just need to dig into memory mapping to find whats happening  
> and hopefully find a solution.
>
> Dan - Do you have any idea on why this could be happening?

Sounds like a bug in QNX to me.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Black, Michael (IS)]

According to the QNX mmap page
http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/lib_ref/m/mmap.html
 
MAP_NOINIT
When specified, the POSIX requirement that the memory be zeroed is relaxed. The 
physical memory being used for this allocation must have been previously freed 
with UNMAP_INIT_OPTIONAL for this flag to have any effect. 


This flag was added in the QNX Neutrino Core OS 6.3.2.  

Interesting that this claims it's a POSIX requirement but I don't think most 
any others do this.

Michael D. Black
Senior Scientist
Advanced Analytics Directorate
Northrop Grumman Information Systems
 



From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
Sent: Fri 10/8/2010 9:44 AM
To: General Discussion of SQLite Database
Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS



Hello,

I debugged the SQLite functions and here is my finding:

The call to "mmap" in the function "unixShmMap" is causing the issue.
void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
  MAP_SHARED, pShmNode->h, iRegion*szRegion);

It is setting the previous memory region/regions to zero while mapping the new 
ones. Mmap call internally uses the QNX API mmap64() to map the required memory 
region. Not sure on what is happening here. Just need to dig into memory 
mapping to find whats happening and hopefully find a solution.

Dan - Do you have any idea on why this could be happening?

Thanks,
Praveen


-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Dan Kennedy
Sent: Wednesday, October 06, 2010 9:05 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS


On Oct 6, 2010, at 9:39 PM, Raj, Praveen wrote:

> Hi Dan,
>
> For debugging purpose we use the QNX Momentics which is the IDE
> provided by QNX. I believe to certain extent
> we should be able to debug the issue using IDE. In what situations
> does the shared memory block be zeroed.
> I believe it should be zeroed when the db is closed or when WAL file
> is checkpointed. Any other scenarios?

Following a checkpoint when the writer wraps around to
the start of the log file.

> Please provide pointers on what needs to be checked along with
> "unixShmMap()" when the problem occurs.

There is only one call to sqlite3OsShmMap() in wal.c. Try
to stop the debugger when this function is called with iPage==1.

At this point there should be a pointer to the first 32KB of
mapped memory in pWal->apWiData[0]. It should not be full of
zeroes as it is in the dumps that you sent. Check if this is
the case. More specifically - 100 bytes or so into the buffer
should not contain all zeroes. It should contain a bunch of
32-bit non-zero frame numbers.

Then check if it is still the case after the call to
sqlite3OsShmMap() has returned. Either way, you're looking
for the point where that buffer pWal->apWiData[0] is
overwritten with zeroes.

Dan.



> Thanks,
> Praveen
>
> -Original Message-
> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org
> ] On Behalf Of Dan Kennedy
> Sent: Wednesday, October 06, 2010 7:37 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
> OS
>
>
> On Oct 6, 2010, at 8:15 PM, Raj, Praveen wrote:
>
>> Hello,
>>
>> I'm new to SQLite, so not sure what is the behavior of these failed
>> tests on Unix/Linux. Are they successfully passing all the WAL
>> journal mode tests on Unix?
>>
>> What could be happening when the shared memory (-shm file) is
>> increased from 32k to 64k?
>> Is a new 64k memory chunk allocated and old 32k data is copied here
>> and then 32k is purged?
>
> At some point in the transaction, the first 32KB of shared-memory
> are being zeroed. I'm not sure why this would happen.
>
> The WAL file looks Ok. And the header in the shared-memory looks
> Ok. But most of the first 32KB of shared-memory has been zeroed
> (it is likely that the whole thing was zeroed and then the header
> rewritten).
>
> Maybe something about running the unixShmMap() function on QNX to
> mmap() the second 32KB of space (we use a separate mapping
> for each 32KB block) is zeroing the first. Somehow. Do you have
> a good debugger for the platform?
>
> Dan.
>
>
>
>
>> Any insights on whats happening here would be of great help.
>>
>> Thanks,
>> Praveen
>>
>> -Original Message-
>> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org
>> ] On Behalf Of Raj, Praveen
>> Sent: Sunday, October 03, 2010 7:42 PM
>> To: General Discussion of 

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Raj, Praveen]

Hello,

I debugged the SQLite functions and here is my finding:

The call to "mmap" in the function "unixShmMap" is causing the issue.
void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
  MAP_SHARED, pShmNode->h, iRegion*szRegion);

It is setting the previous memory region/regions to zero while mapping the new 
ones. Mmap call internally uses the QNX API mmap64() to map the required memory 
region. Not sure on what is happening here. Just need to dig into memory 
mapping to find whats happening and hopefully find a solution.

Dan - Do you have any idea on why this could be happening?

Thanks,
Praveen


-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Dan Kennedy
Sent: Wednesday, October 06, 2010 9:05 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS


On Oct 6, 2010, at 9:39 PM, Raj, Praveen wrote:

> Hi Dan,
>
> For debugging purpose we use the QNX Momentics which is the IDE
> provided by QNX. I believe to certain extent
> we should be able to debug the issue using IDE. In what situations
> does the shared memory block be zeroed.
> I believe it should be zeroed when the db is closed or when WAL file
> is checkpointed. Any other scenarios?

Following a checkpoint when the writer wraps around to
the start of the log file.

> Please provide pointers on what needs to be checked along with
> "unixShmMap()" when the problem occurs.

There is only one call to sqlite3OsShmMap() in wal.c. Try
to stop the debugger when this function is called with iPage==1.

At this point there should be a pointer to the first 32KB of
mapped memory in pWal->apWiData[0]. It should not be full of
zeroes as it is in the dumps that you sent. Check if this is
the case. More specifically - 100 bytes or so into the buffer
should not contain all zeroes. It should contain a bunch of
32-bit non-zero frame numbers.

Then check if it is still the case after the call to
sqlite3OsShmMap() has returned. Either way, you're looking
for the point where that buffer pWal->apWiData[0] is
overwritten with zeroes.

Dan.



> Thanks,
> Praveen
>
> -Original Message-
> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org
> ] On Behalf Of Dan Kennedy
> Sent: Wednesday, October 06, 2010 7:37 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
> OS
>
>
> On Oct 6, 2010, at 8:15 PM, Raj, Praveen wrote:
>
>> Hello,
>>
>> I'm new to SQLite, so not sure what is the behavior of these failed
>> tests on Unix/Linux. Are they successfully passing all the WAL
>> journal mode tests on Unix?
>>
>> What could be happening when the shared memory (-shm file) is
>> increased from 32k to 64k?
>> Is a new 64k memory chunk allocated and old 32k data is copied here
>> and then 32k is purged?
>
> At some point in the transaction, the first 32KB of shared-memory
> are being zeroed. I'm not sure why this would happen.
>
> The WAL file looks Ok. And the header in the shared-memory looks
> Ok. But most of the first 32KB of shared-memory has been zeroed
> (it is likely that the whole thing was zeroed and then the header
> rewritten).
>
> Maybe something about running the unixShmMap() function on QNX to
> mmap() the second 32KB of space (we use a separate mapping
> for each 32KB block) is zeroing the first. Somehow. Do you have
> a good debugger for the platform?
>
> Dan.
>
>
>
>
>> Any insights on whats happening here would be of great help.
>>
>> Thanks,
>> Praveen
>>
>> -Original Message-
>> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org
>> ] On Behalf Of Raj, Praveen
>> Sent: Sunday, October 03, 2010 7:42 PM
>> To: General Discussion of SQLite Database
>> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
>> OS
>>
>> Hi,
>>
>> Yes the files sizes of "test.db", "test.db-wal" and "test.db-shm"
>> are 1024, 4333512 and 65536 bytes respectively as specified below.
>>
>> Most of the test scripts are failing when the -shm file size grows
>> from 32kb to 64kb (though the "test.db-wal"
>> file size varies for different test scripts).
>> But there are cases in WAL tests where the size increases to 64kb,
>> but the problem is not seen.
>> Is there any problem while allocating/increasing shared memory, or
>> is the WAL page indexes getting corrupted when shared memory
>> increases?
>>
>> Please suggest some tips to find out the root cause for this issue.
>

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Dan Kennedy]

On Oct 6, 2010, at 9:39 PM, Raj, Praveen wrote:

> Hi Dan,
>
> For debugging purpose we use the QNX Momentics which is the IDE  
> provided by QNX. I believe to certain extent
> we should be able to debug the issue using IDE. In what situations  
> does the shared memory block be zeroed.
> I believe it should be zeroed when the db is closed or when WAL file  
> is checkpointed. Any other scenarios?

Following a checkpoint when the writer wraps around to
the start of the log file.

> Please provide pointers on what needs to be checked along with  
> "unixShmMap()" when the problem occurs.

There is only one call to sqlite3OsShmMap() in wal.c. Try
to stop the debugger when this function is called with iPage==1.

At this point there should be a pointer to the first 32KB of
mapped memory in pWal->apWiData[0]. It should not be full of
zeroes as it is in the dumps that you sent. Check if this is
the case. More specifically - 100 bytes or so into the buffer
should not contain all zeroes. It should contain a bunch of
32-bit non-zero frame numbers.

Then check if it is still the case after the call to
sqlite3OsShmMap() has returned. Either way, you're looking
for the point where that buffer pWal->apWiData[0] is
overwritten with zeroes.

Dan.



> Thanks,
> Praveen
>
> -Original Message-
> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org 
> ] On Behalf Of Dan Kennedy
> Sent: Wednesday, October 06, 2010 7:37 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX  
> OS
>
>
> On Oct 6, 2010, at 8:15 PM, Raj, Praveen wrote:
>
>> Hello,
>>
>> I'm new to SQLite, so not sure what is the behavior of these failed
>> tests on Unix/Linux. Are they successfully passing all the WAL
>> journal mode tests on Unix?
>>
>> What could be happening when the shared memory (-shm file) is
>> increased from 32k to 64k?
>> Is a new 64k memory chunk allocated and old 32k data is copied here
>> and then 32k is purged?
>
> At some point in the transaction, the first 32KB of shared-memory
> are being zeroed. I'm not sure why this would happen.
>
> The WAL file looks Ok. And the header in the shared-memory looks
> Ok. But most of the first 32KB of shared-memory has been zeroed
> (it is likely that the whole thing was zeroed and then the header
> rewritten).
>
> Maybe something about running the unixShmMap() function on QNX to
> mmap() the second 32KB of space (we use a separate mapping
> for each 32KB block) is zeroing the first. Somehow. Do you have
> a good debugger for the platform?
>
> Dan.
>
>
>
>
>> Any insights on whats happening here would be of great help.
>>
>> Thanks,
>> Praveen
>>
>> -----Original Message-
>> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org
>> ] On Behalf Of Raj, Praveen
>> Sent: Sunday, October 03, 2010 7:42 PM
>> To: General Discussion of SQLite Database
>> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
>> OS
>>
>> Hi,
>>
>> Yes the files sizes of "test.db", "test.db-wal" and "test.db-shm"
>> are 1024, 4333512 and 65536 bytes respectively as specified below.
>>
>> Most of the test scripts are failing when the -shm file size grows
>> from 32kb to 64kb (though the "test.db-wal"
>> file size varies for different test scripts).
>> But there are cases in WAL tests where the size increases to 64kb,
>> but the problem is not seen.
>> Is there any problem while allocating/increasing shared memory, or
>> is the WAL page indexes getting corrupted when shared memory
>> increases?
>>
>> Please suggest some tips to find out the root cause for this issue.
>>
>> Thanks,
>> Praveen
>> 
>> From: sqlite-users-boun...@sqlite.org [sqlite-users-
>> boun...@sqlite.org] On Behalf Of Dan Kennedy [danielk1...@gmail.com]
>> Sent: Saturday, October 02, 2010 2:39 PM
>> To: General Discussion of SQLite Database
>> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
>> OS
>>
>> On Oct 1, 2010, at 2:20 PM, Raj, Praveen wrote:
>>
>>> Hi,
>>>
>>> I'm validating SQLite test suite version 3.7.2 on QNX operating
>>> system. I have built the testfixture using SQLite amalgation file
>>> and other related files/libraries.
>>>
>>> I ran the full test suite and most of the test cases are passing,
>>> except some run in journal_mode = WAL. Test cases whose
>>> autocheckpoint is 

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Raj, Praveen]

Hi Dan,

For debugging purpose we use the QNX Momentics which is the IDE provided by 
QNX. I believe to certain extent
we should be able to debug the issue using IDE. In what situations does the 
shared memory block be zeroed.
I believe it should be zeroed when the db is closed or when WAL file is 
checkpointed. Any other scenarios?

Please provide pointers on what needs to be checked along with "unixShmMap()" 
when the problem occurs.

Thanks,
Praveen

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Dan Kennedy
Sent: Wednesday, October 06, 2010 7:37 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS


On Oct 6, 2010, at 8:15 PM, Raj, Praveen wrote:

> Hello,
>
> I'm new to SQLite, so not sure what is the behavior of these failed
> tests on Unix/Linux. Are they successfully passing all the WAL
> journal mode tests on Unix?
>
> What could be happening when the shared memory (-shm file) is
> increased from 32k to 64k?
> Is a new 64k memory chunk allocated and old 32k data is copied here
> and then 32k is purged?

At some point in the transaction, the first 32KB of shared-memory
are being zeroed. I'm not sure why this would happen.

The WAL file looks Ok. And the header in the shared-memory looks
Ok. But most of the first 32KB of shared-memory has been zeroed
(it is likely that the whole thing was zeroed and then the header
rewritten).

Maybe something about running the unixShmMap() function on QNX to
mmap() the second 32KB of space (we use a separate mapping
for each 32KB block) is zeroing the first. Somehow. Do you have
a good debugger for the platform?

Dan.




> Any insights on whats happening here would be of great help.
>
> Thanks,
> Praveen
>
> -Original Message-
> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org
> ] On Behalf Of Raj, Praveen
> Sent: Sunday, October 03, 2010 7:42 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
> OS
>
> Hi,
>
> Yes the files sizes of "test.db", "test.db-wal" and "test.db-shm"
> are 1024, 4333512 and 65536 bytes respectively as specified below.
>
> Most of the test scripts are failing when the -shm file size grows
> from 32kb to 64kb (though the "test.db-wal"
> file size varies for different test scripts).
> But there are cases in WAL tests where the size increases to 64kb,
> but the problem is not seen.
> Is there any problem while allocating/increasing shared memory, or
> is the WAL page indexes getting corrupted when shared memory
> increases?
>
> Please suggest some tips to find out the root cause for this issue.
>
> Thanks,
> Praveen
> 
> From: sqlite-users-boun...@sqlite.org [sqlite-users-
> boun...@sqlite.org] On Behalf Of Dan Kennedy [danielk1...@gmail.com]
> Sent: Saturday, October 02, 2010 2:39 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
> OS
>
> On Oct 1, 2010, at 2:20 PM, Raj, Praveen wrote:
>
>> Hi,
>>
>> I'm validating SQLite test suite version 3.7.2 on QNX operating
>> system. I have built the testfixture using SQLite amalgation file
>> and other related files/libraries.
>>
>> I ran the full test suite and most of the test cases are passing,
>> except some run in journal_mode = WAL. Test cases whose
>> autocheckpoint is turned off and large amount of
>> data is being inserted without checkpointing are failing. Other WAL
>> tests are passing. I tried increasing the stack size of the
>> testfixture process, but that didn't help.
>>
>> If i reduce the number of inserts or increase the WAL page size,
>> test cases are passing. I believe there is some corruption happening
>> with the WAL index while
>> inserting large number of rows. Here is an example (from SQLite test
>> suite) of the failure
>> (Purposefully added "PRAGMA integrity_check" at end of wal3-1.0)
>
> How large are the "test.db", "test.db-wal" and "test.db-shm" files
> on your system after the transaction in wal3-1.0? i.e. stick
>
>   puts [file size test.db]
>   puts [file size test.db-wal]
>   puts [file size test.db-shm]
>
> or similar after the test case.
>
> Here the files are 1024, 4333512 and 65536 bytes. From what you are
> saying it sounds like the problem is occurring when the -shm file
> grows from 32KB to 64KB.
>
> Dan.
>
>
> ___
> sqlite-users mailing list
> sqlite-

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Dan Kennedy]

On Oct 6, 2010, at 8:15 PM, Raj, Praveen wrote:

> Hello,
>
> I'm new to SQLite, so not sure what is the behavior of these failed  
> tests on Unix/Linux. Are they successfully passing all the WAL  
> journal mode tests on Unix?
>
> What could be happening when the shared memory (-shm file) is  
> increased from 32k to 64k?
> Is a new 64k memory chunk allocated and old 32k data is copied here  
> and then 32k is purged?

At some point in the transaction, the first 32KB of shared-memory
are being zeroed. I'm not sure why this would happen.

The WAL file looks Ok. And the header in the shared-memory looks
Ok. But most of the first 32KB of shared-memory has been zeroed
(it is likely that the whole thing was zeroed and then the header
rewritten).

Maybe something about running the unixShmMap() function on QNX to
mmap() the second 32KB of space (we use a separate mapping
for each 32KB block) is zeroing the first. Somehow. Do you have
a good debugger for the platform?

Dan.




> Any insights on whats happening here would be of great help.
>
> Thanks,
> Praveen
>
> -Original Message-
> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org 
> ] On Behalf Of Raj, Praveen
> Sent: Sunday, October 03, 2010 7:42 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX  
> OS
>
> Hi,
>
> Yes the files sizes of "test.db", "test.db-wal" and "test.db-shm"  
> are 1024, 4333512 and 65536 bytes respectively as specified below.
>
> Most of the test scripts are failing when the -shm file size grows  
> from 32kb to 64kb (though the "test.db-wal"
> file size varies for different test scripts).
> But there are cases in WAL tests where the size increases to 64kb,  
> but the problem is not seen.
> Is there any problem while allocating/increasing shared memory, or  
> is the WAL page indexes getting corrupted when shared memory  
> increases?
>
> Please suggest some tips to find out the root cause for this issue.
>
> Thanks,
> Praveen
> 
> From: sqlite-users-boun...@sqlite.org [sqlite-users- 
> boun...@sqlite.org] On Behalf Of Dan Kennedy [danielk1...@gmail.com]
> Sent: Saturday, October 02, 2010 2:39 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX  
> OS
>
> On Oct 1, 2010, at 2:20 PM, Raj, Praveen wrote:
>
>> Hi,
>>
>> I'm validating SQLite test suite version 3.7.2 on QNX operating
>> system. I have built the testfixture using SQLite amalgation file
>> and other related files/libraries.
>>
>> I ran the full test suite and most of the test cases are passing,
>> except some run in journal_mode = WAL. Test cases whose
>> autocheckpoint is turned off and large amount of
>> data is being inserted without checkpointing are failing. Other WAL
>> tests are passing. I tried increasing the stack size of the
>> testfixture process, but that didn't help.
>>
>> If i reduce the number of inserts or increase the WAL page size,
>> test cases are passing. I believe there is some corruption happening
>> with the WAL index while
>> inserting large number of rows. Here is an example (from SQLite test
>> suite) of the failure
>> (Purposefully added "PRAGMA integrity_check" at end of wal3-1.0)
>
> How large are the "test.db", "test.db-wal" and "test.db-shm" files
> on your system after the transaction in wal3-1.0? i.e. stick
>
>   puts [file size test.db]
>   puts [file size test.db-wal]
>   puts [file size test.db-shm]
>
> or similar after the test case.
>
> Here the files are 1024, 4333512 and 65536 bytes. From what you are
> saying it sounds like the problem is occurring when the -shm file
> grows from 32KB to 64KB.
>
> Dan.
>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
> The information contained in this message may be confidential and  
> legally protected under applicable law. The message is intended  
> solely for the addressee(s). If you are not the intended recipient,  
> you are hereby notified that any use, forwarding, dissemination, or  
> reproduction of this message is strictly prohibited and may be  
> unlawful. If you are not the intended recipient, please contact the  
> sender by return e-mail and destroy all copies of the original  
> message.
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Raj, Praveen]

Hello,

I'm new to SQLite, so not sure what is the behavior of these failed tests on 
Unix/Linux. Are they successfully passing all the WAL journal mode tests on 
Unix?

What could be happening when the shared memory (-shm file) is increased from 
32k to 64k? 
Is a new 64k memory chunk allocated and old 32k data is copied here and then 
32k is purged?

Any insights on whats happening here would be of great help.

Thanks,
Praveen 

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Raj, Praveen
Sent: Sunday, October 03, 2010 7:42 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

Hi,

Yes the files sizes of "test.db", "test.db-wal" and "test.db-shm" are 1024, 
4333512 and 65536 bytes respectively as specified below.

Most of the test scripts are failing when the -shm file size grows from 32kb to 
64kb (though the "test.db-wal"
file size varies for different test scripts).
But there are cases in WAL tests where the size increases to 64kb, but the 
problem is not seen.
Is there any problem while allocating/increasing shared memory, or is the WAL 
page indexes getting corrupted when shared memory increases?

Please suggest some tips to find out the root cause for this issue.

Thanks,
Praveen

From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] On 
Behalf Of Dan Kennedy [danielk1...@gmail.com]
Sent: Saturday, October 02, 2010 2:39 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

On Oct 1, 2010, at 2:20 PM, Raj, Praveen wrote:

> Hi,
>
> I'm validating SQLite test suite version 3.7.2 on QNX operating
> system. I have built the testfixture using SQLite amalgation file
> and other related files/libraries.
>
> I ran the full test suite and most of the test cases are passing,
> except some run in journal_mode = WAL. Test cases whose
> autocheckpoint is turned off and large amount of
> data is being inserted without checkpointing are failing. Other WAL
> tests are passing. I tried increasing the stack size of the
> testfixture process, but that didn't help.
>
> If i reduce the number of inserts or increase the WAL page size,
> test cases are passing. I believe there is some corruption happening
> with the WAL index while
> inserting large number of rows. Here is an example (from SQLite test
> suite) of the failure
> (Purposefully added "PRAGMA integrity_check" at end of wal3-1.0)

How large are the "test.db", "test.db-wal" and "test.db-shm" files
on your system after the transaction in wal3-1.0? i.e. stick

   puts [file size test.db]
   puts [file size test.db-wal]
   puts [file size test.db-shm]

or similar after the test case.

Here the files are 1024, 4333512 and 65536 bytes. From what you are
saying it sounds like the problem is occurring when the -shm file
grows from 32KB to 64KB.

Dan.


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

The information contained in this message may be confidential and legally 
protected under applicable law. The message is intended solely for the 
addressee(s). If you are not the intended recipient, you are hereby notified 
that any use, forwarding, dissemination, or reproduction of this message is 
strictly prohibited and may be unlawful. If you are not the intended recipient, 
please contact the sender by return e-mail and destroy all copies of the 
original message.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Raj, Praveen]

Hi,

Yes the files sizes of "test.db", "test.db-wal" and "test.db-shm" are 1024, 
4333512 and 65536 bytes respectively as specified below.

Most of the test scripts are failing when the -shm file size grows from 32kb to 
64kb (though the "test.db-wal"
file size varies for different test scripts).
But there are cases in WAL tests where the size increases to 64kb, but the 
problem is not seen.
Is there any problem while allocating/increasing shared memory, or is the WAL 
page indexes getting corrupted when shared memory increases?

Please suggest some tips to find out the root cause for this issue.

Thanks,
Praveen

From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] On 
Behalf Of Dan Kennedy [danielk1...@gmail.com]
Sent: Saturday, October 02, 2010 2:39 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

On Oct 1, 2010, at 2:20 PM, Raj, Praveen wrote:

> Hi,
>
> I'm validating SQLite test suite version 3.7.2 on QNX operating
> system. I have built the testfixture using SQLite amalgation file
> and other related files/libraries.
>
> I ran the full test suite and most of the test cases are passing,
> except some run in journal_mode = WAL. Test cases whose
> autocheckpoint is turned off and large amount of
> data is being inserted without checkpointing are failing. Other WAL
> tests are passing. I tried increasing the stack size of the
> testfixture process, but that didn't help.
>
> If i reduce the number of inserts or increase the WAL page size,
> test cases are passing. I believe there is some corruption happening
> with the WAL index while
> inserting large number of rows. Here is an example (from SQLite test
> suite) of the failure
> (Purposefully added "PRAGMA integrity_check" at end of wal3-1.0)

How large are the "test.db", "test.db-wal" and "test.db-shm" files
on your system after the transaction in wal3-1.0? i.e. stick

   puts [file size test.db]
   puts [file size test.db-wal]
   puts [file size test.db-shm]

or similar after the test case.

Here the files are 1024, 4333512 and 65536 bytes. From what you are
saying it sounds like the problem is occurring when the -shm file
grows from 32KB to 64KB.

Dan.


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

The information contained in this message may be confidential and legally 
protected under applicable law. The message is intended solely for the 
addressee(s). If you are not the intended recipient, you are hereby notified 
that any use, forwarding, dissemination, or reproduction of this message is 
strictly prohibited and may be unlawful. If you are not the intended recipient, 
please contact the sender by return e-mail and destroy all copies of the 
original message.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Dan Kennedy]

On Oct 1, 2010, at 2:20 PM, Raj, Praveen wrote:

> Hi,
>
> I'm validating SQLite test suite version 3.7.2 on QNX operating  
> system. I have built the testfixture using SQLite amalgation file  
> and other related files/libraries.
>
> I ran the full test suite and most of the test cases are passing,  
> except some run in journal_mode = WAL. Test cases whose  
> autocheckpoint is turned off and large amount of
> data is being inserted without checkpointing are failing. Other WAL  
> tests are passing. I tried increasing the stack size of the  
> testfixture process, but that didn't help.
>
> If i reduce the number of inserts or increase the WAL page size,  
> test cases are passing. I believe there is some corruption happening  
> with the WAL index while
> inserting large number of rows. Here is an example (from SQLite test  
> suite) of the failure
> (Purposefully added "PRAGMA integrity_check" at end of wal3-1.0)

How large are the "test.db", "test.db-wal" and "test.db-shm" files
on your system after the transaction in wal3-1.0? i.e. stick

   puts [file size test.db]
   puts [file size test.db-wal]
   puts [file size test.db-shm]

or similar after the test case.

Here the files are 1024, 4333512 and 65536 bytes. From what you are
saying it sounds like the problem is occurring when the -shm file
grows from 32KB to 64KB.

Dan.


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS

[Raj, Praveen]

Hi,

I'm validating SQLite test suite version 3.7.2 on QNX operating system. I have 
built the testfixture using SQLite amalgation file and other related 
files/libraries.

I ran the full test suite and most of the test cases are passing, except some 
run in journal_mode = WAL. Test cases whose autocheckpoint is turned off and 
large amount of
data is being inserted without checkpointing are failing. Other WAL tests are 
passing. I tried increasing the stack size of the testfixture process, but that 
didn't help.

If i reduce the number of inserts or increase the WAL page size, test cases are 
passing. I believe there is some corruption happening with the WAL index while
inserting large number of rows. Here is an example (from SQLite test suite) of 
the failure
(Purposefully added "PRAGMA integrity_check" at end of wal3-1.0)

set a_string_counter 1
proc a_string {n} {
  global a_string_counter
  incr a_string_counter
  string range [string repeat "${a_string_counter}." $n] 1 $n
}
db func a_string a_string

do_test wal3-1.0 {
  execsql {
PRAGMA cache_size = 2000;
PRAGMA page_size = 1024;
PRAGMA auto_vacuum = off;
PRAGMA synchronous = normal;
PRAGMA journal_mode = WAL;
PRAGMA wal_autocheckpoint = 0;
BEGIN;
  CREATE TABLE t1(x);
  INSERT INTO t1 VALUES( a_string(800) );  /*1 */
  INSERT INTO t1 SELECT a_string(800) FROM t1; /*2 */
  INSERT INTO t1 SELECT a_string(800) FROM t1; /*4 */
  INSERT INTO t1 SELECT a_string(800) FROM t1; /*8 */
  INSERT INTO t1 SELECT a_string(800) FROM t1; /*   16 */
  INSERT INTO t1 SELECT a_string(800) FROM t1; /*   32 */
  INSERT INTO t1 SELECT a_string(800) FROM t1; /*   64 */
  INSERT INTO t1 SELECT a_string(800) FROM t1; /*  128*/
  INSERT INTO t1 SELECT a_string(800) FROM t1; /*  256 */
  INSERT INTO t1 SELECT a_string(800) FROM t1; /*  512 */
  INSERT INTO t1 SELECT a_string(800) FROM t1; /* 1024 */
  INSERT INTO t1 SELECT a_string(800) FROM t1; /* 2048 */
  INSERT INTO t1 SELECT a_string(800) FROM t1;  /* 4018 */
COMMIT;
PRAGMA cache_size = 10;
PRAGMA integrity_check;
  }
} {ok}

for {set i 1} {$i < 50} {incr i} {

  do_test wal3-1.$i.1 {
set str [a_string 800]
execsql { UPDATE t1 SET x = $str WHERE rowid = $i }
lappend L [wal_frame_count test.db-wal 1024]
execsql {
  BEGIN;
INSERT INTO t1 SELECT a_string(800) FROM t1 LIMIT 100;
  ROLLBACK;
  PRAGMA integrity_check;
}
  } {ok}

  # Check that everything looks OK from the point of view of an
  # external connection.
  #
  sqlite3 db2 test.db
  do_test wal3-1.$i.2 {
execsql { SELECT count(*) FROM t1 } db2
  } 4018
  do_test wal3-1.$i.3 {
execsql { SELECT x FROM t1 WHERE rowid = $i }
  } $str
  do_test wal3-1.$i.4 {
execsql { PRAGMA integrity_check } db2
  } {ok}
  db2 close

  # Check that the file-system in its current state can be recovered.
  #
  file copy -force test.db test2.db
  file copy -force test.db-wal test2.db-wal
  file delete -force test2.db-journal
  sqlite3 db2 test2.db
  do_test wal3-1.$i.5 {
execsql { SELECT count(*) FROM t1 } db2
  } 4018
  do_test wal3-1.$i.6 {
execsql { SELECT x FROM t1 WHERE rowid = $i }
  } $str
  do_test wal3-1.$i.7 {
execsql { PRAGMA integrity_check } db2
  } {ok}
  db2 close
}

The result of above test:

wal3-1.0...
Expected: [ok]
 Got: [wal 0 {*** in database main ***
Page 2: btreeInitPage() returns error code 11
Page 3 is never used
Page 4 is never used
Page 5 is never used

Page 99 is never used
Page 100 is never used
Page 101 is never used
Page 102 is never used}]
wal3-1.1.1...
Error: database disk image is malformed
wal3-1.1.2...
Error: no such table: t1
wal3-1.1.3...
Error: database disk image is malformed
wal3-1.1.4... Ok
wal3-1.1.5...
Error: no such table: t1
wal3-1.1.6...
Error: database disk image is malformed
wal3-1.1.7... Ok

The above error repeats for all iterations.

Any help/suggestions for the above issue will be highly appreciated.

Other tests which fail include:
walfault-6-pre-1
walfault-11-pre-1
wal-13.$tn.$ii.a - wal-13.$tn.$ii.d (after 12th iteration)
wal-20.2 - wal-20.4

Thanks.


The information contained in this message may be confidential and legally 
protected under applicable law. The message is intended solely for the 
addressee(s). If you are not the intended recipient, you are hereby notified 
that any use, forwarding, dissemination, or reproduction of this message is 
strictly prohibited and may be unlawful. If you are not the intended recipient, 
please contact the sender by return e-mail and destroy all copies of the 
original message.
___
sqlite-users mailing list
sqlite-users@sqlite.org