Re: [sqlite] Porting Sqlite to MQX OS: Question 2

2010-03-16 Thread GeoffW

Hello Dan


>Do the tasks share a heap? And do you have multiple connections
>to the database (multiple calls to sqlite3_open() or sqlite3_open_v2().

I would like to keep the implementation as general purpose as possible so we
can use sqlite in the future in differient Applications. Therefore I will
say I need to have multiple connections to the database. Each of the tasks
in the Application, have their own task stack, but mallocs() for example
will all take blocks from a common memory pool. Not too sure of how this
effects my locking strategy implementation.

To be honest, despite having read the locking bits of osUnix.c numerous
times, I am still pretty much lost.

Regards Geoff









Dan Kennedy-4 wrote:
> 
> 
> On Mar 16, 2010, at 5:22 AM, GeoffW wrote:
> 
>>
>> Hello Dan
>>
>> Thanks for your useful input. To answer your questions.
>>
>>> Do you have any file-locking primitives provided by the OS?
>> There are no file locking OS Primitives at all that I can use.
>>
>>> Do you have clients connecting to the database from multiple
>>> processes? Or only multiple threads within the same process?
>>
>> My Application doesnt have a concept of processes and threads as you  
>> might
>> find in Windows.
>> Think of it as a  small data collection embedded Application. The  
>> platform
>> has essentially 1 fixed Application running, made up of say 10  
>> different
>> concurrent tasks.
> 
> Do the tasks share a heap? And do you have multiple connections
> to the database (multiple calls to sqlite3_open() or sqlite3_open_v2().
> 
> If you only have one connection, then you don't need any locking. Have
> xCheckReservedLock() set its result variable to 0 for all calls.
> 
> If all tasks share a heap, maybe you can implement locking in-memory
> using global variables in the VFS layer. Or, if you strictly use
> shared-cache mode, you will not need any locking.
> 
> Dan.
> 
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> 
> 

-- 
View this message in context: 
http://old.nabble.com/Porting-Sqlite-to-MQX-OS%3A-Question-2-tp27874124p27918689.html
Sent from the SQLite mailing list archive at Nabble.com.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Porting Sqlite to MQX OS: Question 2

2010-03-15 Thread Dan Kennedy

On Mar 16, 2010, at 5:22 AM, GeoffW wrote:

>
> Hello Dan
>
> Thanks for your useful input. To answer your questions.
>
>> Do you have any file-locking primitives provided by the OS?
> There are no file locking OS Primitives at all that I can use.
>
>> Do you have clients connecting to the database from multiple
>> processes? Or only multiple threads within the same process?
>
> My Application doesnt have a concept of processes and threads as you  
> might
> find in Windows.
> Think of it as a  small data collection embedded Application. The  
> platform
> has essentially 1 fixed Application running, made up of say 10  
> different
> concurrent tasks.

Do the tasks share a heap? And do you have multiple connections
to the database (multiple calls to sqlite3_open() or sqlite3_open_v2().

If you only have one connection, then you don't need any locking. Have
xCheckReservedLock() set its result variable to 0 for all calls.

If all tasks share a heap, maybe you can implement locking in-memory
using global variables in the VFS layer. Or, if you strictly use
shared-cache mode, you will not need any locking.

Dan.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Porting Sqlite to MQX OS: Question 2

2010-03-15 Thread GeoffW
k functions.
>>>>
>>>> As Sqlite has been ported to so many diverse operating systems I  
>>>> am hoping
>>>> that this question of what to do for an ANSI only File system has  
>>>> already
>>>> been hit and a workaround defined ?
>>>>
>>>> I am pretty hazy on this topic of file locking so I would  
>>>> appreciate some
>>>> basic discussion and guidance. It might help if I explain my  
>>>> application a
>>>> little. It is an embedded system where we are just running one
>> application,
>>>> but it is multithreaded. For my initial porting work I have made
>>>> SQLITE_THREADSAFE = 0 (so I probably dont even need a file lock in  
>>>> the
>>>> configuration I have at present ?)
>>>> However I would of course like to make SQLITE_THREADSAFE = 1 so that
>>>> multiple tasks can safely access a database.
>>>
>>> (Disclaimer: this is not my specialism, I hope it helps
>>> nevertheless)
>>>
>>> SQLITE_THREADSAFE has effect on threads within the same
>>> process (=task), especially when they use the same
>>> connection. You need file locks for synchronisation between
>>> different processes. If the filesystem doesn't provide them,
>>> you could use the dot lockfile paradigm, which seems to be
>>> associated with SQLITE_ENABLE_LOCKING_STYLE.
>>>
>>>> Thanks for any guidance
>>>>
>>>> Regards Geoff
>>> -- 
>>>  (  Kees Nuyt
>>>  )
>>> c[_]
>>> ___
>>> sqlite-users mailing list
>>> sqlite-users@sqlite.org
>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>>
>>>
>>
>> -- 
>> View this message in context:
>> http://old.nabble.com/Porting-Sqlite-to-MQX-OS%3A-Question-2-tp27874124p27905578.html
>> Sent from the SQLite mailing list archive at Nabble.com.
>>
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> 
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> 
> 

-- 
View this message in context: 
http://old.nabble.com/Porting-Sqlite-to-MQX-OS%3A-Question-2-tp27874124p27911159.html
Sent from the SQLite mailing list archive at Nabble.com.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Porting Sqlite to MQX OS: Question 2

2010-03-15 Thread Dan Kennedy

On Mar 15, 2010, at 10:04 PM, GeoffW wrote:

>
> Hi Kees
>
> Thanks for the response and giving me a clue to look at dot locking,  
> I had
> missed the dot lock mechanism as I had concentrated mainly on the  
> osWin.c
> file. I cant find any documentation other than the source code on  
> how this
> low level file locking stuff is implemented.
>
> I am currently reading through the code to see if I can figure it  
> out. The
> unix.c module is horrendously complicated so I am struggling to make  
> sense
> of it.

It is more complicated. But the comments in it are often better than
those in os_win.c. See the comments above unixLock(), at or around line
1208 of os_unix.c. Also here:

   http://www.sqlite.org/lockingv3.html

> In simple terms I was thinking that when the db file is opened for  
> writing,
> I could use the filename with ".lock" appended, and open that new  
> dummy
> file, which is then used to determine if I have a lock on the original
> database file or not ?

You can implement locking that way. But it would be a last
resort. And is the sort of trick that could be prone to race
conditions.

Do you have any file-locking primitives provided by the OS?

Do you have clients connecting to the database from multiple
processes? Or only multiple threads within the same process?

> As the code says for dot locking there is really only a 2 state lock,
> exclusive or no lock. I got lost by the complexity of the code, for  
> vxworks
> it seems to keep linked lists of filenames, not sure why I would  
> need to do
> that, also it does lots of other stuff that I didnt really  
> understand for
> example updating the file timestamp on the ".lock" file.

2 locking states are all SQLite needs to work. An exclusive lock
for writers and a shared lock for readers. The other locks are
only required to increase concurrencly between readers and the
single writer.

If you handle RESERVED and PENDING locks in the same way as EXCLUSIVE
locks, things will work fine, just you lose a little concurrency.




> I was hoping Dr Hipp could manage to find a few minutes to write  
> some notes
> explaining what needs to be implemented re: dot locking and what can  
> be
> simplified and ignored.
>
> Thanks if anyone can fill in some details here.
>
> Geoff
>
>
>
> Kees Nuyt wrote:
>>
>> On Fri, 12 Mar 2010 02:22:31 -0800 (PST), GeoffW
>> <geoffw...@hotmail.com> wrote:
>>
>>>
>>> Hi
>>>
>>> Now I have got a half ported version of Sqlite up and running on  
>>> MQX I
> cant
>>> put off any longer the question of file locking and how I tackle  
>>> that ?
> For
>>> the moment i had made the xLock() and xUnlock() the same as in  
>>> osWin.c
>>> version but with the Windows lock() and unlock() function calls  
>>> commented
>>> out.
>>>
>>> The problem I have is that MQX's file system is just ANSI C, and  
>>> it lacks
>>> any lock/unlock functions.
>>>
>>> As Sqlite has been ported to so many diverse operating systems I  
>>> am hoping
>>> that this question of what to do for an ANSI only File system has  
>>> already
>>> been hit and a workaround defined ?
>>>
>>> I am pretty hazy on this topic of file locking so I would  
>>> appreciate some
>>> basic discussion and guidance. It might help if I explain my  
>>> application a
>>> little. It is an embedded system where we are just running one
> application,
>>> but it is multithreaded. For my initial porting work I have made
>>> SQLITE_THREADSAFE = 0 (so I probably dont even need a file lock in  
>>> the
>>> configuration I have at present ?)
>>> However I would of course like to make SQLITE_THREADSAFE = 1 so that
>>> multiple tasks can safely access a database.
>>
>> (Disclaimer: this is not my specialism, I hope it helps
>> nevertheless)
>>
>> SQLITE_THREADSAFE has effect on threads within the same
>> process (=task), especially when they use the same
>> connection. You need file locks for synchronisation between
>> different processes. If the filesystem doesn't provide them,
>> you could use the dot lockfile paradigm, which seems to be
>> associated with SQLITE_ENABLE_LOCKING_STYLE.
>>
>>> Thanks for any guidance
>>>
>>> Regards Geoff
>> -- 
>>  (  Kees Nuyt
>>  )
>> c[_]
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>>
>
> -- 
> View this message in context: 
> http://old.nabble.com/Porting-Sqlite-to-MQX-OS%3A-Question-2-tp27874124p27905578.html
> Sent from the SQLite mailing list archive at Nabble.com.
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Porting Sqlite to MQX OS: Question 2

2010-03-15 Thread GeoffW

Hi Kees

Thanks for the response and giving me a clue to look at dot locking, I had
missed the dot lock mechanism as I had concentrated mainly on the osWin.c
file. I cant find any documentation other than the source code on how this
low level file locking stuff is implemented.

I am currently reading through the code to see if I can figure it out. The
unix.c module is horrendously complicated so I am struggling to make sense
of it. 

In simple terms I was thinking that when the db file is opened for writing,
I could use the filename with ".lock" appended, and open that new dummy
file, which is then used to determine if I have a lock on the original
database file or not ?
As the code says for dot locking there is really only a 2 state lock,
exclusive or no lock. I got lost by the complexity of the code, for vxworks
it seems to keep linked lists of filenames, not sure why I would need to do
that, also it does lots of other stuff that I didnt really understand for
example updating the file timestamp on the ".lock" file.

I was hoping Dr Hipp could manage to find a few minutes to write some notes
explaining what needs to be implemented re: dot locking and what can be
simplified and ignored.

Thanks if anyone can fill in some details here.

Geoff



Kees Nuyt wrote:
> 
> On Fri, 12 Mar 2010 02:22:31 -0800 (PST), GeoffW
> <geoffw...@hotmail.com> wrote:
> 
>>
>>Hi
>>
>>Now I have got a half ported version of Sqlite up and running on MQX I
cant
>>put off any longer the question of file locking and how I tackle that ?
For
>>the moment i had made the xLock() and xUnlock() the same as in osWin.c
>>version but with the Windows lock() and unlock() function calls commented
>>out.
>>
>>The problem I have is that MQX's file system is just ANSI C, and it lacks
>>any lock/unlock functions.
>>
>>As Sqlite has been ported to so many diverse operating systems I am hoping
>>that this question of what to do for an ANSI only File system has already
>>been hit and a workaround defined ?
>>
>>I am pretty hazy on this topic of file locking so I would appreciate some
>>basic discussion and guidance. It might help if I explain my application a
>>little. It is an embedded system where we are just running one
application,
>>but it is multithreaded. For my initial porting work I have made
>>SQLITE_THREADSAFE = 0 (so I probably dont even need a file lock in the
>>configuration I have at present ?)
>>However I would of course like to make SQLITE_THREADSAFE = 1 so that
>>multiple tasks can safely access a database.
> 
> (Disclaimer: this is not my specialism, I hope it helps
> nevertheless)
> 
> SQLITE_THREADSAFE has effect on threads within the same
> process (=task), especially when they use the same
> connection. You need file locks for synchronisation between
> different processes. If the filesystem doesn't provide them,
> you could use the dot lockfile paradigm, which seems to be
> associated with SQLITE_ENABLE_LOCKING_STYLE.
> 
>>Thanks for any guidance
>>
>>Regards Geoff
> -- 
>   (  Kees Nuyt
>   )
> c[_]
> _______
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> 
> 

-- 
View this message in context: 
http://old.nabble.com/Porting-Sqlite-to-MQX-OS%3A-Question-2-tp27874124p27905578.html
Sent from the SQLite mailing list archive at Nabble.com.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


Re: [sqlite] Porting Sqlite to MQX OS: Question 2

2010-03-12 Thread Kees Nuyt
On Fri, 12 Mar 2010 02:22:31 -0800 (PST), GeoffW
 wrote:

>
>Hi
>
>Now I have got a half ported version of Sqlite up and running on MQX I cant
>put off any longer the question of file locking and how I tackle that ? For
>the moment i had made the xLock() and xUnlock() the same as in osWin.c
>version but with the Windows lock() and unlock() function calls commented
>out.
>
>The problem I have is that MQX's file system is just ANSI C, and it lacks
>any lock/unlock functions.
>
>As Sqlite has been ported to so many diverse operating systems I am hoping
>that this question of what to do for an ANSI only File system has already
>been hit and a workaround defined ?
>
>I am pretty hazy on this topic of file locking so I would appreciate some
>basic discussion and guidance. It might help if I explain my application a
>little. It is an embedded system where we are just running one application,
>but it is multithreaded. For my initial porting work I have made
>SQLITE_THREADSAFE = 0 (so I probably dont even need a file lock in the
>configuration I have at present ?)
>However I would of course like to make SQLITE_THREADSAFE = 1 so that
>multiple tasks can safely access a database.

(Disclaimer: this is not my specialism, I hope it helps
nevertheless)

SQLITE_THREADSAFE has effect on threads within the same
process (=task), especially when they use the same
connection. You need file locks for synchronisation between
different processes. If the filesystem doesn't provide them,
you could use the dot lockfile paradigm, which seems to be
associated with SQLITE_ENABLE_LOCKING_STYLE.

>Thanks for any guidance
>
>Regards Geoff
-- 
  (  Kees Nuyt
  )
c[_]
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] Porting Sqlite to MQX OS: Question 2

2010-03-12 Thread GeoffW

Hi

Now I have got a half ported version of Sqlite up and running on MQX I cant
put off any longer the question of file locking and how I tackle that ? For
the moment i had made the xLock() and xUnlock() the same as in osWin.c
version but with the Windows lock() and unlock() function calls commented
out.

The problem I have is that MQX's file system is just ANSI C, and it lacks
any lock/unlock functions.

As Sqlite has been ported to so many diverse operating systems I am hoping
that this question of what to do for an ANSI only File system has already
been hit and a workaround defined ?

I am pretty hazy on this topic of file locking so I would appreciate some
basic discussion and guidance. It might help if I explain my application a
little. It is an embedded system where we are just running one application,
but it is multithreaded. For my initial porting work I have made
SQLITE_THREADSAFE = 0 (so I probably dont even need a file lock in the
configuration I have at present ?)
However I would of course like to make SQLITE_THREADSAFE = 1 so that
multiple tasks can safely access a database.

Thanks for any guidance

Regards Geoff




-- 
View this message in context: 
http://old.nabble.com/Porting-Sqlite-to-MQX-OS%3A-Question-2-tp27874124p27874124.html
Sent from the SQLite mailing list archive at Nabble.com.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users