Re: [sqlite] File locking additions
[EMAIL PROTECTED] wrote: Helmut Tschemernjak <[EMAIL PROTECTED]> wrote: The current lock offset will not allow to copy open DB files if the database gets larger than 0x4000 bytes. This is because locked regions cannot be copied under Windows, we changed it to: #define PENDING_BYTE I64CONST(0x7fff) #define RESERVED_BYTE I64CONST(0x7fff0001) #define SHARED_FIRST I64CONST(0x7fff0002) You are, of course, free to changes SQLite in any way you want for your own use. But this change cannot be made in the official SQLite implementation for two reasons: (1) It would break backwards compatibility. (2) It will not work on systems that do not support large files (file bigger than 2GiB) such as many old Unix implementations and Win95. -- D. Richard Hipp <[EMAIL PROTECTED]> Every main desktop and server platform supports 64bit file offsets for minimum 5 years and many already for 10 years. If we get a common locking code which is cross platform network compatible (SMB, AFP, NFS) as well supporting all major existing OS by default is a good goal and will introduce incompatibilities anyways. Custom solutions, embedded OS's, etc. must port the file IO, locking and threading anyways to their OS/solution. Helmut Tschemernjak
Re: [sqlite] File locking additions
Hello Adam, all, Cross platform locking is defiantly very important. The goal should be to use the identical sqlite DB via UNIX, AFP, SMB, NFS and others. My opinion is that it is not needed to have any additional open parameters, just use the locking features which are in common between all platforms. Using an additional TCP/IP server is more complex in term of cross platform compatibility and may be an overkill in terms of performance, and will introduce additional problems. At present the os_unix.c has some problems, I don't recall all its details so here is just an estimate: Multiple locks are getting unlocked with one call - This does not work on AFP and SMB A Read lock gets turned into a writer lock (and or back?) - This does not work on AFP, SMB has limited support for it. The current lock offset will not allow to copy open DB files if the database gets larger than 0x4000 bytes. This is because locked regions cannot be copied under Windows, we changed it to: #define PENDING_BYTE I64CONST(0x7fff) #define RESERVED_BYTE I64CONST(0x7fff0001) #define SHARED_FIRST I64CONST(0x7fff0002) Advisory versus mandatory record locking AFP and SMB is doing mandatory locking which means other clients cannot read locked file areas, advisory locking means that we can ready any data therefore we must call lock first before reading bytes. The benefit of supporting the OS based locking is that after a program exits it will automatically cleanup all locks. Windows has some "oplock" features which will handle remote locks completely on the Windows SMB clients as long as only one writer uses the file. It should not be this difficult to use only locking features which are in common cross major platforms and network file systems. Helmut Tschemernjak Adam Swift wrote: All, In order to provide locking support for database files mounted from remote file systems (NFS, SMB, AFP, etc) as well as provide compatible locking support between database clients both local and remote, I would like to propose some additions to the existing database file locking behavior. I have discussed this briefly with D. Richard Hipp and he has expressed interest in pursuing it. We would appreciate feedback/suggestions/concerns on the proposed solutions below. 1. Support a variety of locking mechanisms, from the lowest granularity (using a .lock file) to the highest (the existing advisory locks). A preliminary list: .lock files, flock, afp locks, posix advisory locks. 2. Allow clients to specify type of locking (on database open) manually or ask sqlite to automatically detect the best locking supported by the file system hosting the database file (automatic detection would not work in a mixed local/remote file system situation, all clients of a single database file need to use the same type of locking to avoid conflicts). 3. Extend the sqlite3_open commands to support URI style path references in order to specify the file system locking type (as opposed to modifying the arguments list). After a little poking around on RFC 3986 <http://www.ietf.org/rfc/rfc3986.txt> I'm inclined to specify the locking choice via the query part of the URI. For example: file:///mounts/myhost/dbfile.db?locktype=flock file:///localdisk/otherdbfile.db?locktype=automatic Thanks in advance for your input. Adam Swift
Re: [sqlite] malloc(0) in sqlite3_exec
Hi, malloc(0) on AIX 4.3 and 5.1 returns NULL, which crashes some apps. We fixed it in our applications. As we migrate from ctree to sqlite2 this needs to be fixed or increased from malloc(0) to 1. On AIX it will definitely return NULL and will crash as soon the pointer is referenced. Is the malloc(0) pointer used? Helmut Tschemernjak Lauri Nurmi wrote: D. Richard Hipp wrote: On Wed, 2005-06-29 at 15:19 +0300, Lauri Nurmi wrote: So, basically, is there a reason why sqlite3Malloc(n) & co should even try to allocate zero bytes, instead of just returning NULL if n==0? Does a pointer pointing to zero bytes of memory have any use anyway? Yes. A NULL return from sqlite3Malloc() is an error condition meaning that you have run out of memory. That is very different from returning a zero-length memory allocation. I see the difference, yes. Still, I'm not convinced that doing a malloc(0) is a good thing. I found this quotation from the net <http://lists.freebsd.org/pipermail/freebsd-net/2004-February/002946.html>, it is probably from the C99 standard: --- If the size of the space requested is zero, the behavior is implementation-defined: either a null pointer is returned, or the behavior is as if the size were some nonzero value, except that the returned pointer shall not be used to access an object. --- So malloc(0) may return NULL, and even if it doesn't, the pointer should not be used. You wouldn't want to substitute a NULL for an empty string in SQL would you? Having sqlite3Malloc return NULL for a zero-length allocation is roughly the same thing. Hmm, an empty (C-style) string doesn't fit into zero bytes, it needs one byte for the '\0'. If you have a pointer pointing to zero allocated bytes, you can't even use strlen() or anything else on it [according to the quotation above].
Re: [sqlite] Bug in fsync in SuSE 9.3
Hi, it is hard to believe that there is a bug in fsync under the Linux kernel 2.6.10. What kind of filesystem have you used. (NFS, FAT, Reiser, Ext2/3). Are any special kernel or mount options used? Is it a sparse file? How have you found out that it is related to fsync? best regards Helmut Tschemernjak Cavit Cahit VURAL wrote: Hi , You use fsync to refresh the data to the disk as we were. In SuSE 9.3 (don't know others) fsync does not work properly. Sometimes it do not use the buffer you gave , writes garbage data to the file. It took 25 days to determine the bug :-) Regards CC Vural
Re: [sqlite] Reading German Umlauts correct
Hallo Daniel, on Windows you can use WideCharToMultiByte and MultibyteToWideChar to convert from and to UTF8. I would recommend to use utf8 in the DB because it is a pretty good standard and being using in Mac OS X, Java, Solaris/Linux, HTML, and others. Windows uses either 8 bit client charset e.g. PC850 or 16 bit Unicode. UTF-8 is basically the same like 16 bit Unicode but encoded into 8 bit where basic chars 0-127 stay in ascii. Helmut Tschemernjak [EMAIL PROTECTED] wrote: Hello :) I'm reading Information from a SQLite database. But this Information contains German umlauts like ä, ö, ü. When I get these Information back in my Programm, these umlauts are unreadable. The ü for example is ü. The umlauts are stored in the Database correctly. Can somebody tell me, what I have to do, that the umlauts are displayes correctly exept a string manipulation after data-reading? Thanks in advance :) Daniel
Re: [sqlite] Tiger: Error: file is encrypted or is not a database
Will Leshner wrote: On Jun 6, 2005, at 7:36 AM, Helmut Tschemernjak wrote: The data is not encrypted. What makes you think it is a SQLite database (a slightly rhetorical question as I'm pretty sure I know what makes you think that). But, and this isn't rhetorical, what makes you think it isn't encrypted? Just do a less on the .db files and you can see the information is in clear text. I will investigate into it a little bit more.
Re: [sqlite] Possible 1GB database limit on Windows from PENDING_BYTE
thank you a lot for the feedback. It is not only Windows using advisory locking and aborting other reads, it is the same with Mac OS X using AFP volumes or a Samba CIFS volume mount on a Linux client. I understand the problem and consider to add a ".backup" command to the sqlite shell to backup a running database. I still believe it is an good idea to move the lock range outside of the file range e.g. 0x. Word/Excel using as well a locking in high area for simple interprocess communication about the current Word/Excel file. Helmut Tschemernjak D. Richard Hipp wrote: On Mon, 2005-05-16 at 15:58 +0200, Helmut Tschemernjak wrote: thank you for the info, another possible problem is that I cannot backup the database e.g. "begin exclusive" and copy the entire DB file while the main server app is still running. The copy aborts at offset 1GB. This is only a problem on windows. (Unix lets you make a copy of the file even though it is locked - unix got this part right.) A easy work-around would be to write a custom "copy" program that ignored bytes 1GiB through 1Gib+510.
Re: [sqlite] Possible 1GB database limit on Windows from PENDING_BYTE
thank you for the info, another possible problem is that I cannot backup the database e.g. "begin exclusive" and copy the entire DB file while the main server app is still running. The copy aborts at offset 1GB. Helmut Tschemernjak D. Richard Hipp wrote: On Mon, 2005-05-16 at 15:33 +0200, Helmut Tschemernjak wrote: looking into the PENDING_BYTE 0x4000 define from os.h, I believe there is a problem when the database is larger than 1GB and a second process reads at offset 1GB, the read may aborts with SQLITE_IOERR if a lock is active. This should not happen because no process ever reads data out of the range 1GiB through 1GiB+510. That range of bytes is used for locking only. No data is every stored here. If you grep for PENDING_BYTE in pager.c, you will find the places in the code that automatically skip over the locking bytes.
[sqlite] Possible 1GB database limit on Windows from PENDING_BYTE
Hello, looking into the PENDING_BYTE 0x4000 define from os.h, I believe there is a problem when the database is larger than 1GB and a second process reads at offset 1GB, the read may aborts with SQLITE_IOERR if a lock is active. A "(u64)-10" PENDING_BYTE define and adjusting the low/high dword for LockFileEx should fix this. It is clear to me that all clients need to update the libsqlite to allow concurrent access with the new locking offset. Is this right? Helmut Tschemernjak
[sqlite] enum return type idea for sqlite3 functions
Hello,
I like to recommend to use an enum for the return code of all sqlite3
functions., E.g.:
typedef enum {
SQLITE_OK = 0,/* Successful result */
SQLITE_ERROR= 1,/* SQL error or missing database */
SQLITE_INTERNAL = 2,/* An internal logic error in SQLite */
//...
} sqlite3_retcode;
sqlite3_retcode sqlite3_open(
const char *filename, /* Database filename (UTF-8) */
sqlite3 **ppDb /* OUT: SQLite db handle */
);
The benefit would be that the return value would be displayed in the
debugger.
best regards
Helmut Tschemernjak
