Re[3]: BDB corrupt
Thanks everyone for responding. I'll use QDBM as most attractive from my point of view. Oracle BDB is to complex for my task and have drastic free license with unknown price for commercial use. Licensing is not a main issue for me now, but i'll beware it to be on the safe side. I think that FreeBSD (ideally) must have a few PRIMARY datastore tools which together cover most tasks and able for horizontal and vertical scaling: 1. BDB1 = QDBM 2. CDB 3. SQLite 4. Firebird 5. PostgreSQL I'm absolutely sure that tradition of programmers to write own datastore, memory list, etc has to be dropped. I think that some warning like 'may corrupt data' has to be committed to 'man db'. -- Best regards, Anthonymailto:[EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On 2008-May-13 22:06:21 +0100, James Mansion [EMAIL PROTECTED] wrote: And is the objection to SQL such the sqlite is really out of the running? There is no specific objection to SQL. There is a general objection to adding more utilities to the base system unless a _very_ good case can be made for including them. So far, no-one has made a compelling reason to include SQLite (or other SQL engine) into the base system. SQLite does appear to be a potential candidate for the base system - the license is acceptable and it is relatively small. On the downside, SQLite is currently undergoing rapid development - at a rate much faster than FreeBSD can comfortably accomodate. Anyway, in this case, would writing an RPC server to own the data kill the performance? That depends on how everything is implemented. It doesn't really solve the underlying problem that a database is needed. -- Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. pgpcZu9DtT0lI.pgp Description: PGP signature
Re: BDB corrupt
On May 13, 2008, at 2:06 PM, James Mansion wrote: Kurt J. Lidl wrote: This catapults back into the arena of stuff that isn't in the base system. Not to mention I'm not sure that the Oracle BDB license would allow bundling in the OS as a binary. I doubt it, but that's a different bikeshed to paint :-) Is the LGPL of QDBM and TokyoCabinet also a problem? Could even try grovelling with Mikio? (Partially joking there. I assume he chose LGPL because he wants it that way, but people have been known to change licenses for a base system - like this http://blogs.sun.com/aalok/entry/lzma_on_opensolaris) And is the objection to SQL such the sqlite is really out of the running? Anyway, in this case, would writing an RPC server to own the data kill the performance? It should be easier to write something that can save the database atomically and index it in-core. It could be started on demand and shut down after a short inactivity, a bit like tibco's rvd. There are known problems with certain keys corrupting the DB 1.8x series code. In fact, the release of the 1.86 was an attempt to solve this problem when the KerberosV people at MIT found a repeatable key insert sequence that would corrupt things. (Or at least that's what I remember, it was a long time ago, and I might have the details wrong.) Have to say its a little concerning that such 'mature' code is actually problematic. Particularly since I'm not aware of a non-LGPL alternative. Do you have anything by way of a pointer? Google didn't help me here. James Most of the complaints about other DBs is licensing related, but SQLite's complaint was also the fact that the past stability record was a bit rocky. HTH, -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Wed, May 14, 2008 at 12:25:16AM -0700, Garrett Cooper wrote: Most of the complaints about other DBs is licensing related, but SQLite's complaint was also the fact that the past stability record was a bit rocky. One other thing to watch for in SQLite is the lack of atomicity in updates. It's not ACID, just like BDB 1.8x isn't ACID. Without a write-ahead log, you cannot be sure that the data written actually made it to stable storage, and as such, you cannot be sure that your database didn't get corrupted when the process stops in a non-optimal way. I view SQLite as adding syntactic sugar to your data access language. It doesn't give you real tranactions, it doesn't give you a write-ahead log, it doesn't give you referential integrity. Heck, the last time I looked (admitted, a while ago), it didn't even enforce column type checking on tables. (Put this string in the INT column? No problem!) -Kurt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Tue, May 13, 2008 at 10:06:21PM +0100, James Mansion wrote: Kurt J. Lidl wrote: There are known problems with certain keys corrupting the DB 1.8x series code. In fact, the release of the 1.86 was an attempt to solve this problem when the KerberosV people at MIT found a repeatable key insert sequence that would corrupt things. (Or at least that's what I remember, it was a long time ago, and I might have the details wrong.) Have to say its a little concerning that such 'mature' code is actually problematic. Particularly since I'm not aware of a non-LGPL alternative. Do you have anything by way of a pointer? Google didn't help me here. This is somewhat alluded to here: http://www.mail-archive.com/[EMAIL PROTECTED]/msg01560.html You might want to read the entire thread on that issue. There is the comment in the Oracle web pages about 1.86 vs 1.85: http://www.oracle.com/technology/software/products/berkeley-db/db/index.html Do not upgrade to the 1.86 release other than to fix specific hash access method problems found in the 1.85 release. And in the Berkeley DB: A Retrospective paper (http://sites.computer.org/debull/A07Sept/seltzer.pdf), Margo notes: Db-1.85 enjoyed widespread adoption after its release with 4.4BSD. In particular, both the MIT Kerberos project and the University of Michigan LDAP project incorporated it. A group at Harvard released a minor patch version db-1.86 for the Kerberos team in 1996. So, I think my recollection is correct. -Kurt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On 20080514 09:50:52, Kurt J. Lidl wrote: Heck, the last time I looked (admitted, a while ago), it didn't even enforce column type checking on tables. (Put this string in the INT column? No problem!) They consider it a feature. Nobody knows why. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Wed, 14 May 2008 17:17:28 +1000 Peter Jeremy [EMAIL PROTECTED] wrote: On 2008-May-13 22:06:21 +0100, James Mansion [EMAIL PROTECTED] wrote: And is the objection to SQL such the sqlite is really out of the running? There is no specific objection to SQL. There is a general objection to adding more utilities to the base system unless a _very_ good case can be made for including them. So far, no-one has made a compelling reason to include SQLite (or other SQL engine) into the base system. Just out of curiosity - there seems to be an unspoken assumption that the ports system can only use tools that are part of the base system. But this is clearly false - the ports system currently includes a couple of directories full of software that's not in the base system. Adding compiled code to those tools would mean that installing the ports system gets a bit more complex - you have to run make install after extracting the tarball. Is that so bad it's not going to happen? mike ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
BDB corrupt - patches
Some years ago I mailed patches out to someone regarding Berkeley DB 1.85 btree problems. The two issues which come to mind are... 1) The page split position is improperly computed. This can cause corruption when a very full page has an item which is very large inserted onto it. The split then happens but because the split isn't in the right place the inserted item will not fit and I think it overruns where it should on the page. (I think 1.86 tried to fix this, but the fix is about recovering late i.e. when it sees an item doesn't fit, the proper fix is to split the page right in the first place.) 2) The record put code has a last page put to member which speeds up sequential inserts, however if that last page put to gets deleted, that variable is not cleared causing, if you are unlucky, a put might try to put data onto a deleted page which is then woven into both the tree and the free page list. This causes terrible problems with record scanning and eventual corruption. I'm going to have to dig up these fixes, but presuming I do, who should be alerted? Just file a bug? Recreation is extremely difficult. Jeff Anton ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt - patches
Jeff Anton wrote: Some years ago I mailed patches out to someone regarding Berkeley DB 1.85 btree problems. The two issues which come to mind are... 1) The page split position is improperly computed. ... 2) The record put code has a last page put to member ... I'm going to have to dig up these fixes, but presuming I do, who should be alerted? Just file a bug? Recreation is extremely difficult. Definitely file a PR and include the patches. If nothing else, that will help ensure they don't get lost. When you do file that PR, notify this list. Given how heavily the base system uses DB 1.85, I think there will be a lot of interest in anything that improves the stability and reliability of that code. Cheers, Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt - patches
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jeff Anton wrote: [...] | I'm going to have to dig up these fixes, but presuming | I do, who should be alerted? Just file a bug? Recreation | is extremely difficult. I think Oracle is maintaining a webpage about their known bug/fixes here: http://www.oracle.com/technology/software/products/berkeley-db/db/index.html FreeBSD has applied a number of fixes IIRC, if it was a BerkeleyDB bug I'd suggest that you also give Oracle developers a headsup. Cheers, - -- ** Help China's quake relief at http://www.redcross.org.cn/ | Xin LI [EMAIL PROTECTED]http://www.delphij.net/ FreeBSD - The Power to Serve! -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkgrJq4ACgkQi+vbBBjt66CB3gCfQ/hhQXVsyItDVmbP/j4lbI4v 8k8AnRgqtHZeg7lw9aacSeXNb8uYo9Ae =dWBS -END PGP SIGNATURE- ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
Kurt J. Lidl writes: On Wed, May 14, 2008 at 12:25:16AM -0700, Garrett Cooper wrote: Most of the complaints about other DBs is licensing related, but SQLite's complaint was also the fact that the past stability record was a bit rocky. One other thing to watch for in SQLite is the lack of atomicity in updates. It's not ACID, just like BDB 1.8x isn't ACID. Without a write-ahead log, you cannot be sure that the data written actually made it to stable storage, and as such, you cannot be sure that your database didn't get corrupted when the process stops in a non-optimal way. In what way is SQLite not atomic? The documentation, Atomic Commit In SQLite, suggests that it is: http://www.sqlite.org/atomiccommit.html I don't know that it supports fully ACID (atomic, consist, isolated, durable) transactions or how it handles the various SQL standard transaction isolation levels (Read Uncommitted, Read Committed, Repeatable Read, Serializable) but I believe that updates are atomic and that it does as well as any db (in the face of lying synch. operations, etc...) to handle non-optimal stops. g. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On 2008-May-14 09:50:52 -0400, Kurt J. Lidl [EMAIL PROTECTED] wrote: One other thing to watch for in SQLite is the lack of atomicity in updates. It's not ACID, just like BDB 1.8x isn't ACID. This isn't true. SQLite does provide full ACID. One difference from (eg) Oracle is that you need to explicitly begin a transaction, rather than a transaction implicitly commencing with the first DML statement. (I don't know what the SQL standard requires). Without a write-ahead log, It does have a log to record incomplete transactions. it doesn't give you referential integrity. To quote http://www.sqlite.org/omitted.html: FOREIGN KEY constraints are parsed but are not enforced. However, the equivalent constraint enforcement can be achieved using triggers. I looked (admitted, a while ago), it didn't even enforce column type checking on tables. (Put this string in the INT column? No problem!) As noted, this is seen as a feature, not a bug. No-one is claiming SQLite is perfect. It does, however, provide a very usable SQL engine, that provides most of SQL92, with a very small footprint and a friendly license. If FreeBSD does need something with capabilities than BDB1.x, SQLite is an obvious contender. -- Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. pgpnOQp1sIPty.pgp Description: PGP signature
Re: BDB corrupt
On 2008-May-14 10:24:10 -0400, Mike Meyer [EMAIL PROTECTED] wrote: Just out of curiosity - there seems to be an unspoken assumption that the ports system can only use tools that are part of the base system. There have been suggestions that the ports/package infrastructure (pkg_* tools, portsnap etc) be unbundled from the base OS. The difficulty comes when you want to upgrade those components. I know, from experience, that portugrading portupgrade or ruby usually fails as the running portupgrade unexpectedly trips over changed bits of itself. But this is clearly false - the ports system currently includes a couple of directories full of software that's not in the base system. There is a directory full of Makefile includes and another directory full of optional tools but pkg_* sits in the base system. What are you alluding to here. Adding compiled code to those tools would mean that installing the ports system gets a bit more complex - you have to run make install after extracting the tarball. Is that so bad it's not going to happen? The problem is not the initial install so much as managing packages and upgrades. I see no problem with having the ports/package infrastructure be part of the ports system as long as: a) A user can install/uninstall/audid (and preferably upgrade) packages without needing to compile anything b) The ports system knows how to upgrade itself without tripping over itself in the process. -- Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. pgpjJr6pyWxu1.pgp Description: PGP signature
Re: BDB corrupt
On Thu, May 15, 2008 at 05:20:26AM +1000, Peter Jeremy wrote:
On 2008-May-14 09:50:52 -0400, Kurt J. Lidl [EMAIL PROTECTED] wrote:
One other thing to watch for in SQLite is the lack of atomicity
in updates. It's not ACID, just like BDB 1.8x isn't ACID.
This isn't true. SQLite does provide full ACID. One difference from
(eg) Oracle is that you need to explicitly begin a transaction, rather
than a transaction implicitly commencing with the first DML statement.
(I don't know what the SQL standard requires).
Generally, you get either implicit transactions, or you need
to put your database handle into explicit transaction mode,
typically by bracketing your sql with:
begin transaction;
stuff;
stuff;
commit;
if (error) { rollback; whine() }
Without a write-ahead log,
It does have a log to record incomplete transactions.
Well, thanks for the various pointers. I see that it grew a
transaction log since the last time I bothered to look at it
in depth. That's a very good thing.
I'll retract my assertion that it doesn't have a commit log.
-Kurt
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Thu, 15 May 2008 05:45:29 +1000 Peter Jeremy [EMAIL PROTECTED] wrote: On 2008-May-14 10:24:10 -0400, Mike Meyer [EMAIL PROTECTED] wrote: Just out of curiosity - there seems to be an unspoken assumption that the ports system can only use tools that are part of the base system. There have been suggestions that the ports/package infrastructure (pkg_* tools, portsnap etc) be unbundled from the base OS. The difficulty comes when you want to upgrade those components. I know, from experience, that portugrading portupgrade or ruby usually fails as the running portupgrade unexpectedly trips over changed bits of itself. Having the ports system depend on packages that are maintained via the ports system does make things fairly complex. On the other hand, unbundling can be done. OS X has N (for some N 1) ports systems that run outside the base system. Notably, they don't treat the code for manipulating the ports system as a port, and provide distinct commands for updating an installed port, the ports, and the port system software. However, I don't know that that's necessary for what I asked. But this is clearly false - the ports system currently includes a couple of directories full of software that's not in the base system. There is a directory full of Makefile includes and another directory full of optional tools but pkg_* sits in the base system. What are you alluding to here. I was thinking of those two directories. I wasn't thinking about the pkg_* tools, because I pretty much never use them. Adding compiled code to those tools would mean that installing the ports system gets a bit more complex - you have to run make install after extracting the tarball. Is that so bad it's not going to happen? The problem is not the initial install so much as managing packages and upgrades. I see no problem with having the ports/package infrastructure be part of the ports system as long as: a) A user can install/uninstall/audid (and preferably upgrade) packages without needing to compile anything b) The ports system knows how to upgrade itself without tripping over itself in the process. You could do what I suggested by adding the desired db library to /usr/src/usr.sbin/pkg_install, and linking it statically into the pkg* tools? Yeah, it's not really pretty, but is it any worse than having a BDB in the base system that we recommend be avoided? mike -- Mike Meyer [EMAIL PROTECTED] http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O ascii ribbon campaign - stop html mail - www.asciiribbon.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re[2]: BDB corrupt
Monday, Mike Meyer May 12, 2008, 11:24:30 PM, you wrote: MM On Mon, 12 May 2008 22:35:31 +0400 Anthony Pankov [EMAIL PROTECTED] wrote: Because BDB: 1. do not need additional installation 2. is part of base system which mean it is mature, reliable and stable MM BDB in the base system is mature, reliable and stable *for what it's MM used for in the base system.* So long as your requirements are covered MM by that usage, you'll be ok. MM The uses I know of for BDB in the base system all consist of databases MM of relatively small items that are changed infrequently, and usually MM with a locking mechanism. From what you've said, this doesn't describe MM your requirements. MM More importantly, from what other people are saying, your requirements MM are ones for which it's known that BDB is *not* reliable, or otherwise MM unsuitable. In particular, an effort is underway to allow parallel MM ports builds, which implies concurrent access to the database, which MM is a known source of problems for BDB. MM mike My requirements is 1. there is no need for SQL 2. processes are sharing db file in concurrent mode 3. reading/writing = 60%/40% With BDB clause 1 - satisfied clause 3 - satisfied (databases of relatively small items that are changed infrequently). Is there a problem with concurrent access? And, most important, is this the ONLY problem? (I still don't understand because of blurred mention of data corruption.) If concurrency is the only problem then: 1. Сan data corruption be avoided? Or this is impossible? 2. How? If all BDB readers would use O_SHLOCK and all writers O_EXLOCK is it guarantee for data integrity? -- Best regards, Anthonymailto:[EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Tue, May 13, 2008 at 03:44:06PM +0400, Anthony Pankov wrote: If concurrency is the only problem then: 1. ?an data corruption be avoided? Or this is impossible? 2. How? Use Sleepycat/Oracle DB instead? The libc DB1.x, despite being mature, really should be deprecated in some manner. I'm sure there are others I've forgotten, but the only thing I can think of in the base system which relies on DB1.x is sendmail (which IMHO should really be removed from the base system and replaced with a small standalone mailer -- but that's been discussed in a previous thread in the past). Even simple ports like postgrey pull in db41, even though they could technically work fine with DB1.x. If all BDB readers would use O_SHLOCK and all writers O_EXLOCK is it guarantee for data integrity? The corruption I've seen in the past results in DB operations failing for no particular reason (what do you mean those are all the records? No! I just inserted a bunch more!). It turns out some of the data in the actual .db file is corrupt -- even when locking is used everywhere. It's as if the code has some weird bug where it'll write bogus data to the file for some reason. I'll ask you this: is there some particular reason you can't just write to a file yourself, using your own/documented file format? Why does DB have to be used? -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Tue, May 13, 2008 at 03:44:06PM +0400, Anthony Pankov wrote: 3. reading/writing = 60%/40% I don't know where you get those numbers from, but they feel *very* wrong from the perspective of someone who actually dealt a lot with those tools. Writing is only a very small part of the operations and typically done a single large batch. For the same reason a single reader/writer lock is perfective fine. Joerg ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On May 12, 2008, at 6:56 PM, Kurt Lidl wrote: Garrett Cooper wrote: On May 12, 2008, at 1:38 AM, Anthony Pankov wrote: Please, can anybody explain what is the problem with BDB (1.86). Is there known caveats of using BDB? Is there some rules which guarantee from curruption or it is fully undesirable to use BDB under high load? It is important for me because of using BDB in my project. On Fri, May 09, 2008 at 01:52:46PM +0200, Joerg Sonnenberger wrote: As one of the persons hacking on pkg_install in pkgsrc/NetBSD, I would *strongly* advisy you against storing the files only in a bdb file. The change of major and complete corruption with bdb185 is high, consider pulling the plug in the middle of a long update. Sunday, May 11, 2008, 5:38:25 PM, you wrote: GC +1. BDB is quite easy to corrupt... BDB isn't ATOMic, like SQL or other DB backends. You mean ACID probably. And there are plenty of SQL databases that aren't ACID either. (e.g. Mysql 4.x, Mysql 5.x w/o the right kind of backing store) -Kurt Yes, you're right. Atomicity is the A in ACID. -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Tue, May 13, 2008 at 03:44:06PM +0400, Anthony Pankov wrote: My requirements is 1. there is no need for SQL 2. processes are sharing db file in concurrent mode 3. reading/writing = 60%/40% With BDB clause 1 - satisfied clause 3 - satisfied (databases of relatively small items that are changed infrequently). Is there a problem with concurrent access? And, most important, is this the ONLY problem? (I still don't understand because of blurred mention of data corruption.) If concurrency is the only problem then: 1. Can data corruption be avoided? Or this is impossible? 2. How? If all BDB readers would use O_SHLOCK and all writers O_EXLOCK is it guarantee for data integrity? This is probably good enough, if your writers also do a fflush() before releasing the lock. If the writing process ever dies horribly, you can still have database corruption. You could turn the thing that does the actual I/O into the database into daemon that fullfils read/write requests from a unix domain socket. In fact, there is a sample application that does exactly that for the newer Sleepycat BDB code. It could probably be adapted or re-written for the 1.8x series code. If it were me, I'd just punt on using the 1.8x code, drop straight into using the BDB 4.6 codebase, and use their transactioning code. It's mature, robust, etc, etc, for a much large set of operations than the 1.8x code ever was. But there's still that licensing issue. -Kurt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Tue, May 13, 2008 at 05:14:52AM -0700, Jeremy Chadwick wrote:
On Tue, May 13, 2008 at 03:44:06PM +0400, Anthony Pankov wrote:
If concurrency is the only problem then:
1. ?an data corruption be avoided? Or this is impossible?
2. How?
Use Sleepycat/Oracle DB instead? The libc DB1.x, despite being
mature, really should be deprecated in some manner.
This catapults back into the arena of stuff that isn't in the
base system. Not to mention I'm not sure that the Oracle BDB
license would allow bundling in the OS as a binary. I doubt it,
but that's a different bikeshed to paint :-)
I'm sure there are others I've forgotten, but the only thing I can think
of in the base system which relies on DB1.x is sendmail (which IMHO
should really be removed from the base system and replaced with a small
standalone mailer -- but that's been discussed in a previous thread in
the past). Even simple ports like postgrey pull in db41, even though
they could technically work fine with DB1.x.
Well, on a 7.0 machine, it looks like the following:
/etc/mail/aliases.db
/etc/{s,}pwd.db
/usr/share/misc/{termcap,vgrindefs}.db
And of course, nvi uses the db code's recno interface
to do all its work.
If all BDB readers would use O_SHLOCK and all writers O_EXLOCK is it
guarantee for data integrity?
The corruption I've seen in the past results in DB operations failing
for no particular reason (what do you mean those are all the records?
No! I just inserted a bunch more!). It turns out some of the data in
the actual .db file is corrupt -- even when locking is used everywhere.
It's as if the code has some weird bug where it'll write bogus data to
the file for some reason.
There are known problems with certain keys corrupting the DB 1.8x
series code. In fact, the release of the 1.86 was an attempt
to solve this problem when the KerberosV people at MIT found
a repeatable key insert sequence that would corrupt things.
(Or at least that's what I remember, it was a long time ago, and
I might have the details wrong.)
-Kurt
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Tue, 13 May 2008 15:44:06 +0400 Anthony Pankov [EMAIL PROTECTED] wrote: Monday, Mike Meyer May 12, 2008, 11:24:30 PM, you wrote: MM On Mon, 12 May 2008 22:35:31 +0400 Anthony Pankov [EMAIL PROTECTED] wrote: Because BDB: 1. do not need additional installation 2. is part of base system which mean it is mature, reliable and stable MM BDB in the base system is mature, reliable and stable *for what it's MM used for in the base system.* So long as your requirements are covered MM by that usage, you'll be ok. MM The uses I know of for BDB in the base system all consist of databases MM of relatively small items that are changed infrequently, and usually MM with a locking mechanism. From what you've said, this doesn't describe MM your requirements. MM More importantly, from what other people are saying, your requirements MM are ones for which it's known that BDB is *not* reliable, or otherwise MM unsuitable. In particular, an effort is underway to allow parallel MM ports builds, which implies concurrent access to the database, which MM is a known source of problems for BDB. MM mike My requirements is 1. there is no need for SQL 2. processes are sharing db file in concurrent mode 3. reading/writing = 60%/40% With BDB clause 1 - satisfied clause 3 - satisfied (databases of relatively small items that are changed infrequently). Actually, I don't think you get #3, because my wording was poor. The things in the base system are databases of small items, where the *database* is changed infrequently. I think what you have is a database of small items where the *items* are changed infrequently, but the database itself changes frequently. mike -- Mike Meyer [EMAIL PROTECTED] http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O ascii ribbon campaign - stop html mail - www.asciiribbon.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
Kurt J. Lidl wrote: This catapults back into the arena of stuff that isn't in the base system. Not to mention I'm not sure that the Oracle BDB license would allow bundling in the OS as a binary. I doubt it, but that's a different bikeshed to paint :-) Is the LGPL of QDBM and TokyoCabinet also a problem? Could even try grovelling with Mikio? (Partially joking there. I assume he chose LGPL because he wants it that way, but people have been known to change licenses for a base system - like this http://blogs.sun.com/aalok/entry/lzma_on_opensolaris) And is the objection to SQL such the sqlite is really out of the running? Anyway, in this case, would writing an RPC server to own the data kill the performance? It should be easier to write something that can save the database atomically and index it in-core. It could be started on demand and shut down after a short inactivity, a bit like tibco's rvd. There are known problems with certain keys corrupting the DB 1.8x series code. In fact, the release of the 1.86 was an attempt to solve this problem when the KerberosV people at MIT found a repeatable key insert sequence that would corrupt things. (Or at least that's what I remember, it was a long time ago, and I might have the details wrong.) Have to say its a little concerning that such 'mature' code is actually problematic. Particularly since I'm not aware of a non-LGPL alternative. Do you have anything by way of a pointer? Google didn't help me here. James ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
BDB corrupt
Please, can anybody explain what is the problem with BDB (1.86). Is there known caveats of using BDB? Is there some rules which guarantee from curruption or it is fully undesirable to use BDB under high load? It is important for me because of using BDB in my project. On Fri, May 09, 2008 at 01:52:46PM +0200, Joerg Sonnenberger wrote: As one of the persons hacking on pkg_install in pkgsrc/NetBSD, I would *strongly* advisy you against storing the files only in a bdb file. The change of major and complete corruption with bdb185 is high, consider pulling the plug in the middle of a long update. Sunday, May 11, 2008, 5:38:25 PM, you wrote: GC +1. BDB is quite easy to corrupt... -- Best regards, Anthonymailto:[EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Monday 12 May 2008 10:38, Anthony Pankov wrote: Please, can anybody explain what is the problem with BDB (1.86). Is there known caveats of using BDB? Is there some rules which guarantee from curruption or it is fully undesirable to use BDB under high load? It is important for me because of using BDB in my project. Interesting. I would have thought that the two processes find out advantages and problems of proposed solutions and choose a solution had a natural ordering other than the one you seem to be using. Jonathan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re[2]: BDB corrupt
So, can anyone make clear about BDB 1.86 (which is a part of base system). When 1. there is no need for SQL 2. processes are sharing db file in concurrent mode (key=value pair) 3. reading/writing = 60%/40% the first idea is to use BDB. Because BDB: 1. do not need additional installation 2. is part of base system which mean it is mature, reliable and stable (otherwise why BDB is still a part of FreeBSD?) Discussion Adding .db support to pkg_tools reveal BDB ability to corrupt data. Can anyone suggest BDB alternative (not GPLed)? Monday, May 12, 2008, 1:53:00 PM, you wrote: JM On Monday 12 May 2008 10:38, Anthony Pankov wrote: Please, can anybody explain what is the problem with BDB (1.86). Is there known caveats of using BDB? Is there some rules which guarantee from curruption or it is fully undesirable to use BDB under high load? It is important for me because of using BDB in my project. JM Interesting. I would have thought that the two processes find out advantages JM and problems of proposed solutions and choose a solution had a natural JM ordering other than the one you seem to be using. JM Jonathan -- Best regards, Anthonymailto:[EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Mon, 12 May 2008 22:35:31 +0400 Anthony Pankov [EMAIL PROTECTED] wrote: Because BDB: 1. do not need additional installation 2. is part of base system which mean it is mature, reliable and stable BDB in the base system is mature, reliable and stable *for what it's used for in the base system.* So long as your requirements are covered by that usage, you'll be ok. The uses I know of for BDB in the base system all consist of databases of relatively small items that are changed infrequently, and usually with a locking mechanism. From what you've said, this doesn't describe your requirements. More importantly, from what other people are saying, your requirements are ones for which it's known that BDB is *not* reliable, or otherwise unsuitable. In particular, an effort is underway to allow parallel ports builds, which implies concurrent access to the database, which is a known source of problems for BDB. mike ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On May 12, 2008, at 1:38 AM, Anthony Pankov wrote: Please, can anybody explain what is the problem with BDB (1.86). Is there known caveats of using BDB? Is there some rules which guarantee from curruption or it is fully undesirable to use BDB under high load? It is important for me because of using BDB in my project. On Fri, May 09, 2008 at 01:52:46PM +0200, Joerg Sonnenberger wrote: As one of the persons hacking on pkg_install in pkgsrc/NetBSD, I would *strongly* advisy you against storing the files only in a bdb file. The change of major and complete corruption with bdb185 is high, consider pulling the plug in the middle of a long update. Sunday, May 11, 2008, 5:38:25 PM, you wrote: GC +1. BDB is quite easy to corrupt... BDB isn't ATOMic, like SQL or other DB backends. -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
Garrett Cooper wrote: On May 12, 2008, at 1:38 AM, Anthony Pankov wrote: Please, can anybody explain what is the problem with BDB (1.86). Is there known caveats of using BDB? Is there some rules which guarantee from curruption or it is fully undesirable to use BDB under high load? It is important for me because of using BDB in my project. On Fri, May 09, 2008 at 01:52:46PM +0200, Joerg Sonnenberger wrote: As one of the persons hacking on pkg_install in pkgsrc/NetBSD, I would *strongly* advisy you against storing the files only in a bdb file. The change of major and complete corruption with bdb185 is high, consider pulling the plug in the middle of a long update. Sunday, May 11, 2008, 5:38:25 PM, you wrote: GC +1. BDB is quite easy to corrupt... BDB isn't ATOMic, like SQL or other DB backends. You mean ACID probably. And there are plenty of SQL databases that aren't ACID either. (e.g. Mysql 4.x, Mysql 5.x w/o the right kind of backing store) -Kurt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
