RE: Lightweight database
I have to admit that I have not used it. But I have been following it for a while and looks promising. Sterling NoSQL OODB for .NET 4.0 http://sterling.codeplex.com/ I've also just came across UnQLite, an embedded NoSQL (Key/Value store and Document-store) database engine, http://www.unqlite.org/ that looks interesting, but doesn't seem to have managed code wrappers yet. Reminds me of SQLite, but you know, without the SQL and BSD 2-clause instead of Public Domain ;) - Original Message - Subject: Lightweight database From: Greg Keogh g...@mira.net Date: 6/3/13 10:24 pm To: ozDotNet ozdotnet@ozdotnet.com Folks, an app has just grown with a new feature that needs to store of users, jobs and reports and the joins between them, If I was using SQLite it would be 3 tables with joins. However, rather than use SQLite this time I'd like to consider an alternative that's even more lightweight to setup and use. The app does not currently use any database technology and the guys managing the project are actually scared of them. Can anyone recommend an in-process database (not necessarily relational!) that is has a friendly managed API, small footprint, not too complicated and is easy to get going? I know this is a lot to ask, but there may be some NoSQL options around that I'm not aware of. The most important issues for me are: (1) Minimal dependencies (2) Simple managed API. I'm running a few web searches now for such things, and I can see Redis, Mongo, Couch, Raven, db4o, Cassandra, Eloquera, Lucene, and the list goes on and on. There are too many choices and it would take many days of hard slog to work out which one would be suitable. So perhaps someone has already been through this process?! I've been tempted many times over the last 10 years to write a pure managed single-file database with indexes, and nothing much else (no transactions, no client-server, no schemas, etc). However, I decided to leave it to the experts, and it looks like there are too many of them, and they all over-engineer their works. Cheers, Greg K
Re: Lightweight database
I'm using siaqodb (http://siaqodb.com/) which is a lightweight .Net object database and I'm very happy with it. Very fast and dead easy to use linq syntax. On Wed, Jun 5, 2013 at 12:42 AM, a...@nomscon.com wrote: I have to admit that I have not used it. But I have been following it for a while and looks promising. Sterling NoSQL OODB for .NET 4.0 http://sterling.codeplex.com/ I've also just came across UnQLite, an embedded NoSQL (Key/Value store and Document-store) database engine, http://www.unqlite.org/ that looks interesting, but doesn't seem to have managed code wrappers yet. Reminds me of SQLite, but you know, without the SQL and BSD 2-clause instead of Public Domain ;) - Original Message - Subject: Lightweight database From: Greg Keogh g...@mira.net Date: 6/3/13 10:24 pm To: ozDotNet ozdotnet@ozdotnet.com Folks, an app has just grown with a new feature that needs to store of users, jobs and reports and the joins between them, If I was using SQLite it would be 3 tables with joins. However, rather than use SQLite this time I'd like to consider an alternative that's even more lightweight to setup and use. The app does not currently use any database technology and the guys managing the project are actually scared of them. Can anyone recommend an in-process database (not necessarily relational!) that is has a friendly managed API, small footprint, not too complicated and is easy to get going? I know this is a lot to ask, but there may be some NoSQL options around that I'm not aware of. The most important issues for me are: (1) *Minimal dependencies* (2) *Simple managed API*. I'm running a few web searches now for such things, and I can see Redis, Mongo, Couch, Raven, db4o, Cassandra, Eloquera, Lucene, and the list goes on and on. There are too many choices and it would take many days of hard slog to work out which one would be suitable. So perhaps someone has already been through this process?! I've been tempted many times over the last 10 years to write a pure managed single-file database with indexes, and nothing much else (no transactions, no client-server, no schemas, etc). However, I decided to leave it to the experts, and it looks like there are too many of them, and they all over-engineer their works. Cheers, Greg K
RE: Lightweight database
Greg, We have been developing with SynergyDE for almost 30 years (Synergex) and they have an extremely good ISAM file system with ODBC for access externally via SQL statements. They have just recently released their ISAM file system as a NOSQL offering which may meet your criteria. www.synergex.com [image: cid:image001.png@01CE6128.50483A60] *Regards,* *Steven Parish*** *Software Architect | BusinessCraft Pty Ltd | www.businesscraft.com* Tel: (02) 4965 | Mob: 0417 688 599 | Fax: (02) 4965 5333 | Level 1, 270 Turton Road, New Lambton, NSW 2305 -- *Disclaimer* The information contained in this and associated e-mails and any attachments are confidential and were only intended solely for the use of the addressee. If you have received this e-mail in error, please notify the identified sender by return e-mail and then delete all copies of this and associated emails and their attachments from your computer and or your computers system. Opinions, conclusions and other information in this message that do not relate to the business of BusinessCraft Pty Ltd shall be understood as neither given nor endorsed by it. It is your responsibility to check the message and any attachments for any viruses before use. BusinessCraft does not accept liability for any loss or damage that may result, directly or indirectly from your receipt of this message or any attachments contained within it. *From:* ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] *On Behalf Of *Greg Keogh *Sent:* Tuesday, 4 June 2013 12:25 PM *To:* ozDotNet *Subject:* Lightweight database Folks, an app has just grown with a new feature that needs to store of users, jobs and reports and the joins between them, If I was using SQLite it would be 3 tables with joins. However, rather than use SQLite this time I'd like to consider an alternative that's even more lightweight to setup and use. The app does not currently use any database technology and the guys managing the project are actually scared of them. Can anyone recommend an in-process database (not necessarily relational!) that is has a friendly managed API, small footprint, not too complicated and is easy to get going? I know this is a lot to ask, but there may be some NoSQL options around that I'm not aware of. The most important issues for me are: (1) *Minimal dependencies* (2) *Simple managed API*. I'm running a few web searches now for such things, and I can see Redis, Mongo, Couch, Raven, db4o, Cassandra, Eloquera, Lucene, and the list goes on and on. There are too many choices and it would take many days of hard slog to work out which one would be suitable. So perhaps someone has already been through this process?! I've been tempted many times over the last 10 years to write a pure managed single-file database with indexes, and nothing much else (no transactions, no client-server, no schemas, etc). However, I decided to leave it to the experts, and it looks like there are too many of them, and they all over-engineer their works. Cheers, Greg K image001.png
Re: Lightweight database
Thanks everyone, now there are even more choices, all with different qualities. Too much choice, too little time. Finding a desirable local in-process database is such a time-consuming task that I will put it in the background and look at the candidates in my spare time in the hope that something will eventually pop-out. In the meantime I'll stick to using XML files in the format *job#user.xml*, even though it feels a bit primitive. I'm guessing there might be less that a hundred files at most, so using the file system and a glob pattern as a poor man's query will probably be acceptable for now. Greg K
Lightweight database
Folks, an app has just grown with a new feature that needs to store of users, jobs and reports and the joins between them, If I was using SQLite it would be 3 tables with joins. However, rather than use SQLite this time I'd like to consider an alternative that's even more lightweight to setup and use. The app does not currently use any database technology and the guys managing the project are actually scared of them. Can anyone recommend an in-process database (not necessarily relational!) that is has a friendly managed API, small footprint, not too complicated and is easy to get going? I know this is a lot to ask, but there may be some NoSQL options around that I'm not aware of. The most important issues for me are: (1) *Minimal dependencies* (2) *Simple managed API*. I'm running a few web searches now for such things, and I can see Redis, Mongo, Couch, Raven, db4o, Cassandra, Eloquera, Lucene, and the list goes on and on. There are too many choices and it would take many days of hard slog to work out which one would be suitable. So perhaps someone has already been through this process?! I've been tempted many times over the last 10 years to write a pure managed single-file database with indexes, and nothing much else (no transactions, no client-server, no schemas, etc). However, I decided to leave it to the experts, and it looks like there are too many of them, and they all over-engineer their works. Cheers, Greg K
Re: Lightweight database
Not done this myself recently but what about an MDB file? Should '.net'/Ado/OLEDB/ODBC out of the box and I think the licensing is that if you have developer edition of office then you can distribute the runtime for free (but I could be wrong). Preet On 4 June 2013 14:24, Greg Keogh g...@mira.net wrote: Folks, an app has just grown with a new feature that needs to store of users, jobs and reports and the joins between them, If I was using SQLite it would be 3 tables with joins. However, rather than use SQLite this time I'd like to consider an alternative that's even more lightweight to setup and use. The app does not currently use any database technology and the guys managing the project are actually scared of them. Can anyone recommend an in-process database (not necessarily relational!) that is has a friendly managed API, small footprint, not too complicated and is easy to get going? I know this is a lot to ask, but there may be some NoSQL options around that I'm not aware of. The most important issues for me are: (1) *Minimal dependencies* (2) *Simple managed API*. I'm running a few web searches now for such things, and I can see Redis, Mongo, Couch, Raven, db4o, Cassandra, Eloquera, Lucene, and the list goes on and on. There are too many choices and it would take many days of hard slog to work out which one would be suitable. So perhaps someone has already been through this process?! I've been tempted many times over the last 10 years to write a pure managed single-file database with indexes, and nothing much else (no transactions, no client-server, no schemas, etc). However, I decided to leave it to the experts, and it looks like there are too many of them, and they all over-engineer their works. Cheers, Greg K -- regards, Preet, Overlooking the Ocean, Auckland
Re: Lightweight database
Greg, You could try just serialising to xml files. Of course, no transactions, not really much in the way of relationships. But it's very simple. It kind of depends on the situation. I recently had to convert an app that was using sqlce to using xml files because of reliability issues on the devices in question. It solved a lot of problem and mad the code simpler. David If we can hit that bullseye, the rest of the dominoes will fall like a house of cards... checkmate! -Zapp Brannigan, Futurama On 4 June 2013 12:24, Greg Keogh g...@mira.net wrote: Folks, an app has just grown with a new feature that needs to store of users, jobs and reports and the joins between them, If I was using SQLite it would be 3 tables with joins. However, rather than use SQLite this time I'd like to consider an alternative that's even more lightweight to setup and use. The app does not currently use any database technology and the guys managing the project are actually scared of them. Can anyone recommend an in-process database (not necessarily relational!) that is has a friendly managed API, small footprint, not too complicated and is easy to get going? I know this is a lot to ask, but there may be some NoSQL options around that I'm not aware of. The most important issues for me are: (1) *Minimal dependencies* (2) *Simple managed API*. I'm running a few web searches now for such things, and I can see Redis, Mongo, Couch, Raven, db4o, Cassandra, Eloquera, Lucene, and the list goes on and on. There are too many choices and it would take many days of hard slog to work out which one would be suitable. So perhaps someone has already been through this process?! I've been tempted many times over the last 10 years to write a pure managed single-file database with indexes, and nothing much else (no transactions, no client-server, no schemas, etc). However, I decided to leave it to the experts, and it looks like there are too many of them, and they all over-engineer their works. Cheers, Greg K
Re: Lightweight database
You could try just serialising to xml files. Your powers are strong young Jedi. That's what I'm temporarily doing now, but I'm using the filename as a two-part key like 'jobname#username.xml' which is a bit 1970s! -- Greg
Re: Lightweight database
Not done this myself recently but what about an MDB file? Uneasy about what dependencies it has on other stuff being installed. I'd still have to code a lot of raw ADO and use DataSets or readers and it would get a bit verbose -- Greg
RE: Lightweight database
I think one of the first design parameters you need is how much data. If the amount of data is small, then you can get away with xml. If it starts slowing down you need to move to a different format such as a custom binary. XML - quick to code, easy, the customer can own their own data and work it out themselves if needed. Can be slow as data increases. Binary - slower to code, needs a lot of rules up front, speedy in use, needs custom export as xml functions (perhaps) In both cases you end up loading all the data into memory, so there's a real upper limit. If you get into paging it gets too complex and it's time for a real data provider. XML suffers from having to parse the file, looking for special characters, then a lookup match for each field for each record. Binary on the other hand you can write the length of each record as the record prefix so you just read in chunks of bytes, splitting each field either at fixed length (really quick) or having variable length prefixed with their length. Eg User FirstNameGreg/FirstName LastNameKeogh/LastName User Becomes 30 4 Greg 5 Keogh Where the 30 is the first dword (4 bytes), 4 is the next (4 bytes), then parse the string which is 4 Char (wide is 8 bytes) next 4 bytes is the value of 5, then parse the 5 chars (10 bytes) The problem with doing binary is if they change the schema, it's a real PITA to change the code. |-Original Message- |From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet- |boun...@ozdotnet.com] On Behalf Of Greg Keogh |Sent: Tuesday, 4 June 2013 12:25 PM |To: ozDotNet |Subject: Lightweight database | |Folks, an app has just grown with a new feature that needs to store of users, |jobs and reports and the joins between them, If I was using SQLite it would be |3 tables with joins. However, rather than use SQLite this time I'd like to consider |an alternative that's even more lightweight to setup and use. The app does not |currently use any database technology and the guys managing the project are |actually scared of them. | |Can anyone recommend an in-process database (not necessarily relational!) that |is has a friendly managed API, small footprint, not too complicated and is easy to |get going? I know this is a lot to ask, but there may be some NoSQL options |around that I'm not aware of. The most important issues for me are: (1) Minimal |dependencies (2) Simple managed API. | |I'm running a few web searches now for such things, and I can see Redis, Mongo, |Couch, Raven, db4o, Cassandra, Eloquera, Lucene, and the list goes on and on. |There are too many choices and it would take many days of hard slog to work out |which one would be suitable. So perhaps someone has already been through this |process?! | |I've been tempted many times over the last 10 years to write a pure managed |single-file database with indexes, and nothing much else (no transactions, no |client-server, no schemas, etc). However, I decided to leave it to the experts, and |it looks like there are too many of them, and they all over-engineer their works. | |Cheers, |Greg K
RE: Lightweight database
Agreed. Even with XML, you might to deal with what happens if the schema changes ie: can your new app read an old saved set of data? Regards, Greg Dr Greg Low 1300SQLSQL (1300 775 775) office | +61 419201410 mobile│ +61 3 8676 4913 fax SQL Down Under | Web: www.sqldownunder.com I think one of the first design parameters you need is how much data. If the amount of data is small, then you can get away with xml. If it starts slowing down you need to move to a different format such as a custom binary. XML - quick to code, easy, the customer can own their own data and work it out themselves if needed. Can be slow as data increases. Binary - slower to code, needs a lot of rules up front, speedy in use, needs custom export as xml functions (perhaps) In both cases you end up loading all the data into memory, so there's a real upper limit. If you get into paging it gets too complex and it's time for a real data provider. XML suffers from having to parse the file, looking for special characters, then a lookup match for each field for each record. Binary on the other hand you can write the length of each record as the record prefix so you just read in chunks of bytes, splitting each field either at fixed length (really quick) or having variable length prefixed with their length. Eg User FirstNameGreg/FirstName LastNameKeogh/LastName User Becomes 30 4 Greg 5 Keogh Where the 30 is the first dword (4 bytes), 4 is the next (4 bytes), then parse the string which is 4 Char (wide is 8 bytes) next 4 bytes is the value of 5, then parse the 5 chars (10 bytes) The problem with doing binary is if they change the schema, it's a real PITA to change the code. |-Original Message- |From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet- |boun...@ozdotnet.com] On Behalf Of Greg Keogh |Sent: Tuesday, 4 June 2013 12:25 PM |To: ozDotNet |Subject: Lightweight database | |Folks, an app has just grown with a new feature that needs to store of users, |jobs and reports and the joins between them, If I was using SQLite |it would be |3 tables with joins. However, rather than use SQLite this time I'd like |to consider |an alternative that's even more lightweight to setup and use. The app |does not |currently use any database technology and the guys managing the project |are actually scared of them. | |Can anyone recommend an in-process database (not necessarily |relational!) that |is has a friendly managed API, small footprint, not too complicated and |is easy to |get going? I know this is a lot to ask, but there may be some NoSQL |options around that I'm not aware of. The most important issues for me |are: (1) Minimal |dependencies (2) Simple managed API. | |I'm running a few web searches now for such things, and I can see |Redis, Mongo, |Couch, Raven, db4o, Cassandra, Eloquera, Lucene, and the list goes on |and on. |There are too many choices and it would take many days of hard slog to |work out |which one would be suitable. So perhaps someone has already been |through this |process?! | |I've been tempted many times over the last 10 years to write a pure |managed single-file database with indexes, and nothing much else (no |transactions, no |client-server, no schemas, etc). However, I decided to leave it to the experts, and |it looks like there are too many of them, and they all over-engineer |their works. | |Cheers, |Greg K
RE: Lightweight database
Yep, but with XML the new app will read the old data without a problem (unless there has been field delete's or renaming). With the binary, there's a bit more to the coding (and testing) of the layout. That said, if you write the version number in the data file's header, you can switch out to a conversion routine which would use a lot of the old parser anyway. Where-as the XML is a b it harder to switch out to different parsers. So there's probably not a lot in it, but I know it's the binary ones that seem to get me to have get the pad and pen or a spreadsheet open while I do it ;) |-Original Message- |From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet- |boun...@ozdotnet.com] On Behalf Of GregAtGregLowDotCom |Sent: Tuesday, 4 June 2013 2:25 PM |To: 'ozDotNet' |Subject: RE: Lightweight database | |Agreed. Even with XML, you might to deal with what happens if the schema |changes ie: can your new app read an old saved set of data? | |Regards, | |Greg | |Dr Greg Low | |1300SQLSQL (1300 775 775) office | +61 419201410 mobile│ +61 3 8676 4913 fax | |SQL Down Under | Web: www.sqldownunder.com | |I think one of the first design parameters you need is how much data. If the |amount of data is small, then you can get away with xml. If it starts slowing down |you need to move to a different format such as a custom binary. | |XML - quick to code, easy, the customer can own their own data and work it out |themselves if needed. Can be slow as data increases. |Binary - slower to code, needs a lot of rules up front, speedy in use, needs custom |export as xml functions (perhaps) | |In both cases you end up loading all the data into memory, so there's a real upper |limit. If you get into paging it gets too complex and it's time for a real data |provider. | |XML suffers from having to parse the file, looking for special characters, then a |lookup match for each field for each record. |Binary on the other hand you can write the length of each record as the record |prefix so you just read in chunks of bytes, splitting each field either at fixed length |(really quick) or having variable length prefixed with their length. |Eg | |User | FirstNameGreg/FirstName |LastNameKeogh/LastName |User | |Becomes | |30 4 Greg 5 Keogh | |Where the 30 is the first dword (4 bytes), 4 is the next (4 bytes), then parse the |string which is 4 Char (wide is 8 bytes) next 4 bytes is the value of 5, then parse |the 5 chars (10 bytes) | |The problem with doing binary is if they change the schema, it's a real PITA to |change the code. | | | ||-Original Message- ||From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet- ||boun...@ozdotnet.com] On Behalf Of Greg Keogh ||Sent: Tuesday, 4 June 2013 12:25 PM ||To: ozDotNet ||Subject: Lightweight database || ||Folks, an app has just grown with a new feature that needs to store of |users, ||jobs and reports and the joins between them, If I was using SQLite ||it |would be ||3 tables with joins. However, rather than use SQLite this time I'd like ||to |consider ||an alternative that's even more lightweight to setup and use. The app ||does |not ||currently use any database technology and the guys managing the project ||are actually scared of them. || ||Can anyone recommend an in-process database (not necessarily ||relational!) |that ||is has a friendly managed API, small footprint, not too complicated and ||is |easy to ||get going? I know this is a lot to ask, but there may be some NoSQL ||options around that I'm not aware of. The most important issues for me ||are: (1) |Minimal ||dependencies (2) Simple managed API. || ||I'm running a few web searches now for such things, and I can see ||Redis, |Mongo, ||Couch, Raven, db4o, Cassandra, Eloquera, Lucene, and the list goes on ||and |on. ||There are too many choices and it would take many days of hard slog to ||work |out ||which one would be suitable. So perhaps someone has already been ||through |this ||process?! || ||I've been tempted many times over the last 10 years to write a pure ||managed single-file database with indexes, and nothing much else (no ||transactions, |no ||client-server, no schemas, etc). However, I decided to leave it to the |experts, and ||it looks like there are too many of them, and they all over-engineer ||their |works. || ||Cheers, ||Greg K | |
RE: Lightweight database
Yep, that's what I was getting at. The object structure might change between versions. Extra columns won't be an issue but other sorts of changes probably would be. Regards, Greg Dr Greg Low 1300SQLSQL (1300 775 775) office | +61 419201410 mobile│ +61 3 8676 4913 fax SQL Down Under | Web: www.sqldownunder.com Yep, but with XML the new app will read the old data without a problem (unless there has been field delete's or renaming). With the binary, there's a bit more to the coding (and testing) of the layout. That said, if you write the version number in the data file's header, you can switch out to a conversion routine which would use a lot of the old parser anyway. Where-as the XML is a b it harder to switch out to different parsers. So there's probably not a lot in it, but I know it's the binary ones that seem to get me to have get the pad and pen or a spreadsheet open while I do it ;) |-Original Message- |From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet- |boun...@ozdotnet.com] On Behalf Of GregAtGregLowDotCom |Sent: Tuesday, 4 June 2013 2:25 PM |To: 'ozDotNet' |Subject: RE: Lightweight database | |Agreed. Even with XML, you might to deal with what happens if the |schema changes ie: can your new app read an old saved set of data? | |Regards, | |Greg | |Dr Greg Low | |1300SQLSQL (1300 775 775) office | +61 419201410 mobile│ +61 3 8676 |4913 fax | |SQL Down Under | Web: www.sqldownunder.com | |I think one of the first design parameters you need is how much data. |If the |amount of data is small, then you can get away with xml. If it starts slowing down |you need to move to a different format such as a custom binary. | |XML - quick to code, easy, the customer can own their own data and work |it out |themselves if needed. Can be slow as data increases. |Binary - slower to code, needs a lot of rules up front, speedy in use, needs custom |export as xml functions (perhaps) | |In both cases you end up loading all the data into memory, so there's a real upper |limit. If you get into paging it gets too complex and it's time for a |real data |provider. | |XML suffers from having to parse the file, looking for special |characters, then a |lookup match for each field for each record. |Binary on the other hand you can write the length of each record as the record |prefix so you just read in chunks of bytes, splitting each field either |at fixed length |(really quick) or having variable length prefixed with their length. |Eg | |User | FirstNameGreg/FirstName |LastNameKeogh/LastName |User | |Becomes | |30 4 Greg 5 Keogh | |Where the 30 is the first dword (4 bytes), 4 is the next (4 bytes), |then parse the |string which is 4 Char (wide is 8 bytes) next 4 bytes is the value of |5, then parse |the 5 chars (10 bytes) | |The problem with doing binary is if they change the schema, it's a real PITA to |change the code. | | | ||-Original Message- ||From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet- ||boun...@ozdotnet.com] On Behalf Of Greg Keogh ||Sent: Tuesday, 4 June 2013 12:25 PM ||To: ozDotNet ||Subject: Lightweight database || ||Folks, an app has just grown with a new feature that needs to store ||of |users, ||jobs and reports and the joins between them, If I was using SQLite ||it |would be ||3 tables with joins. However, rather than use SQLite this time I'd ||like to |consider ||an alternative that's even more lightweight to setup and use. The app ||does |not ||currently use any database technology and the guys managing the ||project are actually scared of them. || ||Can anyone recommend an in-process database (not necessarily ||relational!) |that ||is has a friendly managed API, small footprint, not too complicated ||and is |easy to ||get going? I know this is a lot to ask, but there may be some NoSQL ||options around that I'm not aware of. The most important issues for me ||are: (1) |Minimal ||dependencies (2) Simple managed API. || ||I'm running a few web searches now for such things, and I can see ||Redis, |Mongo, ||Couch, Raven, db4o, Cassandra, Eloquera, Lucene, and the list goes on ||and |on. ||There are too many choices and it would take many days of hard slog to ||work |out ||which one would be suitable. So perhaps someone has already been ||through |this ||process?! || ||I've been tempted many times over the last 10 years to write a pure ||managed single-file database with indexes, and nothing much else (no ||transactions, |no ||client-server, no schemas, etc). However, I decided to leave it to the |experts, and ||it looks like there are too many of them, and they all over-engineer ||their |works. || ||Cheers, ||Greg K | |
