Re: [Hibernate] in-memory databases vs query-in-memory databases
- Original Message - From: "Adam Megacz" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Saturday, November 29, 2003 3:04 AM Subject: Re: [Hibernate] in-memory databases vs query-in-memory databases > > "Juozas Baliuka" <[EMAIL PROTECTED]> writes: > > Think about Gavin's example ( some of my applications use messaging > > too) , most of my applications are integrated with "legacy" systems, > > some of them use "import" it takes ~2 min. > > Why couldn't you use optimistic concurrency control for this? > Checkpoint the system, do your import, and if the rows you needed > changed during the 2-minute import, start over. There is no locking in my use case ( no concurent write operations on this table ), updated rows are not visible before commit for read operations and import do not block concurent read transactions. It is always possible to find better ways, but it is easy to implement and it works wo problems, I will find a better way if it will stop to to work, bu I see no meaning to solve not existing problems. I am sure there are a good use cases for Prevayler. BTW most of RDMS are designed the same way 1) redo/undo log files 2) tada and index files As I understand Prevayler uses redo log file and memory for data. BTW you can try to use B+Tree indexed file with agresive cache for 2 and performance will be almost the same on small databases and better on large databases ( index file is faster than swap ) > > The "start over" part sounds awful, but the alternative (in a > traditional multiple-outstanding-transaction ACID system) is to leave > those rows locked for two minutes, which is (IMHO) equally awful. > > Perhaps I'm missing something here; I've dealt with a lot of > highly-concurrent systems and implemented some hierarchical lock > managers, but I've never dealt with attempted distributed transactions > before. Could you elaborate on the advantages of pessimistic locking > in this situation? > > BTW, I don't really think Prevayler is a complete system, but I think > they have hit upon a good layering that would greatly simplify a lot > of systems. They've only built the bottommost (very thin) layer; at a > bare minimum you have to add optimistic locking, query parsing, and > query optimization to get anything useful. > > - a > > -- > "Education is not filling a bucket but lighting a fire." -- WB Yeats > > > --- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > ___ > hibernate-devel mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/hibernate-devel --- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ ___ hibernate-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/hibernate-devel
Re: [Hibernate] in-memory databases vs query-in-memory databases
"Juozas Baliuka" <[EMAIL PROTECTED]> writes: > Think about Gavin's example ( some of my applications use messaging > too) , most of my applications are integrated with "legacy" systems, > some of them use "import" it takes ~2 min. Why couldn't you use optimistic concurrency control for this? Checkpoint the system, do your import, and if the rows you needed changed during the 2-minute import, start over. The "start over" part sounds awful, but the alternative (in a traditional multiple-outstanding-transaction ACID system) is to leave those rows locked for two minutes, which is (IMHO) equally awful. Perhaps I'm missing something here; I've dealt with a lot of highly-concurrent systems and implemented some hierarchical lock managers, but I've never dealt with attempted distributed transactions before. Could you elaborate on the advantages of pessimistic locking in this situation? BTW, I don't really think Prevayler is a complete system, but I think they have hit upon a good layering that would greatly simplify a lot of systems. They've only built the bottommost (very thin) layer; at a bare minimum you have to add optimistic locking, query parsing, and query optimization to get anything useful. - a -- "Education is not filling a bucket but lighting a fire." -- WB Yeats --- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ ___ hibernate-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/hibernate-devel
Re: [Hibernate] in-memory databases vs query-in-memory databases
Yes, 3 is a good way if transactionsa are very "short" (transaction per operation aka autocommit), it solves deadlock problem and most of conflicts ( optimistic loclking can solve the rest ). But I do not think all of applications can use this way, most of my applications can't. Think about Gavin's example ( some of my applications use messaging too) , most of my applications are integrated with "legacy" systems, some of them use "import" it takes ~2 min. to import some files, but transactions can not wait for this import ( sometimes of transactions are orders in stock exhange ). If you are talking about applications like "Web blog", Prevayler must be a good choise. > > "Juozas Baliuka" <[EMAIL PROTECTED]> writes: > > Databases lock updated and deleted rows only and transaction blocks > > on conflict only, it never block query or not conflictiong updates, > > I see three ways to solve update conflict: > > > 1) block transaction > > 2) abort transaction > > 3) no concurent transactions > > > Looks like 3 is prefered in Prevayler, is not it ? > > Right. Interestingly, there is no performance hit to (3) *if* all > your data is in memory. Oracle is a multi-billion dollar company > mainly because this is rarely a valid assumption. > > I'm wondering if a weaker assumption (only live data must be in > memory) would work for a somewhat larger portion of the applications > people are using databases for. > > - a > > -- > "Education is not filling a bucket but lighting a fire." -- WB Yeats > > > --- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > ___ > hibernate-devel mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/hibernate-devel --- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ ___ hibernate-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/hibernate-devel
Re: [Hibernate] in-memory databases vs query-in-memory databases
"Juozas Baliuka" <[EMAIL PROTECTED]> writes: > Databases lock updated and deleted rows only and transaction blocks > on conflict only, it never block query or not conflictiong updates, > I see three ways to solve update conflict: > 1) block transaction > 2) abort transaction > 3) no concurent transactions > Looks like 3 is prefered in Prevayler, is not it ? Right. Interestingly, there is no performance hit to (3) *if* all your data is in memory. Oracle is a multi-billion dollar company mainly because this is rarely a valid assumption. I'm wondering if a weaker assumption (only live data must be in memory) would work for a somewhat larger portion of the applications people are using databases for. - a -- "Education is not filling a bucket but lighting a fire." -- WB Yeats --- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ ___ hibernate-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/hibernate-devel
Re: [Hibernate] in-memory databases vs query-in-memory databases
- Original Message - From: "Adam Megacz" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Thursday, November 27, 2003 4:46 AM Subject: [Hibernate] in-memory databases vs query-in-memory databases > > Okay, this isn't 100% hibernate-related, but I figure you guys think a > lot about stuff like this. > > Prevayler's pitch goes something like this: "if all your data fits in > memory, you don't need concurrent transactions, because every > transaction will be demanding the same resource (cpu/memory > bandwidth)." Obviously this doesn't work if your data set is larger > than (or might grow to become larger than) your available memory. I > can see this being a fairly common situation. > > What if you were sure that the total amount of data needed to perform > any given transaction could fit in memory? In that case, could you: > > 1) Load all pages needed by a transaction and pin them in memory > > 2) Perform the transaction > > Any number of transactions could be doing (1) at the same time, but > there would be only a single thread permitted to do (2). Once a > transaction has pinned all the pages it will need, it queues itself > for (2). Once (2) is complete, the pages are unpinned (if the > reference count for the page falls to zero, of course -- multiple > transactions can pin the same page). > > This would eliminate the need for fine grained locking and most of the > other stuff that makes databases complex, since only one thread is > running transactions. Adding more transaction threads wouldn't win > you anything, because a transaction on strictly-in-memory data never > blocks. Databases lock updated and deleted rows only and transaction blocks on conflict only, it never block query or not conflictiong updates, I see three ways to solve update conflict: 1) block transaction 2) abort transaction 3) no concurent transactions Looks like 3 is prefered in Prevayler, is not it ? > > The downside is that you have to be certain that you'll never run a > query that needs more data than you have memory. OTOH, 4GB of ram is > pretty cheap these days. And if your query needs to pull 4GB of data > off the disk [*], well, it's going to be incredibly slow in the first > place. I don't think too many people run queries like that. > > - a > > -- > "Education is not filling a bucket but lighting a fire." -- WB Yeats > > [*] Note that simply selecting a 4GB table doesn't actually need to > pull 4GB off the disk. > > > --- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > ___ > hibernate-devel mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/hibernate-devel --- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ ___ hibernate-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/hibernate-devel
Re: [Hibernate] in-memory databases vs query-in-memory databases
Gavin King <[EMAIL PROTECTED]> writes: > What of you need to make a remote call, or send a message inside the > txn? Your analysis assumes that the system is not distributed in any way. :) Optimistic locking can be layered on top of a system like this. I don't believe that non-optimistically-locked ACID transactions are possible in a distributed environment. http://www.nada.kth.se/kurser/kth/2D5340/wwwbook/node5.html#SECTION00211000 (The famous "Byzantine Agreement" problem). In other words, you have to allow the transaction processor to force a rollback -- if you don't, the remote call could block forever (for example, due to a network failure) and halt the database. Local I/O is this funny case where it can block, but you know it won't block forever, so people want to be able to do local I/O inside a transaction and not have to write rollback handlers. - a -- "Education is not filling a bucket but lighting a fire." -- WB Yeats --- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ ___ hibernate-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/hibernate-devel
Re: [Hibernate] in-memory databases vs query-in-memory databases
What of you need to make a remote call, or send a message inside the txn? Your analysis assumes that the system is not distributed in any way. :) Adam Megacz wrote: Okay, this isn't 100% hibernate-related, but I figure you guys think a lot about stuff like this. Prevayler's pitch goes something like this: "if all your data fits in memory, you don't need concurrent transactions, because every transaction will be demanding the same resource (cpu/memory bandwidth)." Obviously this doesn't work if your data set is larger than (or might grow to become larger than) your available memory. I can see this being a fairly common situation. What if you were sure that the total amount of data needed to perform any given transaction could fit in memory? In that case, could you: 1) Load all pages needed by a transaction and pin them in memory 2) Perform the transaction Any number of transactions could be doing (1) at the same time, but there would be only a single thread permitted to do (2). Once a transaction has pinned all the pages it will need, it queues itself for (2). Once (2) is complete, the pages are unpinned (if the reference count for the page falls to zero, of course -- multiple transactions can pin the same page). This would eliminate the need for fine grained locking and most of the other stuff that makes databases complex, since only one thread is running transactions. Adding more transaction threads wouldn't win you anything, because a transaction on strictly-in-memory data never blocks. The downside is that you have to be certain that you'll never run a query that needs more data than you have memory. OTOH, 4GB of ram is pretty cheap these days. And if your query needs to pull 4GB of data off the disk [*], well, it's going to be incredibly slow in the first place. I don't think too many people run queries like that. - a -- Gavin King JBoss Group +61 410534454 http://hibernate.org --- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ ___ hibernate-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/hibernate-devel
[Hibernate] in-memory databases vs query-in-memory databases
Okay, this isn't 100% hibernate-related, but I figure you guys think a lot about stuff like this. Prevayler's pitch goes something like this: "if all your data fits in memory, you don't need concurrent transactions, because every transaction will be demanding the same resource (cpu/memory bandwidth)." Obviously this doesn't work if your data set is larger than (or might grow to become larger than) your available memory. I can see this being a fairly common situation. What if you were sure that the total amount of data needed to perform any given transaction could fit in memory? In that case, could you: 1) Load all pages needed by a transaction and pin them in memory 2) Perform the transaction Any number of transactions could be doing (1) at the same time, but there would be only a single thread permitted to do (2). Once a transaction has pinned all the pages it will need, it queues itself for (2). Once (2) is complete, the pages are unpinned (if the reference count for the page falls to zero, of course -- multiple transactions can pin the same page). This would eliminate the need for fine grained locking and most of the other stuff that makes databases complex, since only one thread is running transactions. Adding more transaction threads wouldn't win you anything, because a transaction on strictly-in-memory data never blocks. The downside is that you have to be certain that you'll never run a query that needs more data than you have memory. OTOH, 4GB of ram is pretty cheap these days. And if your query needs to pull 4GB of data off the disk [*], well, it's going to be incredibly slow in the first place. I don't think too many people run queries like that. - a -- "Education is not filling a bucket but lighting a fire." -- WB Yeats [*] Note that simply selecting a 4GB table doesn't actually need to pull 4GB off the disk. --- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ ___ hibernate-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/hibernate-devel
