[Firebird-devel] Collection of garbage in record with uncommitted version
Hello, All. If a record has uncommitted head version created by active transaction and some garbage in backversions "tail", can anyone (background GC thread, sweep or parallel transaction) wipe this garbage from the tail while the transaction is still active? I can't understand why list_staying() is so picky: it scans for every next version from the beginning which raise its complexity to O(N^2). -- WBR, SD. -- "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Collection of garbage in record with uncommitted version
> If a record has uncommitted head version created by active transaction and > some garbage > in backversions "tail", can anyone (background GC thread, sweep or parallel > transaction) > wipe this garbage from the tail while the transaction is still active? Yes > I can't understand why list_staying() is so picky: it scans for every next > version from > the beginning which raise its complexity to O(N^2). list_staying used to undo dead record version. It is not a most often operation. And, yes, it is terrible inefficient. It restarts from the beginning of the versions chain when need next record version because : a) it used handoff's to make sure backpointers is valid b) data page with target version is released Regards, Vlad -- "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Collection of garbage in record with uncommitted version
13.05.2014 13:36, Vlad Khorsun wrote: > Yes How? If I remember your lesson right, garbage in a record is collected only if head version is marked with transaction number lesser that OAT. > list_staying used to undo dead record version. It is not a most often > operation. Actually, it is used in following cases: 1) Undo with VIO_backout(). 2) Undo of update_in_place() in VIO_verb_cleanup(). 3) Applying third and following changes of the same record in one transaction. -- WBR, SD. -- "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Collection of garbage in record with uncommitted version
> 13.05.2014 13:36, Vlad Khorsun wrote: >> Yes > > How? If I remember your lesson right, garbage in a record is collected only > if head > version is marked with transaction number lesser that OAT. This is a most often case. But, if you look at VIO_chase_record_version, you'll see that concurrency (snapshot) transactions could pass not visible for them versions of record and stop at some point. And in this point record version could be - visible for current tx - mature for GC - still have backversions In this case, cooperative GC could happen (i.e. purge will be called) before return to the caller (VIO_next or VIO_data). >> list_staying used to undo dead record version. It is not a most often >> operation. > > Actually, it is used in following cases: > > 1) Undo with VIO_backout(). > 2) Undo of update_in_place() in VIO_verb_cleanup(). > 3) Applying third and following changes of the same record in one transaction. I know. I just a bit lazy when need to wrtite too much words, especially in Engilish :) In any case - all your points above is cases of undo operation. Regards, Vlad -- "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] CORE-3515
> 12.05.2014 16:30, Vlad Khorsun wrote:
>> Sure. And this is why i wrote recently
>>
>> > currently used way of GC ... not 100% safe in highly concurrent
>> environment.
>
> Can this case be fixed by moving call to delete_record() below
> garbage_collect()?
> Something like this:
>
>
>> // If there aren't any old versions, don't worry about garbage collection.
>>
>> if (!rpb->rpb_b_page) {
>> // Delete old versions fetching data for garbage collection.
>>
>> record_param temp = *rpb;
>> RecordStack empty_staying;
>> garbage_collect(tdbb, &temp, rpb->rpb_page, empty_staying);
>> }
>>
> > delete_record(tdbb, rpb, prior_page, 0);
Not as simple. Your code will leave record with broken backversion pointer
in case of
failure while garbage_collect is running. To avoid this, we need to zero
backpointer before
garbage_collect. It also have risk to create orphan backversions but it is not
a big problem.
Also, it require to fetch page with primary version second time and finally
delete that primary
version. It could create some performance penalty.
Regards,
Vlad
--
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Collection of garbage in record with uncommitted version
13.05.2014 14:25, Vlad Khorsun wrote: >>Actually, it is used in following cases: >> > >> >1) Undo with VIO_backout(). >> >2) Undo of update_in_place() in VIO_verb_cleanup(). >> >3) Applying third and following changes of the same record in one >> >transaction. > I know. I just a bit lazy when need to wrtite too much words, especially > in Engilish:) > In any case - all your points above is cases of undo operation. No. Number 3 is savepoint merge which is done on every operation. And I made a mistake, it happens on every call of update_in_place(). -- WBR, SD. -- "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] CORE-3515
13.05.2014 14:33, Vlad Khorsun wrote: > Your code will leave record with broken backversion pointer in case of > failure while garbage_collect is running. To avoid this, we need to zero > backpointer before > garbage_collect. It also have risk to create orphan backversions but it is > not a big problem. > Also, it require to fetch page with primary version second time and finally > delete that primary > version. It could create some performance penalty. You are tight, but this performance penalty won't be bigger than in VIO_backout() case with DPM_backout_mark() call. BTW, isn't call of DPM_backout_mark() there unnecessary in case when there is no backversions? -- WBR, SD. -- "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Collection of garbage in record with uncommitted version
On 5/13/2014 6:08 AM, Dimitry Sibiryakov wrote: > Hello, All. > > If a record has uncommitted head version created by active transaction > and some garbage > in backversions "tail", can anyone (background GC thread, sweep or parallel > transaction) > wipe this garbage from the tail while the transaction is still active? > I can't understand why list_staying() is so picky: it scans for every > next version from > the beginning which raise its complexity to O(N^2). > Index garbage collection is a very thorny problem. The original version predates threads in any operating system. Garbage collection occurred only when a record was visited. Garbage collection of an uncommitted record version was highly problematic since it was impossible to know whether the current version of the record would be going or staying, hence impossible in some cases to know whether an unreachable record version had an index key that could be garbage collected or not. We introduced threading in version 3. It was a nightmare. Virtually all of the operating systems that supported threading were buggy as hell (I wrote our own threading package for VMS, which didn't yet support threads). Sun's threading examples didn't even compile. A infrequent hang on Apollo was finally diagnosed as an "unfixable" design flaw when a signal arriving during a thread switch would be dropped on the floor, requiring that all signal based mechanisms in Interbase be rewritten to use threads. Revisiting garbage collection just wasn't in the cards. For most of its existence, Interbase and Firebird have only had one thread active in the engine. This kept synchronization of garbage collection and record update, which kept garbage collection simple and sane. Fine grain multi-threading, however, opens a major can of worms. My subsequent database systems do all garbage collection other than record backout in a dedicated garbage collect thread. Worker thread note when garbage collection is in order for a record and post it for the garbage collector to take up on next cycle. I don't know the current state of Firebird, but the idea of trying to synchronize cooperative (i.e. worker thread) and dedicated thread garbage collection is something I wouldn't want to take on. However, in direct answer to your question, let ask another: How to do garbage collect old record versions without visiting them? Any how can you even tell if there are collectable old versions without traversing the live versions first? There are probably lots of things that could be done to improve garbage collection. The one I think most fruitful is ability to garbage collect unreachable intermediate records between active versions at the front of a chain and older versions accessible only by very long running transactions so that very long running transaction don't impede all garbage collection. -- "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Feature request & discussion (Reply to Planning the post v3 development)
10.05.2014 14:03, Vlad Khorsun wrote: >As for A-B-A > updates i see no problem as there will be 3 separate index entries with own > tx numbers. Currently we also have 3 index entires for such scenario, btw. > Old IB's optimization to not store existing key second time is not working in > concurrent environment and easy allows to have missing entries. It was > disabled in FB 2.0. If I'm not mistaken, these repeating index entries works as a reference counters and allow to stop using staying list for indexes even in current code for the same reason as for blobs: three versions are enough for decision making. -- WBR, SD. -- "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Feature request & discussion (Reply to Planning the post v3 development)
On 5/13/2014 10:59 AM, Dimitry Sibiryakov wrote: > 10.05.2014 14:03, Vlad Khorsun wrote: >> As for A-B-A >> updates i see no problem as there will be 3 separate index entries with own >> tx numbers. Currently we also have 3 index entires for such scenario, btw. >> Old IB's optimization to not store existing key second time is not working in >> concurrent environment and easy allows to have missing entries. It was >> disabled in FB 2.0. > If I'm not mistaken, these repeating index entries works as a reference > counters and > allow to stop using staying list for indexes even in current code for the > same reason as > for blobs: three versions are enough for decision making. > No, it only makes race conditions more elusive... This area needs to be rethought, gentlemen (and Ann). -- "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Feature request & discussion (Reply to Planning the post v3 development)
13.05.2014 17:10, Jim Starkey wrote: > On 5/13/2014 10:59 AM, Dimitry Sibiryakov wrote: >> > If I'm not mistaken, these repeating index entries works as a >> > reference counters and >> >allow to stop using staying list for indexes even in current code for the >> >same reason as >> >for blobs: three versions are enough for decision making. >> > > No, it only makes race conditions more elusive... Can you provide a draft scenario when it can lead to a problem? Fortunately, we have a right guy for testing such things... :) -- WBR, SD. -- "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Feature request & discussion (Reply to Planning the post v3 development)
On 5/13/2014 11:24 AM, Dimitry Sibiryakov wrote: > 13.05.2014 17:10, Jim Starkey wrote: >> On 5/13/2014 10:59 AM, Dimitry Sibiryakov wrote: If I'm not mistaken, these repeating index entries works as a reference counters and allow to stop using staying list for indexes even in current code for the same reason as for blobs: three versions are enough for decision making. >> No, it only makes race conditions more elusive... > Can you provide a draft scenario when it can lead to a problem? > Fortunately, we have a right guy for testing such things... :) > Nope, can't prove the code doesn't catch all possible race conditions between two complex independent pieces of code. Nor can I prove that somebody in the future won't make an innocent change that fails once every couple of million times. What I've learned over many years is that race conditions are subtle and often lethal and are best architected away rather than coded around. Writing a lot of non-interlocked code using atomic instructions on SMP tends to give one a deep respect for the maliciousness of the computer gods. A one instruction window is really hard to find in testing... I'll concede that A-B-A index updates don't happen enough to loose a lot of sleep over, but it strikes me as inelegant to have two consecutive identical index entries. Much too fragile for my taste. For example, when cleaning up after a crash, does the code scan all record versions to avoiding taking out a single version when the code was expecting a double entry? -- "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Feature request & discussion (Reply to Planning the post v3 development)
13.05.2014 17:43, Jim Starkey wrote: >For example, > when cleaning up after a crash, does the code scan all record versions > to avoiding taking out a single version when the code was expecting a > double entry? Nice catch. Yes, if a version don't have all index entries inserting finished, cleanup can erase more than is healthy. -- WBR, SD. -- "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] [FB-Tracker] Created: (CORE-4426) Attachments remains in MON$ tables, owners >0 in fb_lock_print result after FB bugcheck + restart
Attachments remains in MON$ tables, owners >0 in fb_lock_print result after FB bugcheck + restart - Key: CORE-4426 URL: http://tracker.firebirdsql.org/browse/CORE-4426 Project: Firebird Core Issue Type: Bug Reporter: Pavel Zotov Attachments: attachments_after_FB_restart_with_bugcheck.zip Given: 1) LI-T3.0.0.31129, works as SuperServer; 2) test with ~120 attaches emulating OLTP activity, worked during ~9 hours; database name = 'oltp30.fdb' 3) isql which deliberately does script leading FB to crash (bugcheck): isql 192.168.0.220/:oltp30 SQL> recreate table t(id int primary key, x int, y int); commit; SQL> create index t_yx on t computed by( iif( ( select sum(y) from t tx where tx.x = t.x ) < 1000, id, -1 ) ); SQL> commit; SQL> insert into t values( 1, 100, 300); [Enter] here will fire bugcheck: oel64 Tue May 13 19:37:03 2014 Database: /var/db/fb30/oltp30.fdb unsuccessful metadata update request depth exceeded. (Recursive definition?) internal Firebird consistency check (error during savepoint backout (290), file: exe.cpp line: 1619) oel64 Tue May 13 19:37:26 2014 /opt/fb30trnk/bin/fbguard: /opt/fb30trnk/bin/firebird terminated abnormally (-1) After that I have stopped all attachments and tried to run validation, but got error "resource busy". So I check: lsof oltp30.fdb - and its output was: COMMANDPID USER FD TYPE DEVICE SIZE/OFFNODE NAME firebird 27282 firebird 32uW REG8,4 1456504832 6555457 oltp30.fdb Than looked in mon$-tables and found there NON-empty result for query with "WHERE MON$ATTACHMENT_ID <> CURRENT_CONNECTION" Then run fb_lock_print -a -c -d /var/db/fb30/oltp30.fdb and saw there 58 owners (and 121 free - these are from automatically recreated connections by batch script that run ISQLs). Why FB keeps these "phantom" connections ? PS. Please see attach - several stack traces of FB process, fb_lock_print and MON$-tables results. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://tracker.firebirdsql.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira -- "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] [FB-Tracker] Created: (CORE-4427) Coordinate index numbers between gfix and gstat
Coordinate index numbers between gfix and gstat --- Key: CORE-4427 URL: http://tracker.firebirdsql.org/browse/CORE-4427 Project: Firebird Core Issue Type: Improvement Components: GFIX, GSTAT Reporter: Dimitry Sibiryakov Priority: Trivial When gfix shows missing entries in indexes, it writes index numbers counting from one. gstat shows index statistics counting indexes from zero. It is confusing a little. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://tracker.firebirdsql.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira -- "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Feature request & discussion (Reply to Planning the post v3 development)
Jim Starkey skriver: > On 5/13/2014 11:24 AM, Dimitry Sibiryakov wrote: >> 13.05.2014 17:10, Jim Starkey wrote: >>> On 5/13/2014 10:59 AM, Dimitry Sibiryakov wrote: > If I'm not mistaken, these repeating index entries works as a > reference counters and > allow to stop using staying list for indexes even in current code for the > same reason as > for blobs: three versions are enough for decision making. > >>> No, it only makes race conditions more elusive... >> Can you provide a draft scenario when it can lead to a problem? >> Fortunately, we have a right guy for testing such things... :) >> > Nope, can't prove the code doesn't catch all possible race conditions > between two complex independent pieces of code. Nor can I prove that > somebody in the future won't make an innocent change that fails once > every couple of million times. Just in case it's relevant in this discussion and you don't already know: The people at IBSurgeon seem to know that FB suffers from some kind of race condition that can lead to database corruption when dropping a table while other users are connected. Happened to me a couple of months ago... Regards, Kjell -- "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
