Re: [Firebird-devel] Record storage
My database systems haven't have pages for 15 years, based instead on distributed objects serialized to storage for persistence. Netfrastructure/Falcon did all record operations in memory using database pages only for backfill. OK, I guess, but it didn't work for distributed databases. If you start with a distributed database rather than a single node database, you come up with entirely different architectures. The problem, however, is how does an existing product change architectures? Beats me. On 6/9/2022 9:18 AM, Adriano dos Santos Fernandes wrote: On 09/06/2022 09:58, Dmitry Yemanov wrote: 09.06.2022 15:16, Adriano dos Santos Fernandes wrote: Yes, it should work. However, I'm not going to remove the limit until we introduce a denser compression. Also, we have a number of places where records is stored unpacked in memory (rpb's, RecordBuffer, HashJoin, etc), so longer records could increase server memory usage. This should be improved somehow. Yes, but it's so bad when one needs to improve their schema and hit a limit. And sometimes the arbitrary limit is hit by few margin, like switching a field encoding or small necessary increase in length. The same with record formats limited to 255. It's so awful, and it's related stuff, as the format could also be variable encoded to not always use 2 bytes. The problem, however, is that format-aware processing was found to be slower. The dumb scheme presented above (with no real compression) provided almost the same record size as RLE compression for mixed "real-world" fields and was even denser for records with longish UTF8 fields, but it was also ~20% slower. If the storage takes less space, is this slow down estimation calculated also taking into account the slower number of pages read (when page is not cached)? Surely when page is cached that becomes more problematic. I think Jim's databases had record caches instead of only page caches, probably to avoid this type of problem. Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Jim Starkey, AmorphousDB, LLC Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Record storage
Putting metadata bytes in record data is INCREDIBLY efficient. The heavily overload byte codes that I have used for my last two database systems has, for example, byte codes for: 1. Integers from -10 to 32 (totally arbitrary) 2. Integers of significant length 1 to 8 3. UTF strings from length of 0 to 32 bytes (also arbitrary) 4. UTF strings with encoded lengths from 1 to 4 bytes. Run length encoding works very well for squeezing out blanks in char fields, De Witt's stupid benchmark, and very else little. No rational person would ever using fixed length character string for De Witt's benchmarks. The overloaded byte code scheme was not my idea but a suggestion from very, very Vulcan follower. Unfortunately, that was too may companies back and I no longer have his name. It was a superb suggestion. I'll drop in the codes that I use for Amorphous. National character sets belong in client side conversions and, indirectly, in collations. We're in an era where national boundaries have no part of data (or, unfortunately, armies). On 6/9/2022 8:36 AM, Adriano dos Santos Fernandes wrote: On 09/06/2022 09:29, Dimitry Sibiryakov wrote: Adriano dos Santos Fernandes wrote 09.06.2022 14:16: What do you think and are there any active work in this regard? Using of record encoding instead of record compressing was suggested years ago by Ann and Jim. Yes, but, suggesting things and doing no action during more than 10 years does not make users happy. Self-descriptive record format which makes RDB$FORMATS obsolete and solve problems with garbage collection etc was suggested by me (for replication block buffer in the first place but storage can use it as well). Putting metadata bytes in each record is not efficient. Adriano Firebird-Devel mailing list, web interface athttps://lists.sourceforge.net/lists/listinfo/firebird-devel -- Jim Starkey, AmorphousDB, LLC// Copyright (c) 2014 by James A. Starkey. All rights reserved. #pragma once #include "Stream.h" #include "StringClass.h" #include "UUId.h" #include "Date.h" #include "Timestamp.h" class Opaque; static const int typeNull = 1; static const int typeEnd= 2; static const int typeBoolean= 3; static const int typeClassId= 4; static const int typeAttributeId= 5; static const int typeObjectId = 6; static const int typeList = 7; static const int typeInt= 8; static const int typeLong = 9; static const int typeDouble = 10; static const int typeDate = 11; static const int typeString = 12; static const int typeOpaque = 13; static const int typeUUId = 14; static const int typeTimestamp = 15; static const int typeScaledInt = 16; static const int codeNull = 1; static const int codeEnd= 2; static const int codeBoolean0 = 3; static const int codeBoolean1 = 4; static const int codeClass0 = 5; static const int codeClass1 = 6; static const int codeClass2 = 7; static const int codeClass3 = 8; static const int codeClass4 = 9; static const int codeClass5 = 10; static const int codeClass6 = 11; static const int codeClass7 = 12; static const int codeClass8 = 13; static const int codeClass9 = 14; static const int codeClass10= 15; static const int codeClass11= 16; static const int codeClass12= 17; static const int codeClass13= 18; static const int codeClass14= 19; static const int codeClass15= 20; static const int codeClass16= 21; static const int codeClassCount1= 22; static const int codeClassCount2= 23; static const int codeClassCount3= 24; static const int codeClassCount4= 25; static const int codeIntMinus10 = 26; static const int codeIntMinus9 = 27; static const int codeIntMinus8 = 28; static const int codeIntMinus7 = 29; static const int codeIntMinus6 = 30; static const int codeIntMinus5 = 31; static const int codeIntMinus4 = 32; static const int codeIntMinus3 = 33; static const int codeIntMinus2 = 34; static const int codeIntMinus1 = 35; static const int codeInt0 = 36; static const int codeInt1 = 37; static const int codeInt2 = 38; static const int codeInt3 = 39; static const int codeInt4 = 40; static const int codeInt5 = 41; static const int codeInt6 = 42; static const
Re: [Firebird-devel] Record storage
09.06.2022 16:18, Adriano dos Santos Fernandes wrote: 09.06.2022 15:16, Adriano dos Santos Fernandes wrote: Yes, it should work. However, I'm not going to remove the limit until we introduce a denser compression. Also, we have a number of places where records is stored unpacked in memory (rpb's, RecordBuffer, HashJoin, etc), so longer records could increase server memory usage. This should be improved somehow. Yes, but it's so bad when one needs to improve their schema and hit a limit. And sometimes the arbitrary limit is hit by few margin, like switching a field encoding or small necessary increase in length. Agreed. In the meantime, I will not object to keeping the limit but raising it to e.g. 256KB. The same with record formats limited to 255. It's so awful, and it's related stuff, as the format could also be variable encoded to not always use 2 bytes. True. Another approach is also possible: (optionally) extend sweeping to upgrade the record format of the committed records on the data pages being swept, garbage collect unused formats and re-use them when the counter wraps around. The problem, however, is that format-aware processing was found to be slower. The dumb scheme presented above (with no real compression) provided almost the same record size as RLE compression for mixed "real-world" fields and was even denser for records with longish UTF8 fields, but it was also ~20% slower. If the storage takes less space, is this slow down estimation calculated also taking into account the slower number of pages read (when page is not cached)? Surely reading from disk is way slower than decompressing in memory, so less data pages to read easily outbids the increased decompression penalty. Things are not so cool when the working set is cached though. Dmitry Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Record storage
09.06.2022 15:58, Dmitry Yemanov wrote: (2) skip padding bytes A separate but interesting question is whether we need alignment (and thus padding bytes) at all. Most of our installations are little-endian and it does not crash on unaligned data access. Moreover, modern CPUs access aligned/unaligned data with the same speed, AFAIK. Given that our ODS is different between LE and BE platforms anyway, perhaps it makes sense to optimize the record storage for the LE case. Dmitry Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Record storage
On 09/06/2022 09:58, Dmitry Yemanov wrote: > 09.06.2022 15:16, Adriano dos Santos Fernandes wrote: > > Yes, it should work. However, I'm not going to remove the limit until we > introduce a denser compression. Also, we have a number of places where > records is stored unpacked in memory (rpb's, RecordBuffer, HashJoin, > etc), so longer records could increase server memory usage. This should > be improved somehow. > Yes, but it's so bad when one needs to improve their schema and hit a limit. And sometimes the arbitrary limit is hit by few margin, like switching a field encoding or small necessary increase in length. The same with record formats limited to 255. It's so awful, and it's related stuff, as the format could also be variable encoded to not always use 2 bytes. > The problem, however, is that format-aware processing was found to be > slower. The dumb scheme presented above (with no real compression) > provided almost the same record size as RLE compression for mixed > "real-world" fields and was even denser for records with longish UTF8 > fields, but it was also ~20% slower. If the storage takes less space, is this slow down estimation calculated also taking into account the slower number of pages read (when page is not cached)? Surely when page is cached that becomes more problematic. I think Jim's databases had record caches instead of only page caches, probably to avoid this type of problem. Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Record storage
09.06.2022 15:16, Adriano dos Santos Fernandes wrote: With some frequency people ask me why UTF-8 is slower than single byte charsets. The thing is, they have something using, for example, VARCHAR(30) CHARACTER SET WIN1252 and convert to VARCHAR(30) CHARACTER SET UTF8, test with the same data and have slower queries. Database is also increased in size and record size (based on characters) limit is decreased. But if they test VARCHAR(120) CHARACTER SET WIN1252 vs VARCHAR(30) CHARACTER SET UTF8, database size and query times are similar. But this is just a test, it's not real world scenario user wants. We have old problems, for example, record size limit is tracked here: https://github.com/FirebirdSQL/firebird/issues/1130 Like commented there, I tried to just increase the constant and it seems to just work. Yes, it should work. However, I'm not going to remove the limit until we introduce a denser compression. Also, we have a number of places where records is stored unpacked in memory (rpb's, RecordBuffer, HashJoin, etc), so longer records could increase server memory usage. This should be improved somehow. Then we have the RLE record compression algorithm, that "compress" bytes that is well known to be unused. We had even patches to improve the bad algorithm. Yep. I believe that is not the way to go. So do I, although an improved RLE could be a good workaround until something significantly better is invented. Let's still call it "record compression", I believe it should be more active. Instead of work based only on the record buffer and its length, it should have access to the record format. Then it can encode things in more active way, trimming out unused bytes of CHAR/VARCHAR, better encoding numbers and booleans. We may use protocol-buffers format as inspiration. And then probably we don't need any RLE compression as most of data (not unused bytes) are not so repetitive. I tried something like that in the past, but I called it "packing" as opposed to "compression". The idea was to (1) skip NULL fields as they're already marked in a leading bit mask, (2) skip padding bytes because they can be reconstructed using a record format, (3) copy only meaningful bytes of VARCHAR strings (using its vary_length which is also stored). The rest (numerics/dates/CHARs) was copied "as is" (without compression). Of course, CHARs and the real part of VARCHARs could be compressed one way or another, but I intentionally left it for another day. The problem, however, is that format-aware processing was found to be slower. The dumb scheme presented above (with no real compression) provided almost the same record size as RLE compression for mixed "real-world" fields and was even denser for records with longish UTF8 fields, but it was also ~20% slower. Every field processed/copied separately is slower than processing the record as a whole. It can be proved even for current RLE - the more "runs" (compressed/uncompressed) we have there, the slower is decompression. I know that RedSoft tried to implement a mixed compression where RLE was used together with format-aware logic which decided what should become a compressible run. I don't recall the performance figures though, maybe Roman could share them. What do you think and are there any active work in this regard? Right now (for ODS 13.1) I'm working on improvement to the current RSE that does two things: (1) reduces number of runs (avoid short compressible runs) and (2) allow longish compressible runs with 2-byte (and possibly 3/4-byte) length. That should solve the problem with UTF8 strings without any performance penalty. For the next ODS, I was going to continue researches regarding "packing" and mixed "packing/compression" approaches to address the performance issues. Nothing is done yet. I'd also like to consider a completely different record storage format: where all VARCHARs are stored as in the prefix part and their contents is stored in the suffix part. This makes records variable-length and supposedly many code must be changed for that. However, it reduces memory usage for records (only real length is stored) and it allows flexible encoding: "as is" copying or some clever packing for the fixed prefix and e.g. LZ4 compression for the variable suffix. Dmitry Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Record storage
Adriano dos Santos Fernandes wrote 09.06.2022 14:36: Self-descriptive record format which makes RDB$FORMATS obsolete and solve problems with garbage collection etc was suggested by me (for replication block buffer in the first place but storage can use it as well). Putting metadata bytes in each record is not efficient. It is not worse than current compressed/plain bytes counters. -- WBR, SD. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Record storage
On 09/06/2022 09:29, Dimitry Sibiryakov wrote: > Adriano dos Santos Fernandes wrote 09.06.2022 14:16: >> What do you think and are there any active work in this regard? > > Using of record encoding instead of record compressing was suggested > years ago by Ann and Jim. Yes, but, suggesting things and doing no action during more than 10 years does not make users happy. > Self-descriptive record format which makes RDB$FORMATS obsolete and > solve problems with garbage collection etc was suggested by me (for > replication block buffer in the first place but storage can use it as > well). > Putting metadata bytes in each record is not efficient. Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Record storage
Adriano dos Santos Fernandes wrote 09.06.2022 14:16: What do you think and are there any active work in this regard? Using of record encoding instead of record compressing was suggested years ago by Ann and Jim. Self-descriptive record format which makes RDB$FORMATS obsolete and solve problems with garbage collection etc was suggested by me (for replication block buffer in the first place but storage can use it as well). -- WBR, SD. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Record storage
Hi! With some frequency people ask me why UTF-8 is slower than single byte charsets. The thing is, they have something using, for example, VARCHAR(30) CHARACTER SET WIN1252 and convert to VARCHAR(30) CHARACTER SET UTF8, test with the same data and have slower queries. Database is also increased in size and record size (based on characters) limit is decreased. But if they test VARCHAR(120) CHARACTER SET WIN1252 vs VARCHAR(30) CHARACTER SET UTF8, database size and query times are similar. But this is just a test, it's not real world scenario user wants. We have old problems, for example, record size limit is tracked here: https://github.com/FirebirdSQL/firebird/issues/1130 Like commented there, I tried to just increase the constant and it seems to just work. Then we have the RLE record compression algorithm, that "compress" bytes that is well known to be unused. We had even patches to improve the bad algorithm. I believe that is not the way to go. Let's still call it "record compression", I believe it should be more active. Instead of work based only on the record buffer and its length, it should have access to the record format. Then it can encode things in more active way, trimming out unused bytes of CHAR/VARCHAR, better encoding numbers and booleans. We may use protocol-buffers format as inspiration. And then probably we don't need any RLE compression as most of data (not unused bytes) are not so repetitive. What do you think and are there any active work in this regard? Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
