Re: [Firebird-devel] INT64 and index keys

2022-02-24 Thread Alex Peshkoff via Firebird-devel
On 2/15/22 18:20, Vlad Khorsun wrote: I'd vote to remove idx_numeric2 in favour of idx_decimal and look how to improve Decimal128::makeIndexKey() performance. For example, it stores every 3 decimal digits into 10 bits using relatively complex (computationally) algorithm with shifts and multipl

Re: [Firebird-devel] INT64 and index keys

2022-02-21 Thread Dmitry Yemanov
21.02.2022 20:05, Alex Peshkoff via Firebird-devel wrote: This is possible way to fix a bug: https://github.com/FirebirdSQL/firebird/commit/7dd832f32e9669bcb3007dc675b3ee7cca6f6b7d New type of indexes is added and it works fine. I didn't look at the patch deeply yet, so the question: what i

Re: [Firebird-devel] INT64 and index keys

2022-02-21 Thread Alex Peshkoff via Firebird-devel
On 2/16/22 13:49, Alex Peshkoff via Firebird-devel wrote: On 2/16/22 12:54, Dmitry Yemanov wrote: 16.02.2022 12:19, Mark Rotteveel wrote: However, that was not my main point. My main point was that it sounds like an index format that was created for supporting DECFLOAT(34), and that it is no

Re: [Firebird-devel] INT64 and index keys

2022-02-16 Thread Alex Peshkoff via Firebird-devel
On 2/16/22 12:54, Dmitry Yemanov wrote: 16.02.2022 12:19, Mark Rotteveel wrote: However, that was not my main point. My main point was that it sounds like an index format that was created for supporting DECFLOAT(34), and that it is not suitable for the full range of INT128 and NUMERIC/DECIMA

Re: [Firebird-devel] INT64 and index keys

2022-02-16 Thread Alex Peshkoff via Firebird-devel
On 2/16/22 13:34, Dmitry Yemanov wrote: 16.02.2022 13:28, Dmitry Yemanov wrote: It looks so. Unless we miss something (Alex?), perhaps we need to add a runtime check that rejects key creation for INT128 values longer than 34 decimal digits. Thinking twice, an overflow error should already b

Re: [Firebird-devel] INT64 and index keys

2022-02-16 Thread Dmitry Yemanov
16.02.2022 13:28, Dmitry Yemanov wrote: It looks so. Unless we miss something (Alex?), perhaps we need to add a runtime check that rejects key creation for INT128 values longer than 34 decimal digits. Thinking twice, an overflow error should already been raised when a longish INT128 is conv

Re: [Firebird-devel] INT64 and index keys

2022-02-16 Thread Alex Peshkoff via Firebird-devel
On 2/16/22 13:28, Dmitry Yemanov wrote: 16.02.2022 13:24, Alex Peshkoff via Firebird-devel wrote: It looks so. Unless we miss something (Alex?), perhaps we need to add a runtime check that rejects key creation for INT128 values longer than 34 decimal digits. Thinking twice, an overflow erro

Re: [Firebird-devel] INT64 and index keys

2022-02-16 Thread Dmitry Yemanov
16.02.2022 13:24, Alex Peshkoff via Firebird-devel wrote: It looks so. Unless we miss something (Alex?), perhaps we need to add a runtime check that rejects key creation for INT128 values longer than 34 decimal digits. Thinking twice, an overflow error should already been raised when a long

Re: [Firebird-devel] INT64 and index keys

2022-02-16 Thread Alex Peshkoff via Firebird-devel
On 2/16/22 12:59, Dmitry Yemanov wrote: 16.02.2022 12:54, Dmitry Yemanov wrote: It looks so. Unless we miss something (Alex?), perhaps we need to add a runtime check that rejects key creation for INT128 values longer than 34 decimal digits. Thinking twice, an overflow error should already b

Re: [Firebird-devel] INT64 and index keys

2022-02-16 Thread Alex Peshkoff via Firebird-devel
On 2/16/22 12:45, Alex Peshkoff via Firebird-devel wrote: To be precise - 9 digits into ULONG. No shifts (BTW what a problem with them, CPUs have appropriate fast support since first pentium or even more). There is one division of small integer (range from 0 to 33). It can be easily replaced w

Re: [Firebird-devel] INT64 and index keys

2022-02-16 Thread Vlad Khorsun
16.02.2022 11:45, Alex Peshkoff via Firebird-devel wrote: On 2/15/22 18:20, Vlad Khorsun wrote: I'd vote to remove idx_numeric2 in favour of idx_decimal Appears to be good idea. and look how to improve Decimal128::makeIndexKey() performance. For example, it stores every 3 decimal digits int

Re: [Firebird-devel] INT64 and index keys

2022-02-16 Thread Dmitry Yemanov
16.02.2022 12:54, Dmitry Yemanov wrote: It looks so. Unless we miss something (Alex?), perhaps we need to add a runtime check that rejects key creation for INT128 values longer than 34 decimal digits. Thinking twice, an overflow error should already been raised when a longish INT128 is conv

Re: [Firebird-devel] INT64 and index keys

2022-02-16 Thread Dmitry Yemanov
16.02.2022 12:19, Mark Rotteveel wrote: However, that was not my main point. My main point was that it sounds like an index format that was created for supporting DECFLOAT(34), and that it is not suitable for the full range of INT128 and NUMERIC/DECIMAL backed by INT128 (for the same reasons

Re: [Firebird-devel] INT64 and index keys

2022-02-16 Thread Alex Peshkoff via Firebird-devel
On 2/15/22 18:20, Vlad Khorsun wrote: I'd vote to remove idx_numeric2 in favour of idx_decimal Appears to be good idea. and look how to improve Decimal128::makeIndexKey() performance. For example, it stores every 3 decimal digits into 10 bits using relatively complex (computationally) algo

Re: [Firebird-devel] INT64 and index keys

2022-02-16 Thread Vlad Khorsun
16.02.2022 11:19, Mark Rotteveel wrote: On 2022-02-16 09:56, Vlad Khorsun wrote: 16.02.2022 10:35, Mark Rotteveel wrote: On 2022-02-15 20:08, Vlad Khorsun wrote: 15.02.2022 20:32, Mark Rotteveel wrote: On 2022-02-15 16:20, Vlad Khorsun wrote:   I'd vote to remove idx_numeric2 in favour of id

Re: [Firebird-devel] INT64 and index keys

2022-02-16 Thread Mark Rotteveel
On 2022-02-16 09:56, Vlad Khorsun wrote: 16.02.2022 10:35, Mark Rotteveel wrote: On 2022-02-15 20:08, Vlad Khorsun wrote: 15.02.2022 20:32, Mark Rotteveel wrote: On 2022-02-15 16:20, Vlad Khorsun wrote:   I'd vote to remove idx_numeric2 in favour of idx_decimal and look how to improve Decimal

Re: [Firebird-devel] INT64 and index keys

2022-02-16 Thread Vlad Khorsun
16.02.2022 10:35, Mark Rotteveel wrote: On 2022-02-15 20:08, Vlad Khorsun wrote: 15.02.2022 20:32, Mark Rotteveel wrote: On 2022-02-15 16:20, Vlad Khorsun wrote:   I'd vote to remove idx_numeric2 in favour of idx_decimal and look how to improve Decimal128::makeIndexKey() performance. For examp

Re: [Firebird-devel] INT64 and index keys

2022-02-16 Thread Mark Rotteveel
On 2022-02-15 20:08, Vlad Khorsun wrote: 15.02.2022 20:32, Mark Rotteveel wrote: On 2022-02-15 16:20, Vlad Khorsun wrote:   I'd vote to remove idx_numeric2 in favour of idx_decimal and look how to improve Decimal128::makeIndexKey() performance. For example, it stores every 3 decimal digits int

Re: [Firebird-devel] INT64 and index keys

2022-02-15 Thread Vlad Khorsun
15.02.2022 20:32, Mark Rotteveel wrote: On 2022-02-15 16:20, Vlad Khorsun wrote:   I'd vote to remove idx_numeric2 in favour of idx_decimal and look how to improve Decimal128::makeIndexKey() performance. For example, it stores every 3 decimal digits into 10 bits using relatively complex (computa

Re: [Firebird-devel] INT64 and index keys

2022-02-15 Thread Mark Rotteveel
On 2022-02-15 16:20, Vlad Khorsun wrote: I'd vote to remove idx_numeric2 in favour of idx_decimal and look how to improve Decimal128::makeIndexKey() performance. For example, it stores every 3 decimal digits into 10 bits using relatively complex (computationally) algorithm with shifts and mul

Re: [Firebird-devel] INT64 and index keys

2022-02-15 Thread Dimitry Sibiryakov
Dmitry Yemanov wrote 15.02.2022 18:14: Your schema upgrade may change keys from INT to BIGINT (when tables grow more than expected) or, more usually -- NUMERIC(N, 3) is changed to NUMERIC(N, 5) for quantities, or NUMERIC(12, N) is changed to NUMERIC(18, N), etc. And it should be applied to prod

Re: [Firebird-devel] INT64 and index keys

2022-02-15 Thread Dmitry Yemanov
15.02.2022 18:37, Kjell Rilbe wrote: With respect, since I'm not in the dev team, I fail to see it as an important feature to avoid index rebuild when changing between integer and numeric column types: 1. Changing between such types must be pretty rare, at least on production databases. Dur

Re: [Firebird-devel] INT64 and index keys

2022-02-15 Thread Kjell Rilbe
Den 2022-02-15 kl. 16:10, skrev Vlad Khorsun: 15.02.2022 15:20, Dmitry Yemanov wrote: ... I can think of two ways to proceed: 1) Keep rebuilding the index every time the type is changed. Cast all column values into the new format before storing keys into the index. Forget about being scale-

Re: [Firebird-devel] INT64 and index keys

2022-02-15 Thread Vlad Khorsun
15.02.2022 13:28, Dmitry Yemanov wrote: All, Historically, we store most numerics as double precision inside index keys. It makes stored keys independent of the declared scale and allows both prefix and suffix compression. However, int64 keys (BIGINT and largish NUMERICs/DECIMALs) do not fit th

Re: [Firebird-devel] INT64 and index keys

2022-02-15 Thread Vlad Khorsun
15.02.2022 15:20, Dmitry Yemanov wrote: ... I can think of two ways to proceed: 1) Keep rebuilding the index every time the type is changed. Cast all column values into the new format before storing keys into the index. Forget about being scale-independent, pick the scale from the latest tabl

Re: [Firebird-devel] INT64 and index keys

2022-02-15 Thread Dmitry Yemanov
15.02.2022 14:41, Dimitry Sibiryakov wrote: Is "stored keys independent of the declared scale" a really useful and used feature? That's a separate but also good question. Double precision keys allow independence of both precision (within its range) and scale. AFAIU, the idea was to allow ch

Re: [Firebird-devel] INT64 and index keys

2022-02-15 Thread Adriano dos Santos Fernandes
On 15/02/2022 08:28, Dmitry Yemanov wrote: > > Any other comments? > Would not it be possible to use some form of varint encoding (VLQ) there? Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] INT64 and index keys

2022-02-15 Thread Dimitry Sibiryakov
Dmitry Yemanov wrote 15.02.2022 12:28: Does anyone think we should investigate this possibility more closely? Any other comments? Is "stored keys independent of the declared scale" a really useful and used feature? If we prohibit that, the index key can be a plain integer without rounding