Nicolay,
I am wondering if you ever considered adding large sql identifiers and
schema to your implementation?
On 19 September 2014 18:33, Nikolay Samofatov <
nikolay.samofa...@red-soft.biz> wrote:
> Hello, All!
>
> I implemented intermediate versions GC algorithm and tried to run some
> stress tests on Firebird 3 to
> check if I have broken something.
> The problem is that tests that I created were spending most of their time
> in memory manager. This is
> not healthy.
>
> For example, in my tests I saw up to 300000 iterations in this loop during
> memory allocation:
>
> for (hunk = smallHunks; hunk; hunk = hunk->nextHunk)
> {
> if (length <= hunk->spaceRemaining)
> {
> ....
> }
> }
>
> Dear Firebird engineers, why did you replace an algorithm which has
> O(log(n)) performance with an
> algorithm that has O(n) performance in such a performance-critical part of
> the engine?
>
> O(n) performance in certain scenarios was the reason why original memory
> manager of Interbase was
> replaced in Firebird 1.5.
>
> I created a small query to demonstrate the effect of O(n) memory manager
> performance.
> ===
> create table test7 (id integer);
>
> execute block as
> declare variable i integer = 1;
> begin
> while(i <= 50000000) do
> begin
> insert into test7(id) values(:i);
> i = i + 1;
> end
> end;
>
> commit;
>
> savepoint X;
> update test7 set id = id;
> delete from test7;
> ===
>
> Last statement uses 3.5 Gb of memory in small blocks. I forward-ported
> memory manager from Firebird
> 2.5 to Firebird 3 and compared performance of the last statement.
> With Firebird 3 memory manager the query runs for 634 s on my machine.
> With Firebird 2.5 memory
> manger it completes in 427 seconds.
> I used very small number of rows, so you can run it on a desktop class
> machine. It is easy to
> imagine a query that is a couple of orders of magnitude larger in ETL
> applications, and see that
> such queries now become impossible to run (due to at least an order of
> magnitude estimated slowdown).
>
> So the performance of the server is now O(N^2) for queries using large
> amounts of memory, and the
> server hits the wall rather quickly.
>
> From quick review of new memory manager code I can see the following
> problems:
> 1) O(n) performance in small hunk allocation
> 2) O(n) performance in large hunk deallocation
> 3) Worst case allocation cost in large hunk allocation algorithm is
> bounded, but much worse than for
> older memory manager
> 4) Lots of memory waste in various scenarios, which is a security issue
> with mild risk.
> 5) Not all debug facilities have been preserved.
>
> Problem 1 and 2 can be relatively inexpensively fixed with existing code
> base. Are there volunteers?
> Problems 3 and 4 require going back to global best-fit allocation
> strategy, and thus re-design.
>
> Alternatively, I may fix Firebird 2.5 memory manager to have better
> performance in simple cases
> without compromising worst-case performance.
> This is rather easy. B-Tree can be replaced with specially designed array
> of pointers (see FastMM
> code, for example).
> Dmitry, do you want me to do this?
>
> My understanding is that Firebird project probably does not care about
> large installs and security,
> so it will continue using new memory manager as it is marginally faster in
> simple cases. For Red
> Database version based on Firebird 3 we shall certainly back out this
> memory manager as it has
> unpredictable performance and introduces new security risks.
> Good thing is that changing or replacing memory manager is very simple
> task for existing code base.
>
>
> Best Regards,
> Nikolay Samofatov
>
>
>
> ------------------------------------------------------------------------------
> Slashdot TV. Video for Nerds. Stuff that Matters.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=160591471&iu=/4140/ostg.clktrk
> Firebird-Devel mailing list, web interface at
> https://lists.sourceforge.net/lists/listinfo/firebird-devel
>
------------------------------------------------------------------------------
Slashdot TV. Video for Nerds. Stuff that Matters.
http://pubads.g.doubleclick.net/gampad/clk?id=160591471&iu=/4140/ostg.clktrk
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
