[Firebird-devel] Firebird 3.0 on MacOSX
I need to post this on devel to canvass some opinions re. Firebird 3.x on MacOSX. As you may know in firebird 3.0 on posix now relies on a OS pre-installed version of ICU for both development and runtime, rather than what was our own build of ICU. Although this has its inherent problems (databases need to backed up and restored across different versions of ICU) it does make our use of ICU easier, that is, until we get to MacOSX. OSX only has a core ICU install by default, Firebird needs more. So the solution from a build perspective is to install MacPorts ICU and link to that, however that now creates additional external dependencies... Does that mean - in future - that a Firebird user on MaOSX needs to install ICU via MacPorts into opt/local/lib? Or should I try and ship the ICU libs with the Framework, along with any relevant dependent libs? The same is also true for the usage of libtommath... A find / libicu on my Mac seems to indicate that other software that uses ICU does ship ICU with with their frameworks. Comments? I am at the point where we need to make a decision as I am currently re-writing the MacOSX Firebird installer for Firebird 3.0 Regards Paul -- Slashdot TV. Video for Nerds. Stuff that Matters. http://pubads.g.doubleclick.net/gampad/clk?id=160591471iu=/4140/ostg.clktrk Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird 3.0 on MacOSX
MacPorts shall never touch my OS X Systems. On Sep 19, 2014, at 10:33 AM, Paul Beach wrote: I need to post this on devel to canvass some opinions re. Firebird 3.x on MacOSX. As you may know in firebird 3.0 on posix now relies on a OS pre-installed version of ICU for both development and runtime, rather than what was our own build of ICU. Although this has its inherent problems (databases need to backed up and restored across different versions of ICU) it does make our use of ICU easier, that is, until we get to MacOSX. OSX only has a core ICU install by default, Firebird needs more. So the solution from a build perspective is to install MacPorts ICU and link to that, however that now creates additional external dependencies... Does that mean - in future - that a Firebird user on MaOSX needs to install ICU via MacPorts into opt/local/lib? Or should I try and ship the ICU libs with the Framework, along with any relevant dependent libs? The same is also true for the usage of libtommath... A find / libicu on my Mac seems to indicate that other software that uses ICU does ship ICU with with their frameworks. Comments? I am at the point where we need to make a decision as I am currently re-writing the MacOSX Firebird installer for Firebird 3.0 Regards Paul -- Slashdot TV. Video for Nerds. Stuff that Matters. http://pubads.g.doubleclick.net/gampad/clk?id=160591471iu=/4140/ostg.clktrk Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Rescheduled: Maintenance/outage for tracker.firebirdsql.org and web.firebirdsql.org from Sept 20 08:00 to Sept 21 18:00 (EDT)
All, The previously mentioned ISP problems have been resolved, so the server room equipment move is happening tomorrow. Although the move should be largely completed on the 13th, there may be issues which could drag into the 14th. This will affect the project tracker.firebirdsql.org and web.firebirdsql.org systems. Sean -- Slashdot TV. Video for Nerds. Stuff that Matters. http://pubads.g.doubleclick.net/gampad/clk?id=160591471iu=/4140/ostg.clktrkFirebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Firebird 3 memory manager
Hello, All!
I implemented intermediate versions GC algorithm and tried to run some stress
tests on Firebird 3 to
check if I have broken something.
The problem is that tests that I created were spending most of their time in
memory manager. This is
not healthy.
For example, in my tests I saw up to 30 iterations in this loop during
memory allocation:
for (hunk = smallHunks; hunk; hunk = hunk-nextHunk)
{
if (length = hunk-spaceRemaining)
{
}
}
Dear Firebird engineers, why did you replace an algorithm which has O(log(n))
performance with an
algorithm that has O(n) performance in such a performance-critical part of the
engine?
O(n) performance in certain scenarios was the reason why original memory
manager of Interbase was
replaced in Firebird 1.5.
I created a small query to demonstrate the effect of O(n) memory manager
performance.
===
create table test7 (id integer);
execute block as
declare variable i integer = 1;
begin
while(i = 5000) do
begin
insert into test7(id) values(:i);
i = i + 1;
end
end;
commit;
savepoint X;
update test7 set id = id;
delete from test7;
===
Last statement uses 3.5 Gb of memory in small blocks. I forward-ported memory
manager from Firebird
2.5 to Firebird 3 and compared performance of the last statement.
With Firebird 3 memory manager the query runs for 634 s on my machine. With
Firebird 2.5 memory
manger it completes in 427 seconds.
I used very small number of rows, so you can run it on a desktop class machine.
It is easy to
imagine a query that is a couple of orders of magnitude larger in ETL
applications, and see that
such queries now become impossible to run (due to at least an order of
magnitude estimated slowdown).
So the performance of the server is now O(N^2) for queries using large amounts
of memory, and the
server hits the wall rather quickly.
From quick review of new memory manager code I can see the following problems:
1) O(n) performance in small hunk allocation
2) O(n) performance in large hunk deallocation
3) Worst case allocation cost in large hunk allocation algorithm is bounded,
but much worse than for
older memory manager
4) Lots of memory waste in various scenarios, which is a security issue with
mild risk.
5) Not all debug facilities have been preserved.
Problem 1 and 2 can be relatively inexpensively fixed with existing code base.
Are there volunteers?
Problems 3 and 4 require going back to global best-fit allocation strategy, and
thus re-design.
Alternatively, I may fix Firebird 2.5 memory manager to have better performance
in simple cases
without compromising worst-case performance.
This is rather easy. B-Tree can be replaced with specially designed array of
pointers (see FastMM
code, for example).
Dmitry, do you want me to do this?
My understanding is that Firebird project probably does not care about large
installs and security,
so it will continue using new memory manager as it is marginally faster in
simple cases. For Red
Database version based on Firebird 3 we shall certainly back out this memory
manager as it has
unpredictable performance and introduces new security risks.
Good thing is that changing or replacing memory manager is very simple task for
existing code base.
Best Regards,
Nikolay Samofatov
--
Slashdot TV. Video for Nerds. Stuff that Matters.
http://pubads.g.doubleclick.net/gampad/clk?id=160591471iu=/4140/ostg.clktrk
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird 3 memory manager
Nicolay,
I am wondering if you ever considered adding large sql identifiers and
schema to your implementation?
On 19 September 2014 18:33, Nikolay Samofatov
nikolay.samofa...@red-soft.biz wrote:
Hello, All!
I implemented intermediate versions GC algorithm and tried to run some
stress tests on Firebird 3 to
check if I have broken something.
The problem is that tests that I created were spending most of their time
in memory manager. This is
not healthy.
For example, in my tests I saw up to 30 iterations in this loop during
memory allocation:
for (hunk = smallHunks; hunk; hunk = hunk-nextHunk)
{
if (length = hunk-spaceRemaining)
{
}
}
Dear Firebird engineers, why did you replace an algorithm which has
O(log(n)) performance with an
algorithm that has O(n) performance in such a performance-critical part of
the engine?
O(n) performance in certain scenarios was the reason why original memory
manager of Interbase was
replaced in Firebird 1.5.
I created a small query to demonstrate the effect of O(n) memory manager
performance.
===
create table test7 (id integer);
execute block as
declare variable i integer = 1;
begin
while(i = 5000) do
begin
insert into test7(id) values(:i);
i = i + 1;
end
end;
commit;
savepoint X;
update test7 set id = id;
delete from test7;
===
Last statement uses 3.5 Gb of memory in small blocks. I forward-ported
memory manager from Firebird
2.5 to Firebird 3 and compared performance of the last statement.
With Firebird 3 memory manager the query runs for 634 s on my machine.
With Firebird 2.5 memory
manger it completes in 427 seconds.
I used very small number of rows, so you can run it on a desktop class
machine. It is easy to
imagine a query that is a couple of orders of magnitude larger in ETL
applications, and see that
such queries now become impossible to run (due to at least an order of
magnitude estimated slowdown).
So the performance of the server is now O(N^2) for queries using large
amounts of memory, and the
server hits the wall rather quickly.
From quick review of new memory manager code I can see the following
problems:
1) O(n) performance in small hunk allocation
2) O(n) performance in large hunk deallocation
3) Worst case allocation cost in large hunk allocation algorithm is
bounded, but much worse than for
older memory manager
4) Lots of memory waste in various scenarios, which is a security issue
with mild risk.
5) Not all debug facilities have been preserved.
Problem 1 and 2 can be relatively inexpensively fixed with existing code
base. Are there volunteers?
Problems 3 and 4 require going back to global best-fit allocation
strategy, and thus re-design.
Alternatively, I may fix Firebird 2.5 memory manager to have better
performance in simple cases
without compromising worst-case performance.
This is rather easy. B-Tree can be replaced with specially designed array
of pointers (see FastMM
code, for example).
Dmitry, do you want me to do this?
My understanding is that Firebird project probably does not care about
large installs and security,
so it will continue using new memory manager as it is marginally faster in
simple cases. For Red
Database version based on Firebird 3 we shall certainly back out this
memory manager as it has
unpredictable performance and introduces new security risks.
Good thing is that changing or replacing memory manager is very simple
task for existing code base.
Best Regards,
Nikolay Samofatov
--
Slashdot TV. Video for Nerds. Stuff that Matters.
http://pubads.g.doubleclick.net/gampad/clk?id=160591471iu=/4140/ostg.clktrk
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
--
Slashdot TV. Video for Nerds. Stuff that Matters.
http://pubads.g.doubleclick.net/gampad/clk?id=160591471iu=/4140/ostg.clktrkFirebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
