Re: [firebird-support] Re: Issue with Database Cache Size on FB 3.0

2016-06-02 Thread hv...@users.sourceforge.net [firebird-support] 
> ---In firebird-support@yahoogroups.com,  wrote :
 > > Something funny is going on regarding the memory consumption, I have 
 > > attached screenshots showing the memory counters. 
> If you look at the process itself (file 
> TaskManager_Process_Firebird_Only_Consumed_2GB.jpg), it only consumed 2 GB. 
> If you look at Windows Task Manager (file 
> TaskManager_Performance_ALL_RAM_Consumed.jpg), where it says "Physical 
> Memory, 
> just under the memory graph where it shows 3.13 GB used, the cached memory is 
> 29963 Mb. I know Firebird is who consumed that 
> CACHED memory because I am forcing the DB to read 70GB on purpouse, but why 
> is the memory not showing under the process 
> Firebird ? (it only shows 2GB used there).

  File system cache maintained by OS, not by Firebird (or any other app). It is 
shared between all processes and it is impossible to
assign any part of it to the some process. 

> My aim is to set the config to only use RAM (no file system cache), and to 
> place the hole DB in RAM for caching when reading, 

  Use RamMap utility (by SysInternals) to see which files is cached by OS and 
how much.

> I am struggling to find a config mix that will do that. The config variables 
> are:
 > 
> FileSystemCacheThreshold = 0
> FileSystemCacheSize = 80
> TempCacheLimit = 24576M (I am not sure what this value is for, it seems to 
> mean the maximum amount of temporary things to have in memory, like DB read 
> cache)

  No, it is used mainly for sorting and not for the page cache.

> The above 3 variables are the ones I am playing with, plus the 
> DefaultDBCachePages = 9 in Databases.conf
 
  This is the only setting for cache size. If your db have page size of 8KB it 
will consume ~703MB of memory, if page size is 
16KB, it will take near 1.4 GB.

  

 
 Hope it helps,
Vlad

Re: [firebird-support] Re: Issue with Database Cache Size on FB 3.0

2016-06-01 Thread fabia...@itbizolutions.com.au [firebird-support] 
Hi Robert

Here is the link to both files as the yahoo group may not allow me to attach 
the files
http://document.nucleuscrm.com.au:7770/pdfs/TMPFiles/TaskManager_Performance_ALL_RAM_Consumed.jpg
http://document.nucleuscrm.com.au:7770/pdfs/TMPFiles/TaskManager_Process_Firebird_Only_Consumed_2GB.jpg

The firebird is definetly 64 bits, however the ODBC that access the server is 
32 bits, but the client should have no effect on the way the server handles the 
DB (in theory).

Cheers,
Fabian

  - Original Message - 
  From: Robert martin r...@chreos.com [firebird-support] 
  To: firebird-support@yahoogroups.com 
  Sent: Thursday, June 02, 2016 9:22 AM
  Subject: Re: [firebird-support] Re: Issue with Database Cache Size on FB 3.0




  Hi Fabian


  Can't see your screen shot but FB usage at 2GB sounds like the limit for a 
32Bit app.  Have you installed the 32Bit version of FB accidentally?

  Thanks
  Rob

  On 2/06/2016 11:12 a.m., fabia...@itbizolutions.com.au [firebird-support] 
wrote:

  
Dmitry

Something funny is going on regarding the memory consumption, I have 
attached screenshots showing the memory counters. If you look at the process 
itself (file TaskManager_Process_Firebird_Only_Consumed_2GB.jpg), it only 
consumed 2 GB. If you look at Windows Task Manager (file 
TaskManager_Performance_ALL_RAM_Consumed.jpg), where it says "Physical Memory, 
just under the memory graph where it shows 3.13 GB used, the cached memory is 
29963 Mb. I know Firebird is who consumed that CACHED memory because I am 
forcing the DB to read 70GB on purpouse, but why is the memory not showing 
under the process Firebird ? (it only shows 2GB used there).

My aim is to set the config to only use RAM (no file system cache), and to 
place the hole DB in RAM for caching when reading, I am struggling to find a 
config mix that will do that. The config variables are:

FileSystemCacheThreshold = 0
FileSystemCacheSize = 80
TempCacheLimit = 24576M (I am not sure what this value is for, it seems to 
mean the maximum amount of temporary things to have in memory, like DB read 
cache)

The above 3 variables are the ones I am playing with, plus the 
DefaultDBCachePages = 9 in Databases.conf

Cheers,
Fabian

- Original Message - 
From: Dmitry Yemanov dim...@users.sourceforge.net [firebird-support] 
To: firebird-support@yahoogroups.com 
Sent: Wednesday, June 01, 2016 3:49 PM
Subject: [firebird-support] Re: Issue with Database Cache Size on FB 3.0

01.06.2016 04:16, Fabian Ernesto Chocron wrote:
>
> We are having trouble setting up the database cache size on FB 3.0
> running on Windows 2008 R2 64 bits with 32 GB ram.
>
> The problem we have is we cannot get the server to allocate the ram for
> the cache as we intend. With FB 2.54 we had the DB cache set very high,
> close to 1 GB per database, all running in RAM memory. With FB 3.0 we
> read it can allocate much more RAM to the cache, but it appears the
> server is allocating very small amount of Ram when the first user
> connects to the DB, and as we connect more users to the DB the ram
> consumption increases slowly.

What memory counters do you look at? Virtual (reserved) memory usage 
should grow high with the first connect, physical (committed) memory 
usage should grow together with your load, while pages are being read 
from disk.

> On firebird.conf
>
> FileSystemCacheThreshold = 0
> FileSystemCacheSize = 17179869184 (this is 16 GB - the server has 32 GB
> ram.)

These changes conflict with each other. You disable filesystem cache 
with the first line and still want to use 16GB with the second line.

> On databases.conf
>
> MyTestDB = c:\Temp\MyDb.fdb
> {
> DefaultDbCachePages = 458752
> }

First of all, I'd suggest to reset FileSystemCacheSize back to zero and 
set FileSystemCacheThreshold to something higher than 458752 pages (e.g. 
50). Then re-test your memory usage and only then start tweaking the 
filesystem cache.

BTW, don't you occasionally have page cache override at the database 
level? What is gstat -h output?

Dmitry





++

Visit http://www.firebirdsql.org and click the Documentation item
on the main (top) menu. Try FAQ and other links from the left-side menu 
there.

Also search the knowledgebases at 
http://www.ibphoenix.com/resources/documents/ 

++


Yahoo Groups Links

[Non-text portions of this message have been removed]



No virus found in this message.
Checked by AVG - www.avg.com

Re: [firebird-support] Re: Issue with Database Cache Size on FB 3.0

2016-06-01 Thread Robert martin r...@chreos.com [firebird-support] 

Hi Fabian

Can't see your screen shot but FB usage at 2GB sounds like the limit for 
a 32Bit app.  Have you installed the 32Bit version of FB accidentally?


Thanks
Rob

On 2/06/2016 11:12 a.m., fabia...@itbizolutions.com.au 
[firebird-support] wrote:


Dmitry

Something funny is going on regarding the memory consumption, I have 
attached screenshots showing the memory counters. If you look at the 
process itself (file 
TaskManager_Process_Firebird_Only_Consumed_2GB.jpg), it only consumed 
2 GB. If you look at Windows Task Manager (file 
TaskManager_Performance_ALL_RAM_Consumed.jpg), where it says "Physical 
Memory, just under the memory graph where it shows 3.13 GB used, the 
cached memory is 29963 Mb. I know Firebird is who consumed that CACHED 
memory because I am forcing the DB to read 70GB on purpouse, but why 
is the memory not showing under the process Firebird ? (it only shows 
2GB used there).


My aim is to set the config to only use RAM (no file system cache), 
and to place the hole DB in RAM for caching when reading, I am 
struggling to find a config mix that will do that. The config 
variables are:


FileSystemCacheThreshold = 0
FileSystemCacheSize = 80
TempCacheLimit = 24576M (I am not sure what this value is for, it 
seems to mean the maximum amount of temporary things to have in 
memory, like DB read cache)


The above 3 variables are the ones I am playing with, plus the 
DefaultDBCachePages = 9 in Databases.conf


Cheers,
Fabian

- Original Message -
From: Dmitry Yemanov dim...@users.sourceforge.net [firebird-support]
To: firebird-support@yahoogroups.com
Sent: Wednesday, June 01, 2016 3:49 PM
Subject: [firebird-support] Re: Issue with Database Cache Size on FB 3.0

01.06.2016 04:16, Fabian Ernesto Chocron wrote:
>
> We are having trouble setting up the database cache size on FB 3.0
> running on Windows 2008 R2 64 bits with 32 GB ram.
>
> The problem we have is we cannot get the server to allocate the ram for
> the cache as we intend. With FB 2.54 we had the DB cache set very high,
> close to 1 GB per database, all running in RAM memory. With FB 3.0 we
> read it can allocate much more RAM to the cache, but it appears the
> server is allocating very small amount of Ram when the first user
> connects to the DB, and as we connect more users to the DB the ram
> consumption increases slowly.

What memory counters do you look at? Virtual (reserved) memory usage
should grow high with the first connect, physical (committed) memory
usage should grow together with your load, while pages are being read
from disk.

> On firebird.conf
>
> FileSystemCacheThreshold = 0
> FileSystemCacheSize = 17179869184 (this is 16 GB - the server has 32 GB
> ram.)

These changes conflict with each other. You disable filesystem cache
with the first line and still want to use 16GB with the second line.

> On databases.conf
>
> MyTestDB = c:\Temp\MyDb.fdb
> {
> DefaultDbCachePages = 458752
> }

First of all, I'd suggest to reset FileSystemCacheSize back to zero and
set FileSystemCacheThreshold to something higher than 458752 pages (e.g.
50). Then re-test your memory usage and only then start tweaking the
filesystem cache.

BTW, don't you occasionally have page cache override at the database
level? What is gstat -h output?

Dmitry





++

Visit http://www.firebirdsql.org and click the Documentation item
on the main (top) menu. Try FAQ and other links from the left-side 
menu there.


Also search the knowledgebases at 
http://www.ibphoenix.com/resources/documents/


++


Yahoo Groups Links

[Non-text portions of this message have been removed]



No virus found in this message.
Checked by AVG - www.avg.com <http://www.avg.com>
Version: 2016.0.7598 / Virus Database: 4591/12342 - Release Date: 06/01/16

Re: [firebird-support] Re: Issue with Database Cache Size on FB 3.0

2016-06-01 Thread fabia...@itbizolutions.com.au [firebird-support] 
Dmitry

Something funny is going on regarding the memory consumption, I have attached 
screenshots showing the memory counters. If you look at the process itself 
(file TaskManager_Process_Firebird_Only_Consumed_2GB.jpg), it only consumed 2 
GB. If you look at Windows Task Manager (file 
TaskManager_Performance_ALL_RAM_Consumed.jpg), where it says "Physical Memory, 
just under the memory graph where it shows 3.13 GB used, the cached memory is 
29963 Mb. I know Firebird is who consumed that CACHED memory because I am 
forcing the DB to read 70GB on purpouse, but why is the memory not showing 
under the process Firebird ? (it only shows 2GB used there).

My aim is to set the config to only use RAM (no file system cache), and to 
place the hole DB in RAM for caching when reading, I am struggling to find a 
config mix that will do that. The config variables are:

FileSystemCacheThreshold = 0
FileSystemCacheSize = 80
TempCacheLimit = 24576M  (I am not sure what this value is for, it seems to 
mean the maximum amount of temporary things to have in memory, like DB read 
cache)

The above 3 variables are the ones I am playing with, plus the 
DefaultDBCachePages = 9 in Databases.conf

Cheers,
Fabian


  - Original Message - 
  From: Dmitry Yemanov dim...@users.sourceforge.net [firebird-support] 
  To: firebird-support@yahoogroups.com 
  Sent: Wednesday, June 01, 2016 3:49 PM
  Subject: [firebird-support] Re: Issue with Database Cache Size on FB 3.0


  01.06.2016 04:16, Fabian Ernesto Chocron wrote:
  >
  > We are having trouble setting up the database cache size on FB 3.0
  > running on Windows 2008 R2 64 bits with 32 GB ram.
  >
  > The problem we have is we cannot get the server to allocate the ram for
  > the cache as we intend. With FB 2.54 we had the DB cache set very high,
  > close to 1 GB per database, all running in RAM memory. With FB 3.0 we
  > read it can allocate much more RAM to the cache, but it appears the
  > server is allocating very small amount of Ram when the first user
  > connects to the DB, and as we connect more users to the DB the ram
  > consumption increases slowly.

  What memory counters do you look at? Virtual (reserved) memory usage 
  should grow high with the first connect, physical (committed) memory 
  usage should grow together with your load, while pages are being read 
  from disk.

  > On firebird.conf
  >
  > FileSystemCacheThreshold = 0
  > FileSystemCacheSize = 17179869184 (this is 16 GB - the server has 32 GB
  > ram.)

  These changes conflict with each other. You disable filesystem cache 
  with the first line and still want to use 16GB with the second line.

  > On databases.conf
  >
  > MyTestDB = c:\Temp\MyDb.fdb
  > {
  > DefaultDbCachePages = 458752
  > }

  First of all, I'd suggest to reset FileSystemCacheSize back to zero and 
  set FileSystemCacheThreshold to something higher than 458752 pages (e.g. 
  50). Then re-test your memory usage and only then start tweaking the 
  filesystem cache.

  BTW, don't you occasionally have page cache override at the database 
  level? What is gstat -h output?


  Dmitry




  

  

  ++

  Visit http://www.firebirdsql.org and click the Documentation item
  on the main (top) menu.  Try FAQ and other links from the left-side menu 
there.

  Also search the knowledgebases at 
http://www.ibphoenix.com/resources/documents/ 

  ++
  

  Yahoo Groups Links




[Non-text portions of this message have been removed]

Re: [firebird-support] Re: Issue with Database Cache Size on FB 3.0

2016-06-01 Thread fabia...@itbizolutions.com.au [firebird-support] 
Hi Dmitry

I am progressing with solving this one, the first issue was related with a 
config variable reserving most of the ram upfront, hence there was no ram 
available for the DB cache. The second issue was misinterpreting what the 
following meant:

# File system cache size
# Note that the lowest number presently supported is 10%, and the highest number
# is 95%; numbers outside these limits will be set to the default of 30%.

First interpretation on the above was the variable must be between 0 and 1, as 
50% in math is usually written as 0.5, but then the following made it confusing:
# Type: integer, measured in % of total physical RAM

Where it says "integer" it complicated things, as it would not make sense to 
have a percentage forced to be integer, as there would be no way to represent 
40.5% for instance. So I assume the value is an integer because it is 
representing bytes, and bytes cannot have decimals.

Anyways, as I play along with different configurations I am finding the FB 3 
server VERY responsive interms of multiusers, I am glad the SMP efforts are 
showing very interesting performance, so congrats to all the development 
team!!

As a test I am using IBexpert to compare the data between the FB 2.54 DB and 
the 3.0, it is a 80 GB database, and at the same time I am loading users on the 
FB 3 database, and the performance is almost as good as running without any 
other parallel process, so I am very impressed.


Cheers
Fabian




  - Original Message - 
  From: Dmitry Yemanov dim...@users.sourceforge.net [firebird-support] 
  To: firebird-support@yahoogroups.com 
  Sent: Wednesday, June 01, 2016 3:49 PM
  Subject: [firebird-support] Re: Issue with Database Cache Size on FB 3.0


  01.06.2016 04:16, Fabian Ernesto Chocron wrote:
  >
  > We are having trouble setting up the database cache size on FB 3.0
  > running on Windows 2008 R2 64 bits with 32 GB ram.
  >
  > The problem we have is we cannot get the server to allocate the ram for
  > the cache as we intend. With FB 2.54 we had the DB cache set very high,
  > close to 1 GB per database, all running in RAM memory. With FB 3.0 we
  > read it can allocate much more RAM to the cache, but it appears the
  > server is allocating very small amount of Ram when the first user
  > connects to the DB, and as we connect more users to the DB the ram
  > consumption increases slowly.

  What memory counters do you look at? Virtual (reserved) memory usage 
  should grow high with the first connect, physical (committed) memory 
  usage should grow together with your load, while pages are being read 
  from disk.

  > On firebird.conf
  >
  > FileSystemCacheThreshold = 0
  > FileSystemCacheSize = 17179869184 (this is 16 GB - the server has 32 GB
  > ram.)

  These changes conflict with each other. You disable filesystem cache 
  with the first line and still want to use 16GB with the second line.

  > On databases.conf
  >
  > MyTestDB = c:\Temp\MyDb.fdb
  > {
  > DefaultDbCachePages = 458752
  > }

  First of all, I'd suggest to reset FileSystemCacheSize back to zero and 
  set FileSystemCacheThreshold to something higher than 458752 pages (e.g. 
  50). Then re-test your memory usage and only then start tweaking the 
  filesystem cache.

  BTW, don't you occasionally have page cache override at the database 
  level? What is gstat -h output?


  Dmitry




  

  

  ++

  Visit http://www.firebirdsql.org and click the Documentation item
  on the main (top) menu.  Try FAQ and other links from the left-side menu 
there.

  Also search the knowledgebases at 
http://www.ibphoenix.com/resources/documents/ 

  ++
  

  Yahoo Groups Links

Re: [firebird-support] Re: Issue with Database Cache Size on FB 3.0

2016-06-01 Thread fabia...@itbizolutions.com.au [firebird-support] 
Vlad

Thank you, english is not my first language, I interpreted the sentence as "go 
calculate the total amount of ram, then set the value as a number within such 
percentage of your ram...", but thanks for the clarification. Perhaps to avoid 
other users making the same mistake each configurable variable on firebird .cfg 
files could have a "domain" explaining what type of variable it is, like 
"Domain = From 0 to 100, up to 2 decimals allowed, decimal separator is DOT". 
Furthermore, in this case I first understood what was required was actually a 
percentage number, but then changed my mind as it would be too hard for most 
users to think that way, as percentage mathematically speaking the way to 
represent 20% is actually 0.2, so it goes from 0 to 1.

I hope this helps other users,

Cheers,
Fabian

  - Original Message - 
  From: hv...@users.sourceforge.net [firebird-support] 
  To: firebird-support@yahoogroups.com 
  Sent: Wednesday, June 01, 2016 9:32 PM
  Subject: [firebird-support] Re: Issue with Database Cache Size on FB 3.0




  ---In firebird-support@yahoogroups.com, <fabianch@...> wrote :



  ...

  > The setting we are playing with are:
  > 
  > On firebird.conf
  > 
  > FileSystemCacheThreshold = 0
  > FileSystemCacheSize = 17179869184 (this is 16 GB - the server has 32 GB 
  > ram.)

Does you ever read comments describing this setting ? I can do it for you : 

  "measured in % of total physical RAM". 


  I.e. any value not in range [0..100] is wrong by definition. Read 
firebird.log, it should contain 
  error message:

  "Incorrect FileSystemCacheSize setting %d. Using default (30 percent)."

  If you really want to limit file system cache by 50% of avalable RAM - set 
FileSystemCacheSize 
  value to 50 (and restart Firebird), no need to calc in bytes...

  Regards,
  Vlad

[firebird-support] Re: Issue with Database Cache Size on FB 3.0

2016-06-01 Thread hv...@users.sourceforge.net [firebird-support] 
---In firebird-support@yahoogroups.com,  wrote :
 
 ...
 
> The setting we are playing with are:
 > 
> On firebird.conf
 > 
> FileSystemCacheThreshold = 0
> FileSystemCacheSize = 17179869184 (this is 16 GB - the server has 32 GB 
> ram.)

  Does you ever read comments describing this setting ? I can do it for you : 

"measured in % of total physical RAM". 

 
I.e. any value not in range [0..100] is wrong by definition. Read firebird.log, 
it should contain 
error message:

"Incorrect FileSystemCacheSize setting %d. Using default (30 percent)."

If you really want to limit file system cache by 50% of avalable RAM - set 
FileSystemCacheSize 
value to 50 (and restart Firebird), no need to calc in bytes...

Regards,
Vlad

[firebird-support] Re: Issue with Database Cache Size on FB 3.0

2016-05-31 Thread Dmitry Yemanov dim...@users.sourceforge.net [firebird-support] 
01.06.2016 04:16, Fabian Ernesto Chocron wrote:
>
> We are having trouble setting up the database cache size on FB 3.0
> running on Windows 2008 R2 64 bits with 32 GB ram.
>
> The problem we have is we cannot get the server to allocate the ram for
> the cache as we intend. With FB 2.54 we had the DB cache set very high,
> close to 1 GB per database, all running in RAM memory. With FB 3.0 we
> read it can allocate much more RAM to the cache, but it appears the
> server is allocating very small amount of Ram when the first user
> connects to the DB, and as we connect more users to the DB the ram
> consumption increases slowly.

What memory counters do you look at? Virtual (reserved) memory usage 
should grow high with the first connect, physical (committed) memory 
usage should grow together with your load, while pages are being read 
from disk.

> On firebird.conf
>
> FileSystemCacheThreshold = 0
> FileSystemCacheSize = 17179869184 (this is 16 GB - the server has 32 GB
> ram.)

These changes conflict with each other. You disable filesystem cache 
with the first line and still want to use 16GB with the second line.

> On databases.conf
>
> MyTestDB = c:\Temp\MyDb.fdb
> {
>   DefaultDbCachePages = 458752
> }

First of all, I'd suggest to reset FileSystemCacheSize back to zero and 
set FileSystemCacheThreshold to something higher than 458752 pages (e.g. 
50). Then re-test your memory usage and only then start tweaking the 
filesystem cache.

BTW, don't you occasionally have page cache override at the database 
level? What is gstat -h output?


Dmitry








++

Visit http://www.firebirdsql.org and click the Documentation item
on the main (top) menu.  Try FAQ and other links from the left-side menu there.

Also search the knowledgebases at http://www.ibphoenix.com/resources/documents/ 

++


Yahoo Groups Links

<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/firebird-support/

<*> Your email settings:
Individual Email | Traditional

<*> To change settings online go to:
http://groups.yahoo.com/group/firebird-support/join
(Yahoo! ID required)

<*> To change settings via email:
firebird-support-dig...@yahoogroups.com 
firebird-support-fullfeatu...@yahoogroups.com

<*> To unsubscribe from this group, send an email to:
firebird-support-unsubscr...@yahoogroups.com

<*> Your use of Yahoo Groups is subject to:
https://info.yahoo.com/legal/us/yahoo/utos/terms/