Hello, Sean,
>> May be high memory usage caused by procedure's local variables?
> That is the problem.
>
> I commented out the CREATE_File lines and the memory usage still increased.
>
> What I don't understand is why a temporary/local usage of a blob would
> consume memory and not release it when the reference was cleared.
>
>
> Sean
>
Let me explain:
1) Historically, temporary BLOBs were allocated from transaction pool and were
deallocated when
transaction finished.
User received plain BLOB pointer, and engine dereferenced whatever was passed
by the user.
Memory for temporary BLOBs is usually released because temporary BLOBs get
assigned to table columns
and materialized to database.
But this never happens if BLOBs are not stored in database as in your case.
Thus original Interbase/Firebird implementation for temporary BLOBs was
unstable and could consume
all available RAM until the end of transaction.
2) A few years ago when working in your Toronto office, (during 64-bit porting
effort) I implemented
BLOB handles in Firebird and crude BLOB lifetime tracking that binds BLOB's
lifetime to a request
unless BLOB ID was passed away to the user.
The code like this:
...
// Assign temporary BLOB ownership to top-level request if it is not
assigned yet
jrd_req* own_request;
if (blobIndex->bli_request) {
....
}
else
{
own_request = request;
while (own_request->req_caller)
own_request = own_request->req_caller;
blobIndex->bli_request = own_request;
own_request->req_blobs.add(blob->blb_temp_id);
}
...
This allows to release some memory when request finishes its processing.
3) There is still no mechanism in the engine for BLOB lifetime or reference
tracking. Engine
operation in regards to BLOBs is based on crude heuristics and certain
assumptions about users' code
behavior.
For example, there is no logic to dematerialize BLOB during savepoint backout
(VIO_backout). There
might me surviving references to original temporary BLOB, and they won't work
(you likely get a
BUGCHECK).
We implemented more accurate accounting which BLOB handles are were given out
to the user, for
security purposes. This is not yet merged into Firebird.
If we know all the points where BLOB references are handled this creates a
foundation for perfectly
accurate BLOB status model and lifetime tracking.
It could be used to implement internal reference counts, for example, and fix
both dematerialization
problem and memory consumption problem.
But AFAIK, nobody in the Firebird projects seems to care about accurate BLOB
tracking. Current
half-baked solution to BLOB lifetime problems seems to suit everybody.
Even basic Red Soft changes for BLOB tracking do not get merged (even though
they fix an obvious
security issue).
4) There is a new isc_bpb_storage_temp flag for BLOBs that is used internally
in the engine. This is
a new partial solution to the blob tracking problem (and we use it for
READ_FILE too).
It shifts even more burden towards RAM and temporary storage.
If an SQL query needs to process (e.g. search) terabytes of external content as
in our case, memory
consumption becomes impossible even for our machines with 2 TB of RAM on each
node.
All in all, I believe that proper implementation of BLOB lifetime tracking in
Firebird is long overdue.
Nikolay Samofatov
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
