Hello, First some background: For my application I need to collect and retrieve data at a very fast rate, faster than postgres can produce, but I also have the need to store and access a large range of metadata about the large sets of data I am storing. I am therefore attempting to use postgres for the metadata in combination with the HDF5 filesystem for the large sets of data to store data fast and have all of the power of a relational database over the metadata.
Now onto the problem: To combine the two sets of data I have created a C function which can be dynamically loaded into postgres and takes a primary key index into the data and returns a large table (around 30000 elements) of the data (value and timestamp). When I access the data directly it takes me around 30ms to just read the data out of my test case. As the postgres C function is called many times and I must convert the raw data to tuples it introduces more overhead bringing the data to take around 2000ms to run the testcase, this is still acceptable. My problem is when I perform an actual query over my test case postgres takes around 22000 ms to retrieve the sets of data. My concern is: what is postgres doing in these extra 20 seconds, and how can I make this run faster? In summary: I have a large testcase. I am running a C function from postgres which takes a primary key and returns a setof (int4, int4) with around 30000 tuples returned per call my c function, when timed over the duration of the test case, takes 2 seconds to run. the postgres query "select * from function(args);" takes around 11 times this amount of time to execute my function is defined as create or replace function(IN args, IN..., OUT name1 int4, OUT name2 int4) returns setof record as 'libname.so', 'fcnname' LANGUAGE C IMMUTABLE STRICT; I'm sorry if this is the wrong forum, I was torn between hackers and performance. Thank you for any assistance. -- View this message in context: http://www.nabble.com/Postgres-delays-function-returning-large-set-of-data-tp23853886p23853886.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
