Re: [HACKERS] What is the posix_memalign() equivalent for the PostgreSQL?
The unexpected errors are problems to free the memory that was previously
allocated with posix_memalign. After a number of memory allocation and free
calls, the next free call crashes the system.
On the other hand, if I replace the posix_memalign and the free calls for
the following malloc/free aligned implementations, it works.
void *malloc_aligned(size_t alignment, size_t bytes) {
void *mallocPtr;//Initial pointer returned from malloc
void *newMallocPtr;//New pointer after adjustment
void *alignedPtr;
size_t alignMask;//Need this to get the aligned address
size_t totalBytes = 0;
/* Make sure alignment is power of 2 and it is not zero
* because zero is not power of 2 */
if ( !(!(alignment & (alignment-1)) && alignment) )
return NULL;
/* We need to allocate extra memory to make sure the allocated
* memory will be aligned and need sizeof(size_t) bytes more for
* storing the value of the bytes we padded.
*/
totalBytes = bytes + alignment + sizeof(size_t);
mallocPtr = palloc(totalBytes);
if (NULL == mallocPtr)
return NULL;
newMallocPtr = (void*)((char*)mallocPtr + sizeof(size_t));
alignMask = ~(alignment - 1);
/* Value of alignedPtr should be multiple of alignment */
alignedPtr = (void *)(((size_t)newMallocPtr + alignment) & alignMask);
/* Store the extra bytes info right before alignedPtr */
*((size_t*)alignedPtr - 1) = (size_t)alignedPtr - (size_t)mallocPtr;
return alignedPtr;
}
void free_aligned(void *raw_data) {
void *mallocPtr;//Initial malloc pointer
size_t extraBytes;
if (NULL == raw_data)
return;
/* Retrieve the extra padded byte info */
extraBytes = *((size_t*)raw_data - 1);
/* Get initial malloc ptr */
mallocPtr = (void*) ((size_t)raw_data - extraBytes);
pfree(mallocPtr);
}
Please note that I am using in these functions, the palloc and pfree
instead of malloc and free respectively. But the problem is that the
free_aligned function is not indeed freeing the allocated memory. Thus, I
would like to know if the PostgreSQL provides a memory function that
allocates aligned memory. If not, according to your experience, is there a
significance difference between the performance of the O_DIRECT or not?
Thank you,
Anderson
2016-09-02 7:24 GMT-03:00 Craig Ringer <cr...@2ndquadrant.com>:
> On 2 September 2016 at 01:12, Anderson Carniel <accarn...@gmail.com>
> wrote:
> > Dear all,
> >
> > I am developing an extension for the PostgreSQL that write/read some
> > external files from the PostgreSQL. In order to write/read, I am using
> the
> > O_DIRECT flag and using the posix_memalign to allocate memory. I would
> like
> > to know if the postgresql internal library provides an equivalent
> function
> > for the posix_memalign since I am getting unexpected errors.
>
> "unexpected errors". Details please?
>
> If you're trying to allocate aligned memory, I believe PostgreSQL
> typically uses the TYPEALIGN macros (see c.h) but I'm painfully
> clueless in the area, so ... yeah. Don't trust me.
>
> I was a bit surprised not to see a MemoryContextAlloc or palloc
> variant that returns memory aligned to a given boundary.
>
> > All my
> > allocations are in the TopMemoryContext since I am working with several
> > buffers that must be alive while the PostgreSQL Server is activated.
>
> You can't posix_memalign into TopMemoryContext. Such memory is outside
> the memory context system, like memory directly malloc()'d.
>
> --
> Craig Ringer http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Training & Services
>
[HACKERS] What is the posix_memalign() equivalent for the PostgreSQL?
Dear all, I am developing an extension for the PostgreSQL that write/read some external files from the PostgreSQL. In order to write/read, I am using the O_DIRECT flag and using the posix_memalign to allocate memory. I would like to know if the postgresql internal library provides an equivalent function for the posix_memalign since I am getting unexpected errors. All my allocations are in the TopMemoryContext since I am working with several buffers that must be alive while the PostgreSQL Server is activated. Thanks in advance, Anderson
Re: [HACKERS] Losing memory references - SRF + SPI
Thank you very much Joe. I have followed the crosstab() implementation and understood the idea of per query memory context. Now, I am using a unique SPI instance (which I perform several sql queries), process the result, transform my result into a tuplestore, close the SPI and done. It works perfectly. I have a curiosity with regard to the tuplestore: is there a problem with performance if my tuplestore form a big table with million of tuples? Other question is regarding to SPI: is there a problem to use only one instance of SPI (for instance, if multiple users call the same function)? Thank you again, Anderson Carniel 2016-05-14 12:19 GMT-03:00 Joe Conway <m...@joeconway.com>: > On 05/13/2016 09:35 PM, Anderson Carniel wrote: > > I am writing a function that returns a set of tuples by using also the > > PostGIS. Thuis, I am using SRF too. It successfully returns the expected > > result when it has at most 4 tuples. However, this is not the case when > > more than 4 tuples have to be returned. When I debug the code, I found > > that the problem is in my function that transforms a cstring after a > > SPI_connection. It seems that this cstring is not valid anymore in the > > moment of this conversion (see my comment below). I know that the SPI > > uses different contexts when it init and finish its process. But, I > > don't understand why I have this problem here. Please, note that I tried > > to copy the values of the whole tuple, but I have the same problem: > > system crash after the forth call of the function. Also note that I call > > this function only in the init call of the SRF. Please I would > > appreciate any suggestion and help. > > You probably need to allocate your returned values in a per query memory > context. Take a look at how it is done in, for example, crosstab() in > contrib/tablefunc. > > HTH, > > Joe > > -- > Crunchy Data - http://crunchydata.com > PostgreSQL Support for Secure Enterprises > Consulting, Training, & Open Source Development > >
[HACKERS] Losing memory references - SRF + SPI
I am writing a function that returns a set of tuples by using also the
PostGIS. Thuis, I am using SRF too. It successfully returns the expected
result when it has at most 4 tuples. However, this is not the case when
more than 4 tuples have to be returned. When I debug the code, I found that
the problem is in my function that transforms a cstring after a
SPI_connection. It seems that this cstring is not valid anymore in the
moment of this conversion (see my comment below). I know that the SPI uses
different contexts when it init and finish its process. But, I don't
understand why I have this problem here. Please, note that I tried to copy
the values of the whole tuple, but I have the same problem: system crash
after the forth call of the function. Also note that I call this function
only in the init call of the SRF. Please I would appreciate any suggestion
and help.
--- code of the problematic function here ---
LWGEOM *retrieve_geom_from_postgis(int row_id) {
char query[100];
int err;
char *wkt;
int srid;
LWGEOM *lwgeom = NULL;
HeapTuple cop;
bool null;
TupleDesc tupdesc;
//refin is a prepared select command that returns 2 columns
sprintf(query, "EXECUTE refinplan(%d);", row_id);
if (SPI_OK_CONNECT != SPI_connect()) {
SPI_finish();
_DEBUG(ERROR, "retrieve_geom_from_postgis: could not connect to SPI
manager");
return NULL;
}
err = SPI_execute(query, false, 1);
if (err < 0) {
SPI_finish();
_DEBUG(ERROR, "retrieve_geom_from_postgis: could not execute the
EXECUTE command");
return NULL;
}
if (SPI_processed <= 0) {
SPI_finish();
_DEBUGF(ERROR, "the row_id (%d) does not exist in the table",
row_id)
return NULL;
}
cop = SPI_copytuple(SPI_tuptable->vals[0]);
tupdesc = SPI_tuptable->tupdesc;
/* disconnect from SPI */
SPI_finish();
wkt = text2cstring(DatumGetTextP(heap_getattr(cop, 1, tupdesc, )));
srid = DatumGetInt32(heap_getattr(cop, 2, tupdesc, ));
lwgeom = lwgeom_from_wkt(wkt, LW_PARSER_CHECK_NONE); //error here...
only after the forth call
lwgeom_set_srid(lwgeom, srid);
lwfree(wkt);
return lwgeom;
}
