Re: [HACKERS] Regarding Postgres Dynamic Shared Memory (DSA)
One solution that is striking me is
1. I'll create one background worker and will initialise DSA in it.
2. If there are any callbacks available for client open/close connections,
i'll attach/detach to the DSA in those callbacks.
But i'm not sure there are such callbacks available. If such callbacks
exists in postgres and you guys know please help me out with that.
Thanks & Best Regards,
- Mahi
On Wed, May 24, 2017 at 11:32 AM, Mahi Gurram wrote:
> Hi,
>
> As Michael said, i'm creating DSA too early. Shared_Preload libraries are
> loading prior to memory related stuff.
>
> But i'm totally clueless how to solve my use case.
>
> Please help me with any work around.
>
> Thanks & Best Regards,
> - Mahi
>
> On Tue, May 23, 2017 at 5:52 PM, Mahi Gurram wrote:
>
>> Hi Michael,
>>
>> Thanks for your response.
>>
>> All i'm building is In-Memory Index as an extension over Postgres.
>>
>> Postgres Indexes will get Insert calls and Read calls from various
>> processes(typically client/connection process - forked processes to
>> postmaster process). Hence i have to maintain my In-Memory index in shared
>> memory.
>>
>> If i create DynamicSharedArea (DSA) in postmaster/main process, all these
>> Client/Connection processes(In-Memory Index Processes) need not attach to
>> that DSA using area handle. Because these are forked processes to
>> postmaster/Main process and hence they automatically gets attached.
>>
>> Hence i'm trying to create DSA in _PG_init function as it is called by
>> postmaster/main process.
>>
>> Hope this is clear.
>>
>> Thanks & Best Regards,
>> - Mahi
>>
>>
>> On Tue, May 23, 2017 at 5:30 PM, Michael Paquier <
>> michael.paqu...@gmail.com> wrote:
>>
>>> On Tue, May 23, 2017 at 6:42 AM, Mahi Gurram
>>> wrote:
>>> > I'm building In-Memory index extension for Postgres, for which i'm
>>> trying to
>>> > use DSA. But ended with some issues, as it is not allowing me to create
>>> > DSA(Dynamic Shared Area) in _PG_init function.
>>> >
>>> > Please refer my_PG_init code below:
>>> >>
>>> >> void
>>> >> _PG_init(void)
>>> >> {
>>> >> area = dsa_create(LWLockNewTrancheId(), "CustomIndex_DSA");
>>> >> area_handle = dsa_get_handle(area);
>>> >> }
>>> >
>>> >
>>> > Because of this code, Postgres is not starting. Not even giving any
>>> error
>>> > messages in pg logs. Hence, i'm totally clue less :(
>>>
>>> It seems to me that you are creating those too early. For example, for
>>> a background worker process, DSA segments would likely be created in
>>> the main process routine. Without understanding what you are trying to
>>> achieve, it is hard to make a good answer. You could always use a
>>> ramdisk, but that would be likely be a waste of memory as Postgres has
>>> its own buffer pool, killing the performance gains of OS caching.
>>> --
>>> Michael
>>>
>>
>>
>
Re: [HACKERS] Regarding Postgres Dynamic Shared Memory (DSA)
Hi,
As Michael said, i'm creating DSA too early. Shared_Preload libraries are
loading prior to memory related stuff.
But i'm totally clueless how to solve my use case.
Please help me with any work around.
Thanks & Best Regards,
- Mahi
On Tue, May 23, 2017 at 5:52 PM, Mahi Gurram wrote:
> Hi Michael,
>
> Thanks for your response.
>
> All i'm building is In-Memory Index as an extension over Postgres.
>
> Postgres Indexes will get Insert calls and Read calls from various
> processes(typically client/connection process - forked processes to
> postmaster process). Hence i have to maintain my In-Memory index in shared
> memory.
>
> If i create DynamicSharedArea (DSA) in postmaster/main process, all these
> Client/Connection processes(In-Memory Index Processes) need not attach to
> that DSA using area handle. Because these are forked processes to
> postmaster/Main process and hence they automatically gets attached.
>
> Hence i'm trying to create DSA in _PG_init function as it is called by
> postmaster/main process.
>
> Hope this is clear.
>
> Thanks & Best Regards,
> - Mahi
>
>
> On Tue, May 23, 2017 at 5:30 PM, Michael Paquier <
> michael.paqu...@gmail.com> wrote:
>
>> On Tue, May 23, 2017 at 6:42 AM, Mahi Gurram wrote:
>> > I'm building In-Memory index extension for Postgres, for which i'm
>> trying to
>> > use DSA. But ended with some issues, as it is not allowing me to create
>> > DSA(Dynamic Shared Area) in _PG_init function.
>> >
>> > Please refer my_PG_init code below:
>> >>
>> >> void
>> >> _PG_init(void)
>> >> {
>> >> area = dsa_create(LWLockNewTrancheId(), "CustomIndex_DSA");
>> >> area_handle = dsa_get_handle(area);
>> >> }
>> >
>> >
>> > Because of this code, Postgres is not starting. Not even giving any
>> error
>> > messages in pg logs. Hence, i'm totally clue less :(
>>
>> It seems to me that you are creating those too early. For example, for
>> a background worker process, DSA segments would likely be created in
>> the main process routine. Without understanding what you are trying to
>> achieve, it is hard to make a good answer. You could always use a
>> ramdisk, but that would be likely be a waste of memory as Postgres has
>> its own buffer pool, killing the performance gains of OS caching.
>> --
>> Michael
>>
>
>
Re: [HACKERS] Regarding Postgres Dynamic Shared Memory (DSA)
Hi Michael,
Thanks for your response.
All i'm building is In-Memory Index as an extension over Postgres.
Postgres Indexes will get Insert calls and Read calls from various
processes(typically client/connection process - forked processes to
postmaster process). Hence i have to maintain my In-Memory index in shared
memory.
If i create DynamicSharedArea (DSA) in postmaster/main process, all these
Client/Connection processes(In-Memory Index Processes) need not attach to
that DSA using area handle. Because these are forked processes to
postmaster/Main process and hence they automatically gets attached.
Hence i'm trying to create DSA in _PG_init function as it is called by
postmaster/main process.
Hope this is clear.
Thanks & Best Regards,
- Mahi
On Tue, May 23, 2017 at 5:30 PM, Michael Paquier
wrote:
> On Tue, May 23, 2017 at 6:42 AM, Mahi Gurram wrote:
> > I'm building In-Memory index extension for Postgres, for which i'm
> trying to
> > use DSA. But ended with some issues, as it is not allowing me to create
> > DSA(Dynamic Shared Area) in _PG_init function.
> >
> > Please refer my_PG_init code below:
> >>
> >> void
> >> _PG_init(void)
> >> {
> >> area = dsa_create(LWLockNewTrancheId(), "CustomIndex_DSA");
> >> area_handle = dsa_get_handle(area);
> >> }
> >
> >
> > Because of this code, Postgres is not starting. Not even giving any error
> > messages in pg logs. Hence, i'm totally clue less :(
>
> It seems to me that you are creating those too early. For example, for
> a background worker process, DSA segments would likely be created in
> the main process routine. Without understanding what you are trying to
> achieve, it is hard to make a good answer. You could always use a
> ramdisk, but that would be likely be a waste of memory as Postgres has
> its own buffer pool, killing the performance gains of OS caching.
> --
> Michael
>
Re: [HACKERS] Regarding B-Tree Lookup
Thank you all for your responses to help me out.
The below code worked for me... pasting it here (May be helpful for someone)
Relation heap;
> ItemPointer ht_ctid;
> heap = heap_open(50620, AccessShareLock); /* 50620 - Table/Relation Oid -
> Hardcoded for better understanding */
ScanKeyInit(&skey, 1, BTEqualStrategyNumber, F_INT8EQ, Int64GetDatum(100));
> /* 100 is the Value of PK of which i'm doing a lookup*/
> scan = systable_beginscan(heap, 50624 , true, NULL, 1, &skey); // 50624 is
> the Oid of PKIndex for this Table/Relation.
tuple = systable_getnext(scan);
> if (HeapTupleIsValid(tuple)){
> ht_ctid = &tuple->t_self;
> }
> systable_endscan(scan);
> heap_close(heap, AccessShareLock);
Hope this helps for some one.
PS: You have to use different Procedure(F_INT8EQ/F_TEXTEQ etc..) and Datum
Conversion functions(Int64GetDatum/CStringGetTextDatum etc..) in
ScanKeyInit() function as per your PK Data Types.
Thanks & Best Regards,
- Mahi
On Tue, May 2, 2017 at 5:59 PM, Tom Lane wrote:
> Michael Paquier writes:
> > On Tue, May 2, 2017 at 6:12 PM, Mahi Gurram wrote:
> >> Please suggest me the easiest way to lookup into PK's B-Tree index for
> >> getting TIDs.
>
> > Why don't you just use SPI within your extension? No need to copy the
> > logic for btree lookups this way.
>
> There's not actually that much code needed, though -- basically
> index_beginscan, index_rescan, index_getnext, index_endscan. One possible
> model to follow is systable_beginscan and friends, in genam.c.
>
> I think that you could possibly get away with just applying
> systable_beginscan to random user tables, even. But it's at least a
> conceptual mismatch, and there are some things in there, like the
> test on IgnoreSystemIndexes, that you probably don't want.
>
> regards, tom lane
>
[HACKERS] Regarding Postgres Dynamic Shared Memory (DSA)
Hello everyone,
I'm building In-Memory index extension for Postgres, for which i'm trying
to use DSA. But ended with some issues, as it is not allowing me to create
DSA(Dynamic Shared Area) in _PG_init function.
Please refer my_PG_init code below:
> void
> _PG_init(void)
> {
> area = dsa_create(LWLockNewTrancheId(), "CustomIndex_DSA");
> area_handle = dsa_get_handle(area);
> }
Because of this code, Postgres is not starting. Not even giving any error
messages in pg logs. Hence, i'm totally clue less :(
Please let me know how to proceed. Your help is highly appreciated.
PS: Applied all the DSM related patches over Postgres 9.6.1 as per below
thread and confirmed the same by running some test programs.
https://www.postgresql.org/message-id/flat/CAEepm%3D1z5WLuNoJ80PaCvz6EtG9dN0j-KuHcHtU6QEfcPP5-qA%40mail.gmail.com#CAEepm=1z5wlunoj80pacvz6etg9dn0j-kuhchtu6qefcpp5...@mail.gmail.com
Thanks & Best Regards,
- Mahi
[HACKERS] Regarding B-Tree Lookup
Hi, I'm building some custom extension on top of postgres 9.6.1. As part of that, I would like to read Heap Tuple directly from my extension using Primary Key. By default, postgres table index(B-Tree) its PrimaryKey(PK). So, i would like to get TID by doing a lookup into PK's B-Tree index. Using which i could read HeapTuple directly. Please suggest me the easiest way to lookup into PK's B-Tree index for getting TIDs. Suggesting a postgres extensions which does B-Tree lookup will also helps me. Awaiting your response. Thanks & Best Regards - Mahi
