Re: [HACKERS] Pluggable storage

2017-10-26 Thread Robert Haas
On Wed, Oct 25, 2017 at 1:59 PM, Amit Kapila wrote: >> Another thing to consider is that, if we could replace satisfiesfunc, >> it would probably break some existing code. There are multiple places >> in the code that compare snapshot->satisfies to >>

Re: [HACKERS] Pluggable storage

2017-10-25 Thread Amit Kapila
On Wed, Oct 25, 2017 at 11:37 AM, Robert Haas wrote: > On Fri, Oct 20, 2017 at 5:47 AM, Amit Kapila wrote: >> I think what we need here is a way to register satisfies function >> (SnapshotSatisfiesFunc) in SnapshotData for different storage

Re: [HACKERS] Pluggable storage

2017-10-25 Thread Robert Haas
On Fri, Oct 20, 2017 at 5:47 AM, Amit Kapila wrote: > I think what we need here is a way to register satisfies function > (SnapshotSatisfiesFunc) in SnapshotData for different storage engines. I don't see how that helps very much. SnapshotSatisfiesFunc takes a HeapTuple

Re: [HACKERS] Pluggable storage

2017-10-19 Thread Amit Kapila
On Sat, Oct 14, 2017 at 1:09 AM, Alexander Korotkov wrote: > On Fri, Oct 13, 2017 at 9:41 PM, Robert Haas wrote: >> >> On Fri, Oct 13, 2017 at 1:59 PM, Peter Geoghegan wrote: >> >> Fully agreed. >> > >> > If we implement that

Re: [HACKERS] Pluggable storage

2017-10-19 Thread Amit Kapila
On Fri, Oct 13, 2017 at 1:58 PM, Haribabu Kommi wrote: > > On Fri, Oct 13, 2017 at 11:55 AM, Robert Haas wrote: >> >> On Thu, Oct 12, 2017 at 8:00 PM, Haribabu Kommi >> wrote: >> > Currently I added a snapshot_satisfies

Re: [HACKERS] Pluggable storage

2017-10-13 Thread Alexander Korotkov
On Fri, Oct 13, 2017 at 9:41 PM, Robert Haas wrote: > On Fri, Oct 13, 2017 at 1:59 PM, Peter Geoghegan wrote: > >> Fully agreed. > > > > If we implement that interface, where does that leave EvalPlanQual()? > >From the first glance, it seems that pluggable

Re: [HACKERS] Pluggable storage

2017-10-13 Thread Alexander Korotkov
On Fri, Oct 13, 2017 at 9:37 PM, Robert Haas wrote: > On Fri, Oct 13, 2017 at 5:25 AM, Kuntal Ghosh > wrote: > > For some other > > storage engine, if we maintain the older version in different storage, > > undo for example, and don't require a

Re: [HACKERS] Pluggable storage

2017-10-13 Thread Robert Haas
On Fri, Oct 13, 2017 at 1:59 PM, Peter Geoghegan wrote: >> Fully agreed. > > If we implement that interface, where does that leave EvalPlanQual()? > Do those semantics have to be preserved? For a general-purpose heap storage format, I would say yes. I mean, we don't really have

Re: [HACKERS] Pluggable storage

2017-10-13 Thread Robert Haas
On Fri, Oct 13, 2017 at 5:25 AM, Kuntal Ghosh wrote: > For some other > storage engine, if we maintain the older version in different storage, > undo for example, and don't require a new index entry, should we still > call it HOT-chain? I would say, emphatically, no.

Re: [HACKERS] Pluggable storage

2017-10-13 Thread Peter Geoghegan
On Thu, Oct 12, 2017 at 2:23 PM, Robert Haas wrote: > On Thu, Oct 12, 2017 at 4:38 PM, Alexander Korotkov >> However I imply that alternative storage would share our "MVCC model". So, >> it >> should share our transactional model including transactions, >> subtransactions,

Re: [HACKERS] Pluggable storage

2017-10-13 Thread Kuntal Ghosh
On Fri, Oct 13, 2017 at 1:58 PM, Haribabu Kommi wrote: > > > On Fri, Oct 13, 2017 at 11:55 AM, Robert Haas wrote: >> >> On Thu, Oct 12, 2017 at 8:00 PM, Haribabu Kommi >> wrote: >> >> That seems like a strange choice of

Re: [HACKERS] Pluggable storage

2017-10-13 Thread Haribabu Kommi
On Fri, Oct 13, 2017 at 11:55 AM, Robert Haas wrote: > On Thu, Oct 12, 2017 at 8:00 PM, Haribabu Kommi > wrote: > > Currently I added a snapshot_satisfies API to find out whether the tuple > > satisfies the visibility or not with different types

Re: [HACKERS] Pluggable storage

2017-10-12 Thread Robert Haas
On Thu, Oct 12, 2017 at 8:00 PM, Haribabu Kommi wrote: > Currently I added a snapshot_satisfies API to find out whether the tuple > satisfies the visibility or not with different types of visibility routines. > I feel these > are some how enough to develop a different

Re: [HACKERS] Pluggable storage

2017-10-12 Thread Haribabu Kommi
On Fri, Oct 13, 2017 at 8:23 AM, Robert Haas wrote: > On Thu, Oct 12, 2017 at 4:38 PM, Alexander Korotkov > wrote: > > It's probably that we imply different meaning to "MVCC implementation". > > While writing "MVCC implementation" I meant that,

Re: [HACKERS] Pluggable storage

2017-10-12 Thread Robert Haas
On Thu, Oct 12, 2017 at 4:38 PM, Alexander Korotkov wrote: > It's probably that we imply different meaning to "MVCC implementation". > While writing "MVCC implementation" I meant that, for instance, alternative > storage > may implement UNDO chains to store versions of

Re: [HACKERS] Pluggable storage

2017-10-12 Thread Alexander Korotkov
On Wed, Oct 11, 2017 at 11:08 PM, Robert Haas wrote: > On Mon, Oct 9, 2017 at 10:22 AM, Alexander Korotkov > wrote: > > For me, it's crucial point that pluggable storages should be able to have > > different MVCC implementation, and

Re: [HACKERS] Pluggable storage

2017-10-11 Thread Peter Geoghegan
On Wed, Oct 11, 2017 at 1:08 PM, Robert Haas wrote: > On Mon, Oct 9, 2017 at 10:22 AM, Alexander Korotkov > wrote: >> For me, it's crucial point that pluggable storages should be able to have >> different MVCC implementation, and correspondingly

Re: [HACKERS] Pluggable storage

2017-10-11 Thread Robert Haas
On Mon, Oct 9, 2017 at 10:22 AM, Alexander Korotkov wrote: > For me, it's crucial point that pluggable storages should be able to have > different MVCC implementation, and correspondingly have full control over > its interactions with indexes. > Thus, it would be good

Re: [HACKERS] Pluggable storage

2017-10-09 Thread Alexander Korotkov
On Mon, Oct 9, 2017 at 5:32 PM, Tom Lane wrote: > Alexander Korotkov writes: > > For me, it's crucial point that pluggable storages should be able to have > > different MVCC implementation, and correspondingly have full control over > > its

Re: [HACKERS] Pluggable storage

2017-10-09 Thread Tom Lane
Alexander Korotkov writes: > For me, it's crucial point that pluggable storages should be able to have > different MVCC implementation, and correspondingly have full control over > its interactions with indexes. > Thus, it would be good if we would get consensus on that

Re: [HACKERS] Pluggable storage

2017-10-09 Thread Alexander Korotkov
On Wed, Sep 27, 2017 at 7:51 PM, Alexander Korotkov < a.korot...@postgrespro.ru> wrote: > I took a look on this patch. I've following notes for now. > > tuple_insert_hook tuple_insert; /* heap_insert */ >> tuple_update_hook tuple_update; /* heap_update */ >> tuple_delete_hook tuple_delete; /*

Re: [HACKERS] Pluggable storage

2017-09-27 Thread Alexander Korotkov
Hi! I took a look on this patch. I've following notes for now. tuple_insert_hook tuple_insert; /* heap_insert */ > tuple_update_hook tuple_update; /* heap_update */ > tuple_delete_hook tuple_delete; /* heap_delete */ I don't think this set of functions provides good enough level of

Re: [HACKERS] Pluggable storage

2017-09-27 Thread Alexander Korotkov
On Wed, Aug 23, 2017 at 8:26 AM, Haribabu Kommi wrote: > - Minor: don't think the _function suffix for Storis necessary, just >> makes things long, and every member has it. Besides that, it's also >> easy to misunderstand - for a second I understood >>

Re: [HACKERS] Pluggable storage

2017-09-27 Thread Alexander Korotkov
Hi, Thank you for review on this subject. I think it's extremely important for PostgreSQL to eventually get pluggable storage API. In general I agree with all your points. But I'd like to make couple of comments. On Tue, Aug 15, 2017 at 9:53 AM, Andres Freund wrote: > - I

Re: [HACKERS] Pluggable storage

2017-09-14 Thread Alexander Korotkov
On Thu, Sep 14, 2017 at 8:17 AM, Haribabu Kommi wrote: > Instead of modifying the Bitmap Heap and Sample scan's to avoid referring > the internal members of the HeapScanDesc, I divided the HeapScanDesc > into two parts. > > 1. StorageScanDesc > 2. HeapPageScanDesc > >

Re: [HACKERS] Pluggable storage

2017-09-11 Thread Haribabu Kommi
On Sat, Sep 9, 2017 at 1:23 PM, Haribabu Kommi wrote: > > I rebased the patch to the latest master and also fixed the duplicate OID > and some slot fixes. Updated patches are attached. > While analyzing the removal of HeapScanDesc usage other than heap modules, The

Re: [HACKERS] Pluggable storage

2017-09-06 Thread Thomas Munro
On Fri, Sep 1, 2017 at 1:51 PM, Haribabu Kommi wrote: > Here I attached new set of patches that are rebased to the latest master. Hi Haribabu, Could you please post a new rebased version? 0007-Scan-functions-are-added-to-storage-AM.patch conflicts with recent changes.

Re: [HACKERS] Pluggable storage

2017-08-25 Thread Haribabu Kommi
On Wed, Aug 23, 2017 at 11:59 PM, Amit Kapila wrote: > On Wed, Aug 23, 2017 at 11:05 AM, Haribabu Kommi > wrote: > > > > > > On Mon, Aug 21, 2017 at 7:25 PM, Amit Kapila > > wrote: > >> > >> On Mon, Aug 21, 2017 at

Re: [HACKERS] Pluggable storage

2017-08-23 Thread Amit Kapila
On Wed, Aug 23, 2017 at 11:05 AM, Haribabu Kommi wrote: > > > On Mon, Aug 21, 2017 at 7:25 PM, Amit Kapila > wrote: >> >> On Mon, Aug 21, 2017 at 12:58 PM, Haribabu Kommi >> wrote: >> > >> > On Sun, Aug 13, 2017 at

Re: [HACKERS] Pluggable storage

2017-08-22 Thread Haribabu Kommi
On Mon, Aug 21, 2017 at 7:25 PM, Amit Kapila wrote: > On Mon, Aug 21, 2017 at 12:58 PM, Haribabu Kommi > wrote: > > > > On Sun, Aug 13, 2017 at 5:17 PM, Amit Kapila > > wrote: > >> > >> > >> Also, it is quite possible

Re: [HACKERS] Pluggable storage

2017-08-22 Thread Haribabu Kommi
On Tue, Aug 15, 2017 at 4:53 PM, Andres Freund wrote: > Hi, > > > On 2017-06-13 11:50:27 +1000, Haribabu Kommi wrote: > > Here I attached WIP patches to support pluggable storage. The patch > series > > are may not work individually. Still so many things are under >

Re: [HACKERS] Pluggable storage

2017-08-21 Thread Amit Kapila
On Mon, Aug 21, 2017 at 12:58 PM, Haribabu Kommi wrote: > > On Sun, Aug 13, 2017 at 5:17 PM, Amit Kapila > wrote: >> >> >> Also, it is quite possible that some of the storage Am's don't even >> want to return bool as a parameter from

Re: [HACKERS] Pluggable storage

2017-08-21 Thread Haribabu Kommi
On Sun, Aug 13, 2017 at 5:17 PM, Amit Kapila wrote: > On Sat, Aug 12, 2017 at 10:31 AM, Haribabu Kommi > wrote: > >> > >> Why do we need to store handler function in TupleDesc? As of now, the > >> above patch series has it available in

Re: [HACKERS] Pluggable storage

2017-08-15 Thread Andres Freund
Hi, On 2017-06-13 11:50:27 +1000, Haribabu Kommi wrote: > Here I attached WIP patches to support pluggable storage. The patch series > are may not work individually. Still so many things are under development. > These patches are just to share the approach of the current development. Making a

Re: [HACKERS] Pluggable storage

2017-08-13 Thread Haribabu Kommi
On Sun, Aug 13, 2017 at 5:21 PM, Amit Kapila wrote: > On Sat, Aug 12, 2017 at 10:34 AM, Haribabu Kommi > wrote: > > > > On Tue, Aug 8, 2017 at 2:21 PM, Amit Kapila > wrote: > >> > >> +typedef struct StorageAmRoutine >

Re: [HACKERS] Pluggable storage

2017-08-13 Thread Amit Kapila
On Sat, Aug 12, 2017 at 10:34 AM, Haribabu Kommi wrote: > > On Tue, Aug 8, 2017 at 2:21 PM, Amit Kapila wrote: >> >> On Tue, Jun 13, 2017 at 7:20 AM, Haribabu Kommi >> wrote: >> > >> > >> > On Fri, Oct 14, 2016 at 7:26

Re: [HACKERS] Pluggable storage

2017-08-13 Thread Amit Kapila
On Sat, Aug 12, 2017 at 10:31 AM, Haribabu Kommi wrote: >> >> Why do we need to store handler function in TupleDesc? As of now, the >> above patch series has it available in RelationData and >> TupleTableSlot, I am not sure if instead of that keeping it in >> TupleDesc

Re: [HACKERS] Pluggable storage

2017-08-11 Thread Haribabu Kommi
On Tue, Aug 8, 2017 at 2:21 PM, Amit Kapila wrote: > On Tue, Jun 13, 2017 at 7:20 AM, Haribabu Kommi > wrote: > > > > > > On Fri, Oct 14, 2016 at 7:26 AM, Alvaro Herrera < > alvhe...@2ndquadrant.com> > > wrote: > >> > >> I have sent the partial

Re: [HACKERS] Pluggable storage

2017-08-11 Thread Haribabu Kommi
On Mon, Aug 7, 2017 at 11:12 PM, Amit Kapila wrote: > On Tue, Aug 1, 2017 at 1:56 PM, Haribabu Kommi > wrote: > > > > > > On Sun, Jul 23, 2017 at 4:10 PM, Amit Kapila > > wrote: > >> > > > >> > >> > 1. Design an API

Re: [HACKERS] Pluggable storage

2017-08-07 Thread Amit Kapila
On Tue, Jun 13, 2017 at 7:20 AM, Haribabu Kommi wrote: > > > On Fri, Oct 14, 2016 at 7:26 AM, Alvaro Herrera > wrote: >> >> I have sent the partial patch I have to Hari Babu Kommi. We expect that >> he will be able to further this goal some

Re: [HACKERS] Pluggable storage

2017-08-07 Thread Amit Kapila
On Tue, Aug 1, 2017 at 1:56 PM, Haribabu Kommi wrote: > > > On Sun, Jul 23, 2017 at 4:10 PM, Amit Kapila > wrote: >> > >> >> > 1. Design an API that returns values/nulls array and convert that into a >> > HeapTuple whenever it is required in the

Re: [HACKERS] Pluggable storage

2017-08-01 Thread Haribabu Kommi
On Sun, Jul 23, 2017 at 4:10 PM, Amit Kapila wrote: > On Wed, Jul 19, 2017 at 11:33 AM, Haribabu Kommi > wrote: > > > > I am finding out that eliminating the HeapTuple usage in the upper layers > > needs some major changes, How about not

Re: [HACKERS] Pluggable storage

2017-07-23 Thread Amit Kapila
On Wed, Jul 19, 2017 at 11:33 AM, Haribabu Kommi wrote: > > > On Sat, Jul 15, 2017 at 12:30 PM, Robert Haas wrote: >> > > I am finding out that eliminating the HeapTuple usage in the upper layers > needs some major changes, How about not changing

Re: [HACKERS] Pluggable storage

2017-07-19 Thread Peter Geoghegan
On Wed, Jul 19, 2017 at 10:56 AM, Robert Haas wrote: >> I strongly agree. I simply don't understand how you can adopt UNDO for >> MVCC, and yet expect to get a benefit commensurate with the effort >> without also implementing "retail index tuple deletion" first. > > I agree

Re: [HACKERS] Pluggable storage

2017-07-19 Thread Robert Haas
On Sat, Jul 15, 2017 at 8:58 PM, Peter Geoghegan wrote: > To repeat myself, for emphasis: *Not all bloat is equal*. +1. > I strongly agree. I simply don't understand how you can adopt UNDO for > MVCC, and yet expect to get a benefit commensurate with the effort > without also

Re: [HACKERS] Pluggable storage

2017-07-19 Thread Robert Haas
On Sat, Jul 15, 2017 at 6:36 PM, Alexander Korotkov wrote: > I think in general there are two ways dealing with out index AM API > limitation. One of them is to extend index AM API. That's pretty much what I have in mind. I think it's likely that if we end up with,

Re: [HACKERS] Pluggable storage

2017-07-19 Thread Haribabu Kommi
On Sat, Jul 15, 2017 at 12:30 PM, Robert Haas wrote: > On Fri, Jul 14, 2017 at 8:35 AM, Haribabu Kommi > wrote: > > To replace tuple with slot, I took trigger and SPI calls as the first > step > > in modifying > > from tuple to slot, Here I

Re: [HACKERS] Pluggable storage

2017-07-17 Thread Peter Geoghegan
On Mon, Jul 17, 2017 at 1:24 PM, Alexander Korotkov wrote: > It's probably depends on particular storage (once we have pluggable > storages). Some storages would have additional level of indirection while > others wouldn't. Agreed. Like kill_prior_tuple, it's an

Re: [HACKERS] Pluggable storage

2017-07-17 Thread Alexander Korotkov
On Mon, Jul 17, 2017 at 7:51 PM, Peter Geoghegan wrote: > On Mon, Jul 17, 2017 at 3:22 AM, Alexander Korotkov > wrote: > > I think that "retail index tuple deletion" is the feature which could > give > > us some advantages even independently from

Re: [HACKERS] Pluggable storage

2017-07-17 Thread Peter Geoghegan
On Mon, Jul 17, 2017 at 3:22 AM, Alexander Korotkov wrote: > I think that "retail index tuple deletion" is the feature which could give > us some advantages even independently from pluggable storages. For example, > imagine very large table with only small amount of

Re: [HACKERS] Pluggable storage

2017-07-17 Thread Alexander Korotkov
On Sun, Jul 16, 2017 at 3:58 AM, Peter Geoghegan wrote: > I strongly agree. I simply don't understand how you can adopt UNDO for > MVCC, and yet expect to get a benefit commensurate with the effort > without also implementing "retail index tuple deletion" first. > Pursuing UNDO

Re: [HACKERS] Pluggable storage

2017-07-15 Thread Peter Geoghegan
On Sat, Jul 15, 2017 at 3:36 PM, Alexander Korotkov wrote: > I think that pruning and vacuum are artifacts of not only current heap > formats, but they are also artifacts of current index AM API. And this is > more significant circumstance given that we're going to

Re: [HACKERS] Pluggable storage

2017-07-15 Thread Alexander Korotkov
On Sat, Jul 15, 2017 at 5:14 AM, Robert Haas wrote: > On Thu, Jun 22, 2017 at 9:30 AM, Alexander Korotkov > wrote: > > If #1 is only about changing tuple and page formats, then could be much > > simpler than the patch upthread? We can implement

Re: [HACKERS] Pluggable storage

2017-07-14 Thread Robert Haas
On Fri, Jul 14, 2017 at 8:35 AM, Haribabu Kommi wrote: > To replace tuple with slot, I took trigger and SPI calls as the first step > in modifying > from tuple to slot, Here I attached a WIP patch. The notable changes are, > > 1. Replace most of the HeapTuple with Slot

Re: [HACKERS] Pluggable storage

2017-07-14 Thread Robert Haas
On Thu, Jun 22, 2017 at 4:27 PM, Tomas Vondra wrote: > Can you elaborate a bit more about this TID bit pattern issues? I do > remember that not all TIDs are valid due to safeguards on individual fields, > like for example > > Assert(iptr->ip_posid < (1 <<

Re: [HACKERS] Pluggable storage

2017-07-14 Thread Robert Haas
On Thu, Jun 22, 2017 at 9:30 AM, Alexander Korotkov wrote: > If #1 is only about changing tuple and page formats, then could be much > simpler than the patch upthread? We can implement "page format access > methods" with routines for insertion, update, pruning and

Re: [HACKERS] Pluggable storage

2017-06-28 Thread Amit Kapila
On Wed, Jun 28, 2017 at 7:43 AM, Haribabu Kommi wrote: > > On Wed, Jun 28, 2017 at 12:00 AM, Alexander Korotkov > wrote: >> >> On Tue, Jun 27, 2017 at 4:19 PM, Amit Kapila >> wrote: >>> >>> On Thu, Jun 22, 2017 at

Re: [HACKERS] Pluggable storage

2017-06-28 Thread Amit Kapila
On Tue, Jun 27, 2017 at 7:23 PM, Alexander Korotkov wrote: > On Tue, Jun 27, 2017 at 4:08 PM, Amit Kapila > wrote: >> >> Similarly, >> if the page format is changed you need to consider all page scan API's >> like

Re: [HACKERS] Pluggable storage

2017-06-27 Thread Haribabu Kommi
On Tue, Jun 27, 2017 at 11:53 PM, Alexander Korotkov < a.korot...@postgrespro.ru> wrote: > On Tue, Jun 27, 2017 at 4:08 PM, Amit Kapila > wrote: > >> On Thu, Jun 22, 2017 at 8:00 PM, Alexander Korotkov >> wrote: >> > On Wed, Jun 21, 2017 at

Re: [HACKERS] Pluggable storage

2017-06-27 Thread Haribabu Kommi
On Wed, Jun 28, 2017 at 12:00 AM, Alexander Korotkov < a.korot...@postgrespro.ru> wrote: > On Tue, Jun 27, 2017 at 4:19 PM, Amit Kapila > wrote: > >> On Thu, Jun 22, 2017 at 5:46 PM, Alexander Korotkov >> wrote: >> > On Tue, Jun 13, 2017 at

Re: [HACKERS] Pluggable storage

2017-06-27 Thread Haribabu Kommi
On Thu, Jun 22, 2017 at 5:47 AM, Robert Haas wrote: > On Mon, Jun 12, 2017 at 9:50 PM, Haribabu Kommi > wrote: > > Open Items: > > > > 1. The BitmapHeapScan and TableSampleScan are tightly coupled with > > HeapTuple and HeapScanDesc, So these

Re: [HACKERS] Pluggable storage

2017-06-27 Thread Alexander Korotkov
On Tue, Jun 27, 2017 at 4:19 PM, Amit Kapila wrote: > On Thu, Jun 22, 2017 at 5:46 PM, Alexander Korotkov > wrote: > > On Tue, Jun 13, 2017 at 4:50 AM, Haribabu Kommi < > kommi.harib...@gmail.com> > > wrote: > >> > >> On Fri, Oct 14, 2016 at

Re: [HACKERS] Pluggable storage

2017-06-27 Thread Alexander Korotkov
On Tue, Jun 27, 2017 at 4:08 PM, Amit Kapila wrote: > On Thu, Jun 22, 2017 at 8:00 PM, Alexander Korotkov > wrote: > > On Wed, Jun 21, 2017 at 10:47 PM, Robert Haas > wrote: > >> > >> On Mon, Jun 12, 2017 at 9:50 PM,

Re: [HACKERS] Pluggable storage

2017-06-27 Thread Amit Kapila
On Thu, Jun 22, 2017 at 5:46 PM, Alexander Korotkov wrote: > On Tue, Jun 13, 2017 at 4:50 AM, Haribabu Kommi > wrote: >> >> On Fri, Oct 14, 2016 at 7:26 AM, Alvaro Herrera >> wrote: >>> >>> I have sent the partial

Re: [HACKERS] Pluggable storage

2017-06-27 Thread Amit Kapila
On Thu, Jun 22, 2017 at 8:00 PM, Alexander Korotkov wrote: > On Wed, Jun 21, 2017 at 10:47 PM, Robert Haas wrote: >> >> On Mon, Jun 12, 2017 at 9:50 PM, Haribabu Kommi >> wrote: >> > Open Items: >> > >> > 1. The

Re: [HACKERS] Pluggable storage

2017-06-26 Thread Alexander Korotkov
On Mon, Jun 26, 2017 at 10:55 PM, Tomas Vondra wrote: > On 06/26/2017 05:18 PM, Alexander Korotkov wrote: > >> I see that design question for PostgreSQL pluggable storages is very hard. >> > > IMHO it's mostly expected to be hard. > > Firstly, PostgreSQL is a

Re: [HACKERS] Pluggable storage

2017-06-26 Thread Tomas Vondra
Hi, On 06/26/2017 05:18 PM, Alexander Korotkov wrote: Hackers, I see that design question for PostgreSQL pluggable storages is very hard. IMHO it's mostly expected to be hard. Firstly, PostgreSQL is a mature product with many advanced features, and reworking a low-level feature without

Re: [HACKERS] Pluggable storage

2017-06-26 Thread Alexander Korotkov
Hackers, I see that design question for PostgreSQL pluggable storages is very hard. BTW, I think it worth analyzing existing use-cases of pluggable storages. I think that the most famous DBMS with pluggable storage API is MySQL. This why I decided to start with it. I've added MySQL/MariaDB

Re: [HACKERS] Pluggable storage

2017-06-25 Thread Julien Rouhaud
On 23/06/2017 16:07, Tomas Vondra wrote: > > I think you're probably right - GIN does compress the posting lists by > exploiting the TID redundancy (that it's page/offset structure), and I > don't think there are other index AMs doing that. > > But I'm not sure we can simply rely on that - it's

Re: [HACKERS] Pluggable storage

2017-06-23 Thread Tom Lane
Tomas Vondra writes: > It would be really great if you could explain why BitmapScans are > dubious, instead of just labeling them as dubious. (I guess you mean > Bitmap Heap Scans, right?) The two things I'm aware of are (a) the assumption you noted, that fetching

Re: [HACKERS] Pluggable storage

2017-06-23 Thread Tomas Vondra
Hi, On 6/23/17 10:38 AM, Teodor Sigaev wrote: 1. Table AM with a 6-byte TID. 2. Table AM with a custom locator format, which could be TID-like. 3. Table AM with no locators. Currently TID has its own type in system catalog. Seems, it's possible that storage claims type of TID which it uses.

Re: [HACKERS] Pluggable storage

2017-06-23 Thread Teodor Sigaev
1. Table AM with a 6-byte TID. 2. Table AM with a custom locator format, which could be TID-like. 3. Table AM with no locators. Currently TID has its own type in system catalog. Seems, it's possible that storage claims type of TID which it uses. Then, AM could claim it too, so the core based

Re: [HACKERS] Pluggable storage

2017-06-22 Thread Michael Paquier
On Thu, Jun 22, 2017 at 11:27 PM, Robert Haas wrote: > On Thu, Jun 22, 2017 at 8:32 AM, Alexander Korotkov > wrote: >> On Thu, Jun 22, 2017 at 4:01 AM, Michael Paquier >> wrote: >>> Putting that in a couple of words.

Re: [HACKERS] Pluggable storage

2017-06-22 Thread Tomas Vondra
Hi, On 6/22/17 4:36 PM, Alexander Korotkov wrote: On Thu, Jun 22, 2017 at 5:27 PM, Robert Haas > wrote: On Thu, Jun 22, 2017 at 8:32 AM, Alexander Korotkov > wrote: >

Re: [HACKERS] Pluggable storage

2017-06-22 Thread Tomas Vondra
Hi, On 6/21/17 9:47 PM, Robert Haas wrote: ... > like int8 or numeric, it won't work at all. Even for other things, it's going to cause problems because the bit patterns won't be what the code is expecting; e.g. bitmap scans care about the structure of the TID, not just how many bits it is.

Re: [HACKERS] Pluggable storage

2017-06-22 Thread Alexander Korotkov
On Thu, Jun 22, 2017 at 5:27 PM, Robert Haas wrote: > On Thu, Jun 22, 2017 at 8:32 AM, Alexander Korotkov > wrote: > > On Thu, Jun 22, 2017 at 4:01 AM, Michael Paquier < > michael.paqu...@gmail.com> > > wrote: > >> Putting that in a couple of

Re: [HACKERS] Pluggable storage

2017-06-22 Thread Alexander Korotkov
On Wed, Jun 21, 2017 at 10:47 PM, Robert Haas wrote: > On Mon, Jun 12, 2017 at 9:50 PM, Haribabu Kommi > wrote: > > Open Items: > > > > 1. The BitmapHeapScan and TableSampleScan are tightly coupled with > > HeapTuple and HeapScanDesc, So these

Re: [HACKERS] Pluggable storage

2017-06-22 Thread Robert Haas
On Thu, Jun 22, 2017 at 8:32 AM, Alexander Korotkov wrote: > On Thu, Jun 22, 2017 at 4:01 AM, Michael Paquier > wrote: >> Putting that in a couple of words. >> 1. Table AM with a 6-byte TID. >> 2. Table AM with a custom locator format, which

Re: [HACKERS] Pluggable storage

2017-06-22 Thread Alexander Korotkov
On Thu, Jun 22, 2017 at 4:01 AM, Michael Paquier wrote: > Putting that in a couple of words. > 1. Table AM with a 6-byte TID. > 2. Table AM with a custom locator format, which could be TID-like. > 3. Table AM with no locators. > > Getting into having #1 first to work

Re: [HACKERS] Pluggable storage

2017-06-22 Thread Alexander Korotkov
On Tue, Jun 13, 2017 at 4:50 AM, Haribabu Kommi wrote: > On Fri, Oct 14, 2016 at 7:26 AM, Alvaro Herrera > wrote: > >> I have sent the partial patch I have to Hari Babu Kommi. We expect that >> he will be able to further this goal some more.

Re: [HACKERS] Pluggable storage

2017-06-21 Thread Michael Paquier
On Thu, Jun 22, 2017 at 11:12 AM, Amit Langote wrote: > On 2017/06/22 10:01, Michael Paquier wrote: >> #3 implies that the index AM logic is implemented in the table >> AM. Not saying that it is not useful, but it does not feel natural to >> have the planner request

Re: [HACKERS] Pluggable storage

2017-06-21 Thread Amit Langote
On 2017/06/22 10:01, Michael Paquier wrote: > #3 implies that the index AM logic is implemented in the table > AM. Not saying that it is not useful, but it does not feel natural to > have the planner request for a sequential scan, just to have the table > AM secretly do some kind of index/skipping

Re: [HACKERS] Pluggable storage

2017-06-21 Thread Michael Paquier
On Thu, Jun 22, 2017 at 4:47 AM, Robert Haas wrote: > I think that BitmapHeapScan, at least, is applicable to any table AM > that has TIDs. It seems to me that in general we can imagine three > kinds of table AMs: > > 1. Table AMs where a tuple can be efficiently located

Re: [HACKERS] Pluggable storage

2017-06-21 Thread Robert Haas
On Mon, Jun 12, 2017 at 9:50 PM, Haribabu Kommi wrote: > Open Items: > > 1. The BitmapHeapScan and TableSampleScan are tightly coupled with > HeapTuple and HeapScanDesc, So these scans are directly operating > on those structures and providing the result. > > These scan

Re: [HACKERS] Pluggable storage

2016-10-13 Thread Alvaro Herrera
I have sent the partial patch I have to Hari Babu Kommi. We expect that he will be able to further this goal some more. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Pluggable storage

2016-08-24 Thread Alvaro Herrera
Alvaro Herrera wrote: > Many have expressed their interest in this topic, but I haven't seen any > design of how it should work. Here's my attempt; I've been playing with > this for some time now and I think what I propose here is a good initial > plan. I regret to announce that I'll have to

Re: [HACKERS] Pluggable storage

2016-08-18 Thread Andres Freund
On 2016-08-18 08:58:11 +0100, Simon Riggs wrote: > On 16 August 2016 at 19:46, Andres Freund wrote: > > On 2016-08-15 12:02:18 -0400, Robert Haas wrote: > >> Thanks for taking a stab at this. I'd like to throw out a few concerns. > >> > >> One, I'm worried that adding an

Re: [HACKERS] Pluggable storage

2016-08-18 Thread Andres Freund
On August 18, 2016 7:44:50 AM PDT, Ants Aasma wrote: >On Tue, Aug 16, 2016 at 9:46 PM, Andres Freund >wrote: >> On 2016-08-15 12:02:18 -0400, Robert Haas wrote: >>> I am somewhat inclined to >>> believe that we need to restructure the executor in a

Re: [HACKERS] Pluggable storage

2016-08-18 Thread Ants Aasma
On Tue, Aug 16, 2016 at 9:46 PM, Andres Freund wrote: > On 2016-08-15 12:02:18 -0400, Robert Haas wrote: >> I am somewhat inclined to >> believe that we need to restructure the executor in a bigger way so >> that it passes around datums instead of tuples; I'm inclined to >>

Re: [HACKERS] Pluggable storage

2016-08-18 Thread Amit Kapila
On Wed, Aug 17, 2016 at 10:33 PM, Alvaro Herrera wrote: > Anastasia Lubennikova wrote: >> >> Except these, there are some pretty strange and unrelated functions in >> src/backend/catalog. >> I'm willing to fix them, but I'd like to synchronize our efforts. > > I very

Re: [HACKERS] Pluggable storage

2016-08-18 Thread Alexander Korotkov
On Thu, Aug 18, 2016 at 10:58 AM, Simon Riggs wrote: > On 16 August 2016 at 19:46, Andres Freund wrote: > > On 2016-08-15 12:02:18 -0400, Robert Haas wrote: > >> Thanks for taking a stab at this. I'd like to throw out a few concerns. > >> > >> One,

Re: [HACKERS] Pluggable storage

2016-08-18 Thread Simon Riggs
On 16 August 2016 at 19:46, Andres Freund wrote: > On 2016-08-15 12:02:18 -0400, Robert Haas wrote: >> Thanks for taking a stab at this. I'd like to throw out a few concerns. >> >> One, I'm worried that adding an additional layer of pointer-jumping is >> going to slow things

Re: [HACKERS] Pluggable storage

2016-08-17 Thread Alvaro Herrera
Anastasia Lubennikova wrote: > 13.08.2016 02:15, Alvaro Herrera: > >To support this, we introduce StorageTuple and StorageScanDesc. > >StorageTuples represent a physical tuple coming from some storage AM. > >It is necessary to have a pointer to a StorageAmRoutine in order to > >manipulate the

Re: [HACKERS] Pluggable storage

2016-08-17 Thread Anastasia Lubennikova
13.08.2016 02:15, Alvaro Herrera: Many have expressed their interest in this topic, but I haven't seen any design of how it should work. Here's my attempt; I've been playing with this for some time now and I think what I propose here is a good initial plan. This will allow us to write

Re: [HACKERS] Pluggable storage

2016-08-16 Thread Andres Freund
On 2016-08-15 12:02:18 -0400, Robert Haas wrote: > Thanks for taking a stab at this. I'd like to throw out a few concerns. > > One, I'm worried that adding an additional layer of pointer-jumping is > going to slow things down and make Andres' work to speed up the > executor more difficult. I

Re: [HACKERS] Pluggable storage

2016-08-15 Thread Anastasia Lubennikova
13.08.2016 02:15, Alvaro Herrera: Many have expressed their interest in this topic, but I haven't seen any design of how it should work. Here's my attempt; I've been playing with this for some time now and I think what I propose here is a good initial plan. This will allow us to write

Re: [HACKERS] Pluggable storage

2016-08-15 Thread Robert Haas
On Fri, Aug 12, 2016 at 7:15 PM, Alvaro Herrera wrote: > Many have expressed their interest in this topic, but I haven't seen any > design of how it should work. Here's my attempt; I've been playing with > this for some time now and I think what I propose here is a good

Re: [HACKERS] Pluggable storage

2016-08-15 Thread Alexander Korotkov
On Sat, Aug 13, 2016 at 2:15 AM, Alvaro Herrera wrote: > Many have expressed their interest in this topic, but I haven't seen any > design of how it should work. That's it. My design presented at PGCon was very sketchy, and I didn't deliver any prototype yet. >

[HACKERS] Pluggable storage

2016-08-12 Thread Alvaro Herrera
Many have expressed their interest in this topic, but I haven't seen any design of how it should work. Here's my attempt; I've been playing with this for some time now and I think what I propose here is a good initial plan. This will allow us to write permanent table storage that works