Re: Zedstore - compressed in-core columnar storage

2021-07-12 Thread Andy Fan
Greetings. Thanks for the project. I see the code in github has not been updated for a long time, is it still in active development? Thanks -- Best Regards Andy Fan (https://www.aliyun.com/)

Re: Zedstore - compressed in-core columnar storage

2021-01-07 Thread Robert Eckhardt
On Dec 31, 2020, at 9:22 AM, Simon Riggs mailto:si...@2ndquadrant.com>> wrote: On Wed, 18 Nov 2020 at 00:31, Jacob Champion mailto:pchamp...@vmware.com>> wrote: So that's in need of resolution. I'd expect gin and gist to be pretty flaky until we fix that. Jacob and Soumyadeep, Thanks for

Re: Zedstore - compressed in-core columnar storage

2020-12-31 Thread Simon Riggs
On Wed, 18 Nov 2020 at 00:31, Jacob Champion wrote: > > So that's in need of resolution. I'd expect gin and gist to be pretty > flaky until we fix that. Jacob and Soumyadeep, Thanks for submitting this. I think a fix is still outstanding? and the patch fails to apply on HEAD in two places.

Re: Zedstore - compressed in-core columnar storage

2020-11-17 Thread Jacob Champion
On Nov 13, 2020, at 2:00 PM, Tomas Vondra wrote: > > Fedora 32, nothing special. I'm not sure if I ran the tests with pglz or > lz4, maybe there's some dependence on that, but it does fail for me > quite reliably with this: > > ./configure --enable-debug --enable-cassert --enable-tap-tests >

Re: Zedstore - compressed in-core columnar storage

2020-11-16 Thread Merlin Moncure
On Mon, Nov 16, 2020 at 10:07 AM Tomas Vondra wrote: > > > On 11/16/20 1:59 PM, Merlin Moncure wrote: > > On Thu, Nov 12, 2020 at 4:40 PM Tomas Vondra > > wrote: > >>masterzedstore/pglzzedstore/lz4 > >> - > >>copy

Re: Zedstore - compressed in-core columnar storage

2020-11-16 Thread Tomas Vondra
On 11/16/20 1:59 PM, Merlin Moncure wrote: > On Thu, Nov 12, 2020 at 4:40 PM Tomas Vondra > wrote: >>masterzedstore/pglzzedstore/lz4 >> - >>copy 1855680922131 >>dump 751

Re: Zedstore - compressed in-core columnar storage

2020-11-16 Thread Merlin Moncure
On Thu, Nov 12, 2020 at 4:40 PM Tomas Vondra wrote: >masterzedstore/pglzzedstore/lz4 > - >copy 1855680922131 >dump 751 905 811 > > And the size of the lineitem

Re: Zedstore - compressed in-core columnar storage

2020-11-13 Thread Tomas Vondra
On 11/13/20 8:07 PM, Jacob Champion wrote: > On Nov 12, 2020, at 2:40 PM, Tomas Vondra > wrote: >> >> Hi, >> >> Thanks for the updated patch. It's a quite massive amount of code - I I >> don't think we had many 2MB patches in the past, so this is by no means >> a full review. > > Thanks for

Re: Zedstore - compressed in-core columnar storage

2020-11-13 Thread Jacob Champion
On Nov 12, 2020, at 2:40 PM, Tomas Vondra wrote: > > Hi, > > Thanks for the updated patch. It's a quite massive amount of code - I I > don't think we had many 2MB patches in the past, so this is by no means > a full review. Thanks for taking a look! You're not kidding about the patch size.

Re: Zedstore - compressed in-core columnar storage

2020-11-12 Thread Tomas Vondra
Hi, Thanks for the updated patch. It's a quite massive amount of code - I I don't think we had many 2MB patches in the past, so this is by no means a full review. 1) the psql_1.out is missing a bit of expected output (due to 098fb0079) 2) I'm getting crashes in intarray contrib, due to hitting

Re: Zedstore - compressed in-core columnar storage

2020-02-14 Thread Soumyadeep Chakraborty
Hello, We (David and I) recently observed that a Zedstore table can be considerably bloated when we load data into it with concurrent copies. Also, we found that concurrent insert performance was less than desirable. This is a detailed analysis of the extent of the problem, the cause of the

Re: Zedstore - compressed in-core columnar storage

2019-11-04 Thread Taylor Vesely
> When a zedstore table is queried using *invalid* ctid, the server > crashes due to assertion failure. See below, > > postgres=# select * from t1 where ctid = '(0, 0)'; > server closed the connection unexpectedly > This probably means the server terminated abnormally > before or while processing

Re: Zedstore - compressed in-core columnar storage

2019-10-30 Thread Taylor Vesely
Alex Wang and I have been doing some performance analysis of the most recent version of the zedstore branch, and have some interesting statistics to share. We specifically focused on TPC-DS query 2, because it plays to what should be the strength of zedstore- namely it does a full table scan of

Re: Zedstore - compressed in-core columnar storage

2019-10-28 Thread Taylor Vesely
> When a zedstore table is queried using *invalid* ctid, the server > crashes due to assertion failure. See below, > > postgres=# select * from t2 where ctid = '(0, 0)'; > server closed the connection unexpectedly > This probably means the server terminated abnormally > before or while processing

Re: Zedstore - compressed in-core columnar storage

2019-10-24 Thread Ashutosh Sharma
On Thu, Oct 17, 2019 at 2:11 PM Heikki Linnakangas wrote: > > On 15/10/2019 13:49, Ashutosh Sharma wrote: > > Hi, > > > > I got chance to spend some time looking into the recent changes done > > in the zedstore code, basically the functions for packing datums into > > the attribute streams and

Re: Zedstore - compressed in-core columnar storage

2019-10-17 Thread Heikki Linnakangas
On 15/10/2019 13:49, Ashutosh Sharma wrote: Hi, I got chance to spend some time looking into the recent changes done in the zedstore code, basically the functions for packing datums into the attribute streams and handling attribute leaf pages. I didn't find any issues but there are some minor

Re: Zedstore - compressed in-core columnar storage

2019-10-15 Thread Ashutosh Sharma
Hi, I got chance to spend some time looking into the recent changes done in the zedstore code, basically the functions for packing datums into the attribute streams and handling attribute leaf pages. I didn't find any issues but there are some minor comments that I found when reviewing. I have

Re: Zedstore - compressed in-core columnar storage

2019-09-30 Thread Ashutosh Sharma
On Fri, Sep 27, 2019 at 3:09 PM Alexandra Wang wrote: > > Hi Ashutosh, > > Sorry I indeed missed your question, thanks for the reminder! > > On Wed, Sep 25, 2019 at 4:10 AM Ashutosh Sharma wrote: >> >> > Further, the UPDATE operation on zedstore table is very slow. I think >> > that's because in

Re: Zedstore - compressed in-core columnar storage

2019-09-27 Thread Alexandra Wang
Hi Ashutosh, Sorry I indeed missed your question, thanks for the reminder! On Wed, Sep 25, 2019 at 4:10 AM Ashutosh Sharma wrote: > > Further, the UPDATE operation on zedstore table is very slow. I think > > that's because in case of zedstore table we have to update all the > > btree data

Re: Zedstore - compressed in-core columnar storage

2019-09-25 Thread Ashutosh Sharma
Hi Alexandra, On Tue, Sep 17, 2019 at 4:45 PM Ashutosh Sharma wrote: > > On Thu, Aug 29, 2019 at 5:39 PM Heikki Linnakangas wrote: > > > > On 29/08/2019 14:30, Ashutosh Sharma wrote: > > > > > > On Wed, Aug 28, 2019 at 5:30 AM Alexandra Wang > > > wrote: > > Further,

Re: Zedstore - compressed in-core columnar storage

2019-09-19 Thread Ashutosh Sharma
On Fri, Sep 20, 2019 at 5:48 AM Taylor Vesely wrote: > > > When doing update operation, for each tuple being modified, > > *tuplebuffers_insert()* says that there is no entry for the relation > > being modified in the hash table although it was already added when > > the first tuple in the table

Re: Zedstore - compressed in-core columnar storage

2019-09-19 Thread Taylor Vesely
> When doing update operation, for each tuple being modified, > *tuplebuffers_insert()* says that there is no entry for the relation > being modified in the hash table although it was already added when > the first tuple in the table was updated. Why is it so? Currently, when doing an update, it

Re: Zedstore - compressed in-core columnar storage

2019-09-19 Thread Ashutosh Sharma
On Thu, Sep 19, 2019 at 11:35 AM Ashutosh Sharma wrote: > > On Thu, Sep 19, 2019 at 8:10 AM Alexandra Wang wrote: > > > > On Tue, Sep 17, 2019 at 4:15 AM Ashutosh Sharma > > wrote: > >> > >> create table t1(a int, b int) using zedstore; > >> insert into t1 select i, i+10 from

Re: Zedstore - compressed in-core columnar storage

2019-09-18 Thread Alexandra Wang
On Tue, Sep 17, 2019 at 4:15 AM Ashutosh Sharma wrote: > create table t1(a int, b int) using zedstore; > insert into t1 select i, i+10 from generate_series(1, 100) i; > postgres=# update t1 set b = 200; > server closed the connection unexpectedly > This probably means the server terminated

Re: Zedstore - compressed in-core columnar storage

2019-09-17 Thread Ashutosh Sharma
On Thu, Aug 29, 2019 at 5:39 PM Heikki Linnakangas wrote: > > On 29/08/2019 14:30, Ashutosh Sharma wrote: > > > > On Wed, Aug 28, 2019 at 5:30 AM Alexandra Wang > > wrote: > > > > You are correct that we currently go through each item in the leaf > > page that >

Re: Zedstore - compressed in-core columnar storage

2019-08-29 Thread Heikki Linnakangas
On 29/08/2019 14:30, Ashutosh Sharma wrote: On Wed, Aug 28, 2019 at 5:30 AM Alexandra Wang > wrote: You are correct that we currently go through each item in the leaf page that contains the given tid, specifically, the logic to retrieve all the

Re: Zedstore - compressed in-core columnar storage

2019-08-29 Thread Ashutosh Sharma
On Wed, Aug 28, 2019 at 5:30 AM Alexandra Wang wrote: > You are correct that we currently go through each item in the leaf page > that > contains the given tid, specifically, the logic to retrieve all the > attribute > items inside a ZSAttStream is now moved to decode_attstream() in the latest >

Re: Zedstore - compressed in-core columnar storage

2019-08-27 Thread Alexandra Wang
On Tue, Aug 27, 2019 at 12:03 AM Ashutosh Sharma wrote: > My point is, once we find the leaf page containing the given tid, we go > through each item in the page until we find the data corresponding to the > given tid which means we kind of perform a sequential scan at the page > level. I'm

Re: Zedstore - compressed in-core columnar storage

2019-08-27 Thread Ashutosh Sharma
On Tue, Aug 27, 2019 at 6:03 AM Ashwin Agrawal wrote: > Hope that helps to clarify the confusion. > Thanks for the explanation. Yes, it does clarify my doubt to some extent. My point is, once we find the leaf page containing the given tid, we go through each item in the page until we find the

Re: Zedstore - compressed in-core columnar storage

2019-08-26 Thread Ashwin Agrawal
On Mon, Aug 26, 2019 at 5:36 AM Ashutosh Sharma wrote: > Thanks Ashwin and Heikki for your responses. I've one more query here, > > If BTree index is created on a zedstore table, the t_tid field of > Index tuple contains the physical tid that is not actually pointing to > the data block instead

Re: Zedstore - compressed in-core columnar storage

2019-08-26 Thread Ashutosh Sharma
Thanks Ashwin and Heikki for your responses. I've one more query here, If BTree index is created on a zedstore table, the t_tid field of Index tuple contains the physical tid that is not actually pointing to the data block instead it contains something from which the logical tid can be derived.

Re: Zedstore - compressed in-core columnar storage

2019-08-20 Thread Heikki Linnakangas
On 20/08/2019 05:04, Justin Pryzby wrote: it looks like zedstore with lz4 gets ~4.6x for our largest customer's largest table. zfs using compress=gzip-1 gives 6x compression across all their partitioned tables, and I'm surprised it beats zedstore . I did a quick test, with 10

Re: Zedstore - compressed in-core columnar storage

2019-08-19 Thread Justin Pryzby
On Mon, Aug 19, 2019 at 04:15:30PM -0700, Alexandra Wang wrote: > On Sun, Aug 18, 2019 at 12:35 PM Justin Pryzby wrote: > > > . I was missing a way to check for compression ratio; > > Here are the ways to check compression ratio for zedstore: > > Table level: > SELECT

Re: Zedstore - compressed in-core columnar storage

2019-08-19 Thread Alexandra Wang
On Sun, Aug 18, 2019 at 12:35 PM Justin Pryzby wrote: > > . I was missing a way to check for compression ratio; Here are the ways to check compression ratio for zedstore: Table level: select sum(uncompressedsz::numeric) / sum(totalsz) as compratio from pg_zs_btree_pages(); Per column level:

Re: Zedstore - compressed in-core columnar storage

2019-08-18 Thread Justin Pryzby
On Thu, Aug 15, 2019 at 01:05:49PM +0300, Heikki Linnakangas wrote: > We've continued hacking on Zedstore, here's a new patch version against > current PostgreSQL master (commit f1bf619acdf). If you want to follow the > development in real-time, we're working on this branch: >

Re: Zedstore - compressed in-core columnar storage

2019-08-15 Thread Heikki Linnakangas
On 14/08/2019 20:32, Ashwin Agrawal wrote: On Wed, Aug 14, 2019 at 2:51 AM Ashutosh Sharma wrote: 2) Is there a chance that IndexOnlyScan would ever be required for zedstore tables considering the design approach taken for it? We have not given much thought to IndexOnlyScans so far. But I

Re: Zedstore - compressed in-core columnar storage

2019-08-14 Thread Ashwin Agrawal
On Wed, Aug 14, 2019 at 2:51 AM Ashutosh Sharma wrote: > Hi Ashwin, > > I tried playing around with the zedstore code a bit today and there > are couple questions that came into my mind. > Great! Thank You. > > 1) Can zedstore tables be vacuumed? If yes, does VACUUM on zedstore > table set

Re: Zedstore - compressed in-core columnar storage

2019-08-14 Thread Ashutosh Sharma
Hi Ashwin, I tried playing around with the zedstore code a bit today and there are couple questions that came into my mind. 1) Can zedstore tables be vacuumed? If yes, does VACUUM on zedstore table set the VM bits associated with it. 2) Is there a chance that IndexOnlyScan would ever be

Re: Zedstore - compressed in-core columnar storage

2019-07-01 Thread Ashwin Agrawal
On Thu, May 30, 2019 at 8:07 AM DEV_OPS wrote: > > it's really cool and very good progress, > > I'm interesting if SIDM/JIT will be supported > That's something outside of Zedstore work directly at least now. The intent is to work with current executor code or enhance it only wherever needed.

Re: Zedstore - compressed in-core columnar storage

2019-07-01 Thread Ashwin Agrawal
On Sun, Jun 30, 2019 at 7:59 PM Tsunakawa, Takayuki < tsunakawa.ta...@jp.fujitsu.com> wrote: > From: Ashwin Agrawal [mailto:aagra...@pivotal.io] > > The objective is to gather feedback on design and approach to the same. > > The implementation has core basic pieces working but not close to >

RE: Zedstore - compressed in-core columnar storage

2019-06-30 Thread Tsunakawa, Takayuki
From: Ashwin Agrawal [mailto:aagra...@pivotal.io] > The objective is to gather feedback on design and approach to the same. > The implementation has core basic pieces working but not close to complete. Thank you for proposing a very interesting topic. Are you thinking of including this in

Re: Zedstore - compressed in-core columnar storage

2019-05-30 Thread DEV_OPS
it's really cool and very good progress,  I'm interesting if SIDM/JIT will be supported best wishes TY On 2019/5/23 08:07, Ashwin Agrawal wrote: > We (Heikki, me and Melanie) are continuing to build Zedstore. Wish to > share the recent additions and modifications. Attaching a patch > with

Re: Zedstore - compressed in-core columnar storage

2019-05-24 Thread Mark Kirkwood
On 23/05/19 12:07 PM, Ashwin Agrawal wrote: We (Heikki, me and Melanie) are continuing to build Zedstore. Wish to share the recent additions and modifications. Attaching a patch with the latest code. Link to github branch [1] to follow along. The approach we have been leaning towards is to

Re: Zedstore - compressed in-core columnar storage

2019-05-24 Thread Ashwin Agrawal
On Thu, May 23, 2019 at 7:30 PM Ajin Cherian wrote: > Hi Ashwin, > > - how to pass the "column projection list" to table AM? (as stated in > initial email, currently we have modified table am API to pass the > projection to AM) > > We were working on a similar columnar storage using

Re: Zedstore - compressed in-core columnar storage

2019-05-23 Thread Ajin Cherian
Hi Ashwin, - how to pass the "column projection list" to table AM? (as stated in initial email, currently we have modified table am API to pass the projection to AM) We were working on a similar columnar storage using pluggable APIs; one idea that we thought of was to modify the scan slot

Re: Zedstore - compressed in-core columnar storage

2019-04-25 Thread Heikki Linnakangas
On 15/04/2019 22:32, Peter Geoghegan wrote: On Mon, Apr 15, 2019 at 12:19 PM Tom Lane wrote: Perhaps, but we won't know if we don't try. I think we should try, and be willing to add hooks and flexibility to core as needed to make it possible. We could approach it without taking a firm

Re: Zedstore - compressed in-core columnar storage

2019-04-24 Thread Ashwin Agrawal
On Tue, Apr 16, 2019 at 9:15 AM Tomas Vondra wrote: > > I'm not sure it's that clear cut, actually. Sure, it's not the usual > (block,item) pair so it's not possible to jump to the exact location, so > it's not the raw physical identifier as regular TID. But the data are > organized in a btree,

Re: Zedstore - compressed in-core columnar storage

2019-04-16 Thread Tomas Vondra
On Mon, Apr 15, 2019 at 10:45:51PM -0700, Ashwin Agrawal wrote: On Mon, Apr 15, 2019 at 12:50 PM Peter Geoghegan wrote: On Mon, Apr 15, 2019 at 9:16 AM Ashwin Agrawal wrote: > Would like to know more specifics on this Peter. We may be having different context on hybrid row/column design.

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Ashwin Agrawal
On Mon, Apr 15, 2019 at 12:50 PM Peter Geoghegan wrote: > On Mon, Apr 15, 2019 at 9:16 AM Ashwin Agrawal > wrote: > > Would like to know more specifics on this Peter. We may be having > different context on hybrid row/column design. > > I'm confused about how close your idea of a TID is to the

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Tomas Vondra
On Mon, Apr 15, 2019 at 11:57:49AM -0700, Ashwin Agrawal wrote: On Mon, Apr 15, 2019 at 11:18 AM Tomas Vondra wrote: Maybe. I'm not going to pretend I fully understand the internals. Does that mean the container contains ZSUncompressedBtreeItem as elements? Or just the plain

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Peter Geoghegan
On Mon, Apr 15, 2019 at 1:02 PM Andres Freund wrote: > There's not much of an alternative currently. Indexes require tid > looking things, and as a consequence (and some other comparatively small > changes that'd be required) tableam does too. I'm trying to establish whether or not that's the

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Andres Freund
Hi, On 2019-04-15 12:50:14 -0700, Peter Geoghegan wrote: > On Mon, Apr 15, 2019 at 9:16 AM Ashwin Agrawal wrote: > > Would like to know more specifics on this Peter. We may be having different > > context on hybrid row/column design. > > I'm confused about how close your idea of a TID is to

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Peter Geoghegan
On Mon, Apr 15, 2019 at 9:16 AM Ashwin Agrawal wrote: > Would like to know more specifics on this Peter. We may be having different > context on hybrid row/column design. I'm confused about how close your idea of a TID is to the traditional definition from heapam (and even zheap). If it's a

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Andres Freund
Hi, On 2019-04-15 15:19:41 -0400, Tom Lane wrote: > Peter Geoghegan writes: > > I am also concerned by the broad scope of ZedStore, and I tend to > > agree that it will be difficult to maintain in core. At the same time, > > I think that Andres and Robert are probably right about the difficulty

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Tom Lane
Andres Freund writes: > We also have at times pretty explicitly resisted making crucial pieces > of infrastructure usable outside of core. E.g. because it's legitimately > hard (grammar extensibility), or because we'd some concerns around > stability and the exact approach (WAL - the generic

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Peter Geoghegan
On Mon, Apr 15, 2019 at 12:19 PM Tom Lane wrote: > Perhaps, but we won't know if we don't try. I think we should try, > and be willing to add hooks and flexibility to core as needed to make > it possible. We could approach it without taking a firm position on inclusion in core until the project

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Andres Freund
Hi, On 2019-04-15 14:35:43 -0400, Tom Lane wrote: > Yeah, and that's something I've regretted more than once; I think SP-GiST > is a sterling example of something that isn't nearly useful enough in the > real world to justify the amount of maintenance effort we've been forced > to expend on it.

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Tom Lane
Peter Geoghegan writes: > I am also concerned by the broad scope of ZedStore, and I tend to > agree that it will be difficult to maintain in core. At the same time, > I think that Andres and Robert are probably right about the difficulty > of maintaining it outside of core -- that would be

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Robert Haas
On Mon, Apr 15, 2019 at 2:35 PM Tom Lane wrote: > Robert Haas writes: > > On Mon, Apr 15, 2019 at 11:10 AM Tom Lane wrote: > >> There is a finite limit to how much stuff we can maintain as part of core. > > > I don't agree with that at all. > > Really? Let's have a discussion of how

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Peter Geoghegan
On Mon, Apr 15, 2019 at 11:35 AM Tom Lane wrote: > We do need to limit what we accept into core PG. I do not buy your > argument that users expect everything to be in core. Or more accurately, > the people who do think that way won't be using PG anyway --- they'll > be using MSSQL because it

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Ashwin Agrawal
On Mon, Apr 15, 2019 at 11:18 AM Tomas Vondra wrote: > > Maybe. I'm not going to pretend I fully understand the internals. Does > that mean the container contains ZSUncompressedBtreeItem as elements? Or > just the plain Datum values? > First, your reading of code and all the comments/questions

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Andres Freund
Hi, On 2019-04-15 14:11:02 -0400, Robert Haas wrote: > I hate to pick on any particular part of the tree, but it seems > entirely plausible to me that a second columnar storage implementation > could deliver more incremental value than spgist, an index AM you > committed. We should not move the

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Tom Lane
Robert Haas writes: > On Mon, Apr 15, 2019 at 11:10 AM Tom Lane wrote: >> There is a finite limit to how much stuff we can maintain as part of core. > I don't agree with that at all. Really? Let's have a discussion of how thermodynamics applies to software management sometime. >>> If we have

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Andres Freund
Hi, On 2019-04-15 14:11:02 -0400, Robert Haas wrote: > Furthermore, different table AMs are going to have different > needs. It has already been remarked by both Andres and on this thread > that for columnar storage to really zip along, the executor is going > to need to be much smarter about

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Andres Freund
Hi, On 2019-04-15 11:10:38 -0400, Tom Lane wrote: > Stephen Frost writes: > > * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > >> I think having a colstore in core is important not just for adoption, > >> but also for testing and development of the executor / planner bits. > > > Agreed. >

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Tomas Vondra
On Mon, Apr 15, 2019 at 10:50:21AM -0700, Ashwin Agrawal wrote: On Mon, Apr 15, 2019 at 10:33 AM Tomas Vondra wrote: ... I see. Perhaps it'd be better to call the flag ZSBT_CONTAINER, when it means "this is a container". And then have another flag to track whether the

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Robert Haas
On Mon, Apr 15, 2019 at 11:10 AM Tom Lane wrote: > TBH, I thought the reason we were expending so much effort on a tableam > API was exactly so we *wouldn't* have to include such stuff in core. > > There is a finite limit to how much stuff we can maintain as part of core. > We should embrace the

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Robert Haas
On Sun, Apr 14, 2019 at 12:22 PM Tomas Vondra wrote: > It's not clear to me whether you're arguing for not having any such > implementation in core, or having multiple ones? I think we should aim > to have at least one in-core implementation, even if it's not the best > possible one for all

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Ashwin Agrawal
On Mon, Apr 15, 2019 at 10:33 AM Tomas Vondra wrote: > On Mon, Apr 15, 2019 at 09:29:37AM -0700, Ashwin Agrawal wrote: > > On Sun, Apr 14, 2019 at 9:40 AM Tomas Vondra > >wrote: > > > > On Thu, Apr 11, 2019 at 06:20:47PM +0300, Heikki Linnakangas wrote: > > >On 11/04/2019 17:54,

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Ashwin Agrawal
On Sun, Apr 14, 2019 at 9:40 AM Tomas Vondra wrote: > On Thu, Apr 11, 2019 at 06:20:47PM +0300, Heikki Linnakangas wrote: > >On 11/04/2019 17:54, Tom Lane wrote: > >>Ashwin Agrawal writes: > >>>Thank you for trying it out. Yes, noticed for certain patterns > pg_lzcompress() actually requires

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Ashwin Agrawal
On Sat, Apr 13, 2019 at 4:22 PM Peter Geoghegan wrote: > On Thu, Apr 11, 2019 at 6:06 AM Rafia Sabih > wrote: > > Reading about it reminds me of this work -- TAG column storage( >

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Tom Lane
Stephen Frost writes: > * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: >> I think having a colstore in core is important not just for adoption, >> but also for testing and development of the executor / planner bits. > Agreed. TBH, I thought the reason we were expending so much effort on a

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Tomas Vondra
On Sun, Apr 14, 2019 at 06:39:47PM +0200, Tomas Vondra wrote: On Thu, Apr 11, 2019 at 06:20:47PM +0300, Heikki Linnakangas wrote: On 11/04/2019 17:54, Tom Lane wrote: Ashwin Agrawal writes: Thank you for trying it out. Yes, noticed for certain patterns pg_lzcompress() actually requires much

Re: Zedstore - compressed in-core columnar storage

2019-04-15 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > On Tue, Apr 09, 2019 at 02:29:09PM -0400, Robert Haas wrote: > >On Tue, Apr 9, 2019 at 11:51 AM Alvaro Herrera > >wrote: > >>This is not surprising, considering that columnar store is precisely the > >>reason for starting the

Re: Zedstore - compressed in-core columnar storage

2019-04-14 Thread Tomas Vondra
On Sun, Apr 14, 2019 at 09:45:10AM -0700, Andres Freund wrote: Hi, On 2019-04-14 18:36:18 +0200, Tomas Vondra wrote: I think those comparisons are cute and we did a fair amount of them when considering a drop-in replacement for pglz, but ultimately it might be a bit pointless because: (a) it

Re: Zedstore - compressed in-core columnar storage

2019-04-14 Thread Tomas Vondra
On Mon, Apr 08, 2019 at 05:27:05PM -0700, Ashwin Agrawal wrote: Heikki and I have been hacking recently for few weeks to implement in-core columnar storage for PostgreSQL. Here's the design and initial implementation of Zedstore, compressed in-core columnar storage (table access method

Re: Zedstore - compressed in-core columnar storage

2019-04-14 Thread Andres Freund
Hi, On 2019-04-14 18:36:18 +0200, Tomas Vondra wrote: > I think those comparisons are cute and we did a fair amount of them when > considering a drop-in replacement for pglz, but ultimately it might be a > bit pointless because: > > (a) it very much depends on the dataset (one algorithm may work

Re: Zedstore - compressed in-core columnar storage

2019-04-14 Thread Tomas Vondra
On Thu, Apr 11, 2019 at 06:20:47PM +0300, Heikki Linnakangas wrote: On 11/04/2019 17:54, Tom Lane wrote: Ashwin Agrawal writes: Thank you for trying it out. Yes, noticed for certain patterns pg_lzcompress() actually requires much larger output buffers. Like for one 86 len source it required

Re: Zedstore - compressed in-core columnar storage

2019-04-14 Thread Tomas Vondra
On Thu, Apr 11, 2019 at 04:52:33PM +0300, Konstantin Knizhnik wrote: On 11.04.2019 16:18, Andreas Karlsson wrote: On 4/11/19 10:46 AM, Konstantin Knizhnik wrote: This my results of compressing pbench data using different compressors:

Re: Zedstore - compressed in-core columnar storage

2019-04-14 Thread Tomas Vondra
>>> Heikki and I have been hacking recently for few weeks to implement >>> in-core columnar storage for PostgreSQL. Here's the design and initial >>> implementation of Zedstore, compressed in-core columnar storage (table >>> access method).

Re: Zedstore - compressed in-core columnar storage

2019-04-14 Thread Tomas Vondra
On Tue, Apr 09, 2019 at 02:29:09PM -0400, Robert Haas wrote: On Tue, Apr 9, 2019 at 11:51 AM Alvaro Herrera wrote: This is not surprising, considering that columnar store is precisely the reason for starting the work on table AMs. We should certainly look into integrating some sort of

Re: Zedstore - compressed in-core columnar storage

2019-04-13 Thread Peter Geoghegan
On Thu, Apr 11, 2019 at 6:06 AM Rafia Sabih wrote: > Reading about it reminds me of this work -- TAG column storage( > http://www09.sigmod.org/sigmod/record/issues/0703/03.article-graefe.pdf ). > Isn't this storage system inspired from there, with TID as the TAG? > > It is not referenced here so

Re: Zedstore - compressed in-core columnar storage

2019-04-11 Thread Heikki Linnakangas
On 11/04/2019 17:54, Tom Lane wrote: Ashwin Agrawal writes: Thank you for trying it out. Yes, noticed for certain patterns pg_lzcompress() actually requires much larger output buffers. Like for one 86 len source it required 2296 len output buffer. Current zedstore code doesn’t handle this

Re: Zedstore - compressed in-core columnar storage

2019-04-11 Thread Tom Lane
Ashwin Agrawal writes: > Thank you for trying it out. Yes, noticed for certain patterns > pg_lzcompress() actually requires much larger output buffers. Like for one 86 > len source it required 2296 len output buffer. Current zedstore code doesn’t > handle this case and errors out. LZ4 for same

Re: Zedstore - compressed in-core columnar storage

2019-04-11 Thread Rafia Sabih
On Tue, 9 Apr 2019 at 02:27, Ashwin Agrawal wrote: > Heikki and I have been hacking recently for few weeks to implement > in-core columnar storage for PostgreSQL. Here's the design and initial > implementation of Zedstore, compressed in-core columnar storage (table > access method

Re: Zedstore - compressed in-core columnar storage

2019-04-11 Thread Magnus Hagander
On Thu, Apr 11, 2019 at 3:15 PM Heikki Linnakangas wrote: > On 11/04/2019 16:12, Rafia Sabih wrote: > > On Tue, 9 Apr 2019 at 20:29, Robert Haas > > wrote: > > > > BTW, can I express a small measure of disappointment that the name > for > > the thing under

Re: Zedstore - compressed in-core columnar storage

2019-04-11 Thread Konstantin Knizhnik
On 11.04.2019 16:18, Andreas Karlsson wrote: On 4/11/19 10:46 AM, Konstantin Knizhnik wrote: This my results of compressing pbench data using different compressors: Configuration Size (Gb) Time (sec) no compression 15.31 92 zlib (default level)2.37284 zlib (best

Re: Zedstore - compressed in-core columnar storage

2019-04-11 Thread Andreas Karlsson
On 4/11/19 10:46 AM, Konstantin Knizhnik wrote: This my results of compressing pbench data using different compressors: Configuration Size (Gb) Time (sec) no compression 15.31 92 zlib (default level)2.37284 zlib (best speed) 2.43191 postgres internal lz

Re: Zedstore - compressed in-core columnar storage

2019-04-11 Thread Heikki Linnakangas
On 11/04/2019 16:12, Rafia Sabih wrote: On Tue, 9 Apr 2019 at 20:29, Robert Haas > wrote: BTW, can I express a small measure of disappointment that the name for the thing under discussion on this thread chose to be called "zedstore"?  That seems to

Re: Zedstore - compressed in-core columnar storage

2019-04-11 Thread Rafia Sabih
On Tue, 9 Apr 2019 at 20:29, Robert Haas wrote: > On Tue, Apr 9, 2019 at 11:51 AM Alvaro Herrera > wrote: > > This is not surprising, considering that columnar store is precisely the > > reason for starting the work on table AMs. > > > > We should certainly look into integrating some sort of

Re: Zedstore - compressed in-core columnar storage

2019-04-11 Thread Rafia Sabih
On Tue, 9 Apr 2019 at 02:27, Ashwin Agrawal wrote: > Heikki and I have been hacking recently for few weeks to implement > in-core columnar storage for PostgreSQL. Here's the design and initial > implementation of Zedstore, compressed in-core columnar storage (table > access method

Re: Zedstore - compressed in-core columnar storage

2019-04-11 Thread Konstantin Knizhnik
. Here's the design and initial implementation of Zedstore, compressed in-core columnar storage (table access method). Attaching the patch and link to github branch [1] to follow along. Very nice. I realize that it is very early days, but applying this patch I've managed to stumble over some

Re: Zedstore - compressed in-core columnar storage

2019-04-10 Thread Ashwin Agrawal
ore columnar storage for PostgreSQL. Here's the design and initial >>> implementation of Zedstore, compressed in-core columnar storage (table >>> access method). Attaching the patch and link to github branch [1] to >>> follow along. >>> >>> >> >

Re: Zedstore - compressed in-core columnar storage

2019-04-10 Thread Mark Kirkwood
On 11/04/19 4:01 PM, Mark Kirkwood wrote: On 9/04/19 12:27 PM, Ashwin Agrawal wrote: Heikki and I have been hacking recently for few weeks to implement in-core columnar storage for PostgreSQL. Here's the design and initial implementation of Zedstore, compressed in-core columnar storage

Re: Zedstore - compressed in-core columnar storage

2019-04-10 Thread Mark Kirkwood
On 9/04/19 12:27 PM, Ashwin Agrawal wrote: Heikki and I have been hacking recently for few weeks to implement in-core columnar storage for PostgreSQL. Here's the design and initial implementation of Zedstore, compressed in-core columnar storage (table access method). Attaching the patch

Re: Zedstore - compressed in-core columnar storage

2019-04-10 Thread Amit Kapila
On Wed, Apr 10, 2019 at 12:55 PM Heikki Linnakangas wrote: > > On 10/04/2019 09:29, Amit Kapila wrote: > > On Tue, Apr 9, 2019 at 5:57 AM Ashwin Agrawal wrote: > >> Row store > >> - > >> > >> The tuples are stored one after another, sorted by TID. For each > >> tuple, we store its 48-bit

Re: Zedstore - compressed in-core columnar storage

2019-04-10 Thread Heikki Linnakangas
On 10/04/2019 10:38, Konstantin Knizhnik wrote: I also a little bit confused about UNDO records and MVCC support in Zedstore. Actually columnar store is mostly needed for analytic for read-only or append-only data. One of the disadvantages of Postgres is quite larger per-record space overhead

Re: Zedstore - compressed in-core columnar storage

2019-04-10 Thread Konstantin Knizhnik
On 10.04.2019 10:25, Heikki Linnakangas wrote: On 10/04/2019 09:29, Amit Kapila wrote: On Tue, Apr 9, 2019 at 5:57 AM Ashwin Agrawal wrote: Row store - The tuples are stored one after another, sorted by TID. For each tuple, we store its 48-bit TID, a undo record pointer, and the

Re: Zedstore - compressed in-core columnar storage

2019-04-10 Thread Heikki Linnakangas
On 10/04/2019 09:29, Amit Kapila wrote: On Tue, Apr 9, 2019 at 5:57 AM Ashwin Agrawal wrote: Row store - The tuples are stored one after another, sorted by TID. For each tuple, we store its 48-bit TID, a undo record pointer, and the actual tuple data uncompressed. Storing undo

Re: Zedstore - compressed in-core columnar storage

2019-04-10 Thread Amit Kapila
On Tue, Apr 9, 2019 at 5:57 AM Ashwin Agrawal wrote: > > Heikki and I have been hacking recently for few weeks to implement > in-core columnar storage for PostgreSQL. Here's the design and initial > implementation of Zedstore, compressed in-core columnar storage (table > access met

  1   2   >