Re: [GENERAL] Performance appending to an array column

2017-09-22 Thread Imre Samu
>I also tried cstore_fdw for this, but my queries >(building a 2-D histogram) were taking 4+ seconds, >compared to 500ms using arrays. > ... > but maybe I could write my own extension Have you checked the new TimescaleDB extension? [ https://github.com/timescale/timescaledb ] "TimescaleDB is pack

Re: [GENERAL] Performance appending to an array column

2017-09-21 Thread Thomas Kellerer
Paul A Jungwirth schrieb am 21.09.2017 um 23:05: but maybe I could write my own extension to load regular files into Postgres arrays, sort of getting the best of both worlds. There is a foreign data wrapper for that: https://github.com/adunstan/file_text_array_fdw but it's pretty old and s

Re: [GENERAL] Performance appending to an array column

2017-09-21 Thread Paul A Jungwirth
> It's going to suck big-time :-(. Ha ha that's what I thought, but thank you for confirming. :-) > We ended up keeping > the time series data outside the DB; I doubt the conclusion would be > different today. Interesting. That seems a little radical to me, but I'll consider it more seriously no

Re: [GENERAL] Performance appending to an array column

2017-09-21 Thread Tom Lane
Paul A Jungwirth writes: > I'm considering a table structure where I'd be continuously appending > to long arrays of floats (10 million elements or more). Keeping the > data in arrays gives me much faster SELECT performance vs keeping it > in millions of rows. > But since these arrays keep growin

[GENERAL] Performance appending to an array column

2017-09-21 Thread Paul A Jungwirth
I'm considering a table structure where I'd be continuously appending to long arrays of floats (10 million elements or more). Keeping the data in arrays gives me much faster SELECT performance vs keeping it in millions of rows. But since these arrays keep growing, I'm wondering about the UPDATE pe