Re: [HACKERS] buildfarm: strange OOM failures on markhor (running CLOBBER_CACHE_RECURSIVELY)

2014-05-19 Thread Tomas Vondra
On 18.5.2014 20:49, Tom Lane wrote: With both of these things fixed, I'm not seeing any significant memory bloat during the first parallel group of the regression tests. I don't think I'll have the patience to let it run much further than that (the uuid and enum tests are still running after

Re: [HACKERS] buildfarm animals and 'snapshot too old'

2014-05-19 Thread Tomas Vondra
On 17.5.2014 22:35, Tomas Vondra wrote: On 17.5.2014 19:58, Andrew Dunstan wrote: On 05/15/2014 07:47 PM, Tomas Vondra wrote: On 15.5.2014 22:07, Andrew Dunstan wrote: Yes, I've seen that. Frankly, a test that takes something like 500 hours is a bit crazy. Maybe. It certainly is not a test

Re: [HACKERS] buildfarm animals and 'snapshot too old'

2014-05-19 Thread Tomas Vondra
On 19.5.2014 23:04, Andrew Dunstan wrote: On 05/19/2014 03:40 PM, Tomas Vondra wrote: On 17.5.2014 22:35, Tomas Vondra wrote: On 17.5.2014 19:58, Andrew Dunstan wrote: On 05/15/2014 07:47 PM, Tomas Vondra wrote: On 15.5.2014 22:07, Andrew Dunstan wrote: Yes, I've seen that. Frankly, a test

Re: [HACKERS] buildfarm: strange OOM failures on markhor (running CLOBBER_CACHE_RECURSIVELY)

2014-05-19 Thread Tomas Vondra
On 19.5.2014 22:11, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: On 18.5.2014 20:49, Tom Lane wrote: With both of these things fixed, I'm not seeing any significant memory bloat during the first parallel group of the regression tests. I don't think I'll have the patience to let it run

[HACKERS] buildfarm: strange OOM failures on markhor (running CLOBBER_CACHE_RECURSIVELY)

2014-05-17 Thread Tomas Vondra
Hi all, a few days ago I setup an buildfarm animal markhor, running the tests with CLOBBER_CACHE_RECURSIVELY. As the tests are running very long, reporting the results back to the server fails because of a safeguard limit in the buildfarm server. Anyway, that's being discussed in a different

Re: [HACKERS] buildfarm: strange OOM failures on markhor (running CLOBBER_CACHE_RECURSIVELY)

2014-05-17 Thread Tomas Vondra
On 17.5.2014 19:55, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: ... then of course the usual 'terminating connection because of crash of another server process' warning. Apparently, it's getting killed by the OOM killer, because it exhausts all the memory assigned to that VM (2GB

Re: [HACKERS] buildfarm animals and 'snapshot too old'

2014-05-17 Thread Tomas Vondra
On 17.5.2014 19:58, Andrew Dunstan wrote: On 05/15/2014 07:47 PM, Tomas Vondra wrote: On 15.5.2014 22:07, Andrew Dunstan wrote: Yes, I've seen that. Frankly, a test that takes something like 500 hours is a bit crazy. Maybe. It certainly is not a test people will use during development

Re: [HACKERS] buildfarm: strange OOM failures on markhor (running CLOBBER_CACHE_RECURSIVELY)

2014-05-17 Thread Tomas Vondra
On 17.5.2014 22:33, Tomas Vondra wrote: On 17.5.2014 21:31, Andres Freund wrote: On 2014-05-17 20:41:37 +0200, Tomas Vondra wrote: On 17.5.2014 19:55, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: The tests are already running, and there are a few postgres processes: PID VIRT RES

[HACKERS] buildfarm animals and 'snapshot too old'

2014-05-15 Thread Tomas Vondra
Hi all, today I got a few of errors like these (this one is from last week, though): Status Line: 493 snapshot too old: Wed May 7 04:36:57 2014 GMT Content: snapshot to old: Wed May 7 04:36:57 2014 GMT on the new buildfarm animals. I believe it was my mistake (incorrectly configured

Re: [HACKERS] buildfarm animals and 'snapshot too old'

2014-05-15 Thread Tomas Vondra
On 15 Květen 2014, 19:46, Andrew Dunstan wrote: On 05/15/2014 12:43 PM, Tomas Vondra wrote: Hi all, today I got a few of errors like these (this one is from last week, though): Status Line: 493 snapshot too old: Wed May 7 04:36:57 2014 GMT Content: snapshot to old: Wed May

Re: [HACKERS] buildfarm animals and 'snapshot too old'

2014-05-15 Thread Tomas Vondra
On 15.5.2014 22:56, Andrew Dunstan wrote: On 05/15/2014 04:30 PM, Stefan Kaltenbrunner wrote: well I'm not sure about about misconfigured but both my personal buildfarm members and pginfra run ones (like gaibasaurus) got errors complaining about snapshot too old in the past for long running

Re: [HACKERS] buildfarm animals and 'snapshot too old'

2014-05-15 Thread Tomas Vondra
On 15.5.2014 22:07, Andrew Dunstan wrote: Yes, I've seen that. Frankly, a test that takes something like 500 hours is a bit crazy. Maybe. It certainly is not a test people will use during development. But if it can detect some hard-to-find errors in the code, that might possibly lead to

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-14 Thread Tomas Vondra
On 14 Květen 2014, 13:51, Andres Freund wrote: On 2014-05-13 20:42:16 +0200, Tomas Vondra wrote: Can someone please approve the animals I've requested a few days ago? I'm already running the clobber tests with '--nosend --nostatus' and it's already reporting some errors. Would be nice to get

Re: [HACKERS] buildfarm / handling (undefined) locales

2014-05-14 Thread Tomas Vondra
On 14 Květen 2014, 16:26, Heikki Linnakangas wrote: On 05/14/2014 05:19 PM, Tom Lane wrote: Anyway, given that we seem to have consensus on doing this (modulo exact text of the error message), the next question is whether we want to change this only in HEAD, or consider it a back-patchable

Re: [HACKERS] Cache invalidation bug in RelationGetIndexAttrBitmap()

2014-05-14 Thread Tomas Vondra
On 14.5.2014 17:52, Andres Freund wrote: On 2014-05-14 15:17:39 +0200, Andres Freund wrote: On 2014-05-14 15:08:08 +0200, Tomas Vondra wrote: Apparently there's something wrong with 'test-decoding-check': Man. I shouldn't have asked... My code. There's some output in there that's probably

Re: [HACKERS] Cache invalidation bug in RelationGetIndexAttrBitmap()

2014-05-14 Thread Tomas Vondra
On 14.5.2014 22:29, Andres Freund wrote: Hi, On 2014-05-14 21:04:41 +0200, Tomas Vondra wrote: On 14.5.2014 17:52, Andres Freund wrote: On 2014-05-14 15:17:39 +0200, Andres Freund wrote: On 2014-05-14 15:08:08 +0200, Tomas Vondra wrote: Apparently there's something wrong with 'test

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-14 Thread Tomas Vondra
On 13.5.2014 20:42, Tomas Vondra wrote: On 10.5.2014 20:21, Tomas Vondra wrote: On 9.5.2014 00:47, Tomas Vondra wrote: And I've requested 6 more animals - two for each compiler. One set for tests with basic CLOBBER, one set for recursive CLOBBER. Can someone please approve the animals I've

Re: [HACKERS] buildfarm / handling (undefined) locales

2014-05-14 Thread Tomas Vondra
On 13.5.2014 23:07, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: On 05/13/2014 04:14 PM, Tom Lane wrote: But independently of whether it's a fatal error or not: when there's no relevant command-line argument then we print the invalid locale name message which is surely

[HACKERS] buildfarm / handling (undefined) locales

2014-05-13 Thread Tomas Vondra
Hi all, a few days ago I switched magpie into an LXC container, and while fixinig unrelated issue there, I noticed that although the tests seemed to run, some of the results are actually rubbish because of missing locales. So for example when the system misses cs_CZ.WIN-1250 (i.e. czech locale

Re: [HACKERS] buildfarm / handling (undefined) locales

2014-05-13 Thread Tomas Vondra
On 13.5.2014 20:27, Tomas Vondra wrote: Hi all, a few days ago I switched magpie into an LXC container, and while fixinig unrelated issue there, I noticed that although the tests seemed to run, some of the results are actually rubbish because of missing locales. I forgot to mention that I

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-13 Thread Tomas Vondra
On 10.5.2014 20:21, Tomas Vondra wrote: On 9.5.2014 00:47, Tomas Vondra wrote: So, if I get this right, the proposal is to have 7 animals: 1) all branches/locales, frequent builds (every few hours) magpie - gcc fulmar - icc treepie - clang 2) single branch/locale, CLOBBER, built

Re: [HACKERS] buildfarm / handling (undefined) locales

2014-05-13 Thread Tomas Vondra
On 13.5.2014 21:03, Andrew Dunstan wrote: On 05/13/2014 02:39 PM, Tomas Vondra wrote: On 13.5.2014 20:27, Tomas Vondra wrote: Hi all, a few days ago I switched magpie into an LXC container, and while fixinig unrelated issue there, I noticed that although the tests seemed to run, some

Re: [HACKERS] buildfarm / handling (undefined) locales

2014-05-13 Thread Tomas Vondra
On 13.5.2014 20:58, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: a few days ago I switched magpie into an LXC container, and while fixinig unrelated issue there, I noticed that although the tests seemed to run, some of the results are actually rubbish because of missing locales. So

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-10 Thread Tomas Vondra
On 9.5.2014 00:47, Tomas Vondra wrote: On 8.5.2014 23:48, Andrew Dunstan wrote: On 05/08/2014 05:21 PM, Alvaro Herrera wrote: Andrew Dunstan wrote: I really don't get what your objection to the setup is. And no, I don't want them to run concurrently, I'd rather spread out the cycles. I

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-09 Thread Tomas Vondra
On 9.5.2014 17:18, Alvaro Herrera wrote: Tomas Vondra wrote: So, if I get this right, the proposal is to have 7 animals: It's your machine, so you decide what you want. I'm only throwing out some ideas. 1) all branches/locales, frequent builds (every few hours) magpie - gcc fulmar

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-09 Thread Tomas Vondra
On 9.5.2014 20:09, Andrew Dunstan wrote: I've done that a bit in the past. At one stage all my Windows animals were some sort of bat. There's nothing magical about the names. It's just a text field and can be whatever we like. I initially started with animals because it seemed like a

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-09 Thread Tomas Vondra
On 9.5.2014 00:47, Tomas Vondra wrote: On 8.5.2014 23:48, Andrew Dunstan wrote: On 05/08/2014 05:21 PM, Alvaro Herrera wrote: Andrew Dunstan wrote: I really don't get what your objection to the setup is. And no, I don't want them to run concurrently, I'd rather spread out the cycles. I

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-08 Thread Tomas Vondra
On 6.5.2014 23:01, Tomas Vondra wrote: On 6.5.2014 22:24, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: I recall there was a call for more animals with CLOBBER_CACHE_ALWAYS some time ago, so I went and enabled that on all three animals. Let's see how long that will take. I see

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-08 Thread Tomas Vondra
On 8.5.2014 23:48, Andrew Dunstan wrote: On 05/08/2014 05:21 PM, Alvaro Herrera wrote: Andrew Dunstan wrote: I really don't get what your objection to the setup is. And no, I don't want them to run concurrently, I'd rather spread out the cycles. I wasn't objecting, merely an observation.

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-06 Thread Tomas Vondra
On 4.5.2014 21:29, Tomas Vondra wrote: On 3.5.2014 19:01, Andrew Dunstan wrote: On 05/03/2014 12:42 PM, Tomas Vondra wrote: On 3.5.2014 03:07, Noah Misch wrote: More coverage of non-gcc compilers would be an asset to the buildfarm. Does that include non-gcc compilers on Linux/x86 platforms

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-06 Thread Tomas Vondra
On 6.5.2014 22:24, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: I recall there was a call for more animals with CLOBBER_CACHE_ALWAYS some time ago, so I went and enabled that on all three animals. Let's see how long that will take. I see there are more 'clobber' options in the code

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-04 Thread Tomas Vondra
On 3.5.2014 19:01, Andrew Dunstan wrote: On 05/03/2014 12:42 PM, Tomas Vondra wrote: On 3.5.2014 03:07, Noah Misch wrote: More coverage of non-gcc compilers would be an asset to the buildfarm. Does that include non-gcc compilers on Linux/x86 platforms? Magpie is pretty much dedicated

Re: [HACKERS] Sending out a request for more buildfarm animals?

2014-05-03 Thread Tomas Vondra
On 3.5.2014 03:07, Noah Misch wrote: More coverage of non-gcc compilers would be an asset to the buildfarm. Does that include non-gcc compilers on Linux/x86 platforms? Magpie is pretty much dedicated to the buildfarm, and it's pretty much doing nothing most of the time, so running the tests

Re: [HACKERS] [GENERAL] aggregate returning anyarray and 'cannot determine result data type'

2014-04-26 Thread Tomas Vondra
On 25.4.2014 23:26, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: On 23.4.2014 16:07, Tom Lane wrote: To be concrete: let's add a new boolean parameter with the semantics of final function takes extra dummy arguments (default false). There would need to be one for the separate moving

Re: [HACKERS] [GENERAL] aggregate returning anyarray and 'cannot determine result data type'

2014-04-25 Thread Tomas Vondra
On 23.4.2014 16:07, Tom Lane wrote: To be concrete: let's add a new boolean parameter with the semantics of final function takes extra dummy arguments (default false). There would need to be one for the separate moving-aggregate final function too, of course. The best naming idea I've got

Re: [HACKERS] PATCH: decreasing memory needlessly consumed by array_agg

2014-04-01 Thread Tomas Vondra
On 31.3.2014 21:04, Robert Haas wrote: On Thu, Mar 27, 2014 at 10:00 PM, Tomas Vondra t...@fuzzy.cz wrote: The patch also does one more thing - it changes how the arrays (in the aggregate state) grow. Originally it worked like this /* initial size */ astate-alen = 64; /* when

Re: [HACKERS] PATCH: decreasing memory needlessly consumed by array_agg

2014-04-01 Thread Tomas Vondra
On 1.4.2014 19:08, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: I've been thinking about it a bit more and maybe the doubling is not that bad idea, after all. It is not. There's a reason why that's our standard behavior. The current array_agg however violates some

Re: [HACKERS] PATCH: decreasing memory needlessly consumed by array_agg

2014-04-01 Thread Tomas Vondra
On 1.4.2014 20:56, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: On 1.4.2014 19:08, Tom Lane wrote: You're conveniently ignoring the callers that set release=true. Reverse engineering a query that exhibits memory bloat is left as an exercise for the reader (but in a quick look, I'll bet

[HACKERS] PATCH: decreasing memory needlessly consumed by array_agg

2014-03-27 Thread Tomas Vondra
Hi, this is a patch for issue reported in October 2013 in pgsql-bugs: http://www.postgresql.org/message-id/3839201.nfa2rvc...@techfox.foxi Frank van Vugt reported that a simple query with array_agg() and large number of groups (1e7) fails because of OOM even on machine with 32GB of RAM. So

Re: [HACKERS] jsonb and nested hstore

2014-03-23 Thread Tomas Vondra
On 21.3.2014 08:23, Peter Geoghegan wrote: On Thu, Mar 13, 2014 at 3:39 PM, Peter Geoghegan p...@heroku.com wrote: On Thu, Mar 13, 2014 at 2:21 AM, Greg Stark st...@mit.edu wrote: It does sound like the main question here is which opclass should be the default. From the discussion there's a

Re: [HACKERS] jsonb and nested hstore

2014-03-15 Thread Tomas Vondra
On 15.3.2014 06:40, Peter Geoghegan wrote: On Fri, Mar 14, 2014 at 6:44 PM, Tomas Vondra t...@fuzzy.cz wrote: Well, depends on how you define useful. With the sample dataset 'delicious' (see Peter's post) I can do this: SELECT doc FROM delicious WHERE doc @ '{title_detail

Re: [HACKERS] jsonb and nested hstore

2014-03-15 Thread Tomas Vondra
On 15.3.2014 02:15, Peter Geoghegan wrote: On Fri, Mar 14, 2014 at 5:10 PM, Tomas Vondra t...@fuzzy.cz wrote: I'm on commit a3115f0d, which is just 2 days old, so I suppose this was not fixed yet. Try merging the feature branch now, which will get you commit 16923d, which you're missing

Re: [HACKERS] jsonb and nested hstore

2014-03-14 Thread Tomas Vondra
On 13 Březen 2014, 23:39, Peter Geoghegan wrote: On Thu, Mar 13, 2014 at 2:21 AM, Greg Stark st...@mit.edu wrote: It does sound like the main question here is which opclass should be the default. From the discussion there's a jsonb_hash_ops which works on all input values but supports fewer

Re: [HACKERS] jsonb and nested hstore

2014-03-14 Thread Tomas Vondra
On 14.3.2014 20:18, Josh Berkus wrote: On 03/14/2014 04:52 AM, Oleg Bartunov wrote: VODKA index will have no lenght limitation. Yeah, so I think we go with what we have, and tell people if you're hitting these length issues, wait for 9.5, where they will be fixed. VODKA may be great, but

Re: [HACKERS] jsonb and nested hstore

2014-03-14 Thread Tomas Vondra
On 14.3.2014 22:54, Peter Geoghegan wrote: On Fri, Mar 14, 2014 at 2:21 PM, Tomas Vondra t...@fuzzy.cz wrote: I'm not awfully familiar with the GIN code, but based on Alexander's feedback I presume fixing the GIN length limit (or rather removing it, as it's a feature, not a bug) is quite

Re: [HACKERS] jsonb and nested hstore

2014-03-14 Thread Tomas Vondra
On 14.3.2014 23:06, Peter Geoghegan wrote: For the benefit of anyone that would like to try the patch out, I make available a custom format dump of some delicious sample data. I can query the sample data as follows on my local installation: [local]/jsondata=# select count(*) from delicious ;

Re: [HACKERS] jsonb and nested hstore

2014-03-14 Thread Tomas Vondra
On 15.3.2014 02:03, Greg Stark wrote: On Fri, Mar 14, 2014 at 9:21 PM, Tomas Vondra t...@fuzzy.cz wrote: I'm not awfully familiar with the GIN code, but based on Alexander's feedback I presume fixing the GIN length limit (or rather removing it, as it's a feature, not a bug) is quite

Re: [HACKERS] jsonb and nested hstore

2014-03-13 Thread Tomas Vondra
On 13.3.2014 13:28, Oleg Bartunov wrote: On Thu, Mar 13, 2014 at 4:21 PM, Alexander Korotkov aekorot...@gmail.com wrote: On Thu, Mar 13, 2014 at 1:21 PM, Greg Stark st...@mit.edu wrote: Well these are just normal gin and gist indexes. If we want to come up with new index operator classess we

Re: [HACKERS] jsonb and nested hstore

2014-03-12 Thread Tomas Vondra
On 12 Březen 2014, 0:41, Peter Geoghegan wrote: On Tue, Mar 11, 2014 at 3:58 PM, Tomas Vondra t...@fuzzy.cz wrote: ERROR: index row size 1416 exceeds maximum 1352 for index gin_idx All index AMs have similar restrictions. Yes, I know and I have no problem with restrictions in general. You

Re: [HACKERS] jsonb and nested hstore

2014-03-12 Thread Tomas Vondra
On 12 Březen 2014, 0:51, Peter Geoghegan wrote: On Tue, Mar 11, 2014 at 4:41 PM, Peter Geoghegan p...@heroku.com wrote: I think that in practice the general recommendation will be that when indexing at the top level, use jsonb_hash_ops. When indexing nested items, use the more flexible

Re: [HACKERS] pgstat wait timeout (RE: contrib/cache_scan)

2014-03-12 Thread Tomas Vondra
On 12 Březen 2014, 14:54, Kouhei Kaigai wrote: It is another topic from the main thread, I noticed the following message under the test cases that takes heavy INSERT workload; provided by Haribabu. [kaigai@iwashi ~]$ createdb mytest [kaigai@iwashi ~]$ psql -af ~/cache_scan.sql mytest

Re: [HACKERS] jsonb and nested hstore

2014-03-12 Thread Tomas Vondra
On 12.3.2014 20:40, Peter Geoghegan wrote: On Wed, Mar 12, 2014 at 6:20 AM, Tomas Vondra t...@fuzzy.cz wrote: I'm still not sure how would that look. Does that mean I'd have to create multiple GIN indexes - one for each possible key or something like that? Can you give an example? It could

Re: [HACKERS] jsonb and nested hstore

2014-03-12 Thread Tomas Vondra
On 12.3.2014 21:58, Peter Geoghegan wrote: The use case you describe here doesn't sound like something similar to full text search. It sounds like something identical. I think this very depends on the definition of full text search. In any case, let's focus on what we have right now. I

Re: [HACKERS] jsonb and nested hstore

2014-03-12 Thread Tomas Vondra
On 12.3.2014 21:55, Josh Berkus wrote: Andrew, Peter: Just so I'm clear on the limits here, lemme make sure I understand this: a) GIN indexing is limited to ~~1500chars The exact message I get is this: ERROR: index row size 1944 exceeds maximum 1352 for index tmp_idx so it's 1352B. But

Re: [HACKERS] jsonb and nested hstore

2014-03-12 Thread Tomas Vondra
On 12.3.2014 22:43, Peter Geoghegan wrote: On Wed, Mar 12, 2014 at 2:30 PM, Tomas Vondra t...@fuzzy.cz wrote: I think that's unfounded assumption. Many users actually have very little control over the documents or queries - a nice example may be the mail archive, with headers stored

Re: [HACKERS] GIN improvements part2: fast scan

2014-03-11 Thread Tomas Vondra
Hi all, a quick question that just occured to me - do you plan to tweak the cost estimation fot GIN indexes, in this patch? IMHO it would be appropriate, given the improvements and gains, but it seems to me gincostestimate() was not touched by this patch. I just ran into this while testing some

Re: [HACKERS] jsonb and nested hstore

2014-03-11 Thread Tomas Vondra
Hi, I've spent a few hours stress-testing this a bit - loading a mail archive with ~1M of messages (with headers stored in a jsonb column) and then doing queries on that. Good news - no crashes or any such issues so far. The queries that I ran manually seem to return sane results. The only

Re: [HACKERS] commit fest status and release timeline

2014-03-03 Thread Tomas Vondra
On 1.3.2014 18:01, Peter Eisentraut wrote: Status Summary. Needs Review: 36, Waiting on Author: 7, Ready for Committer: 16, Committed: 43, Returned with Feedback: 8, Rejected: 4. Total: 114. We're still on track to achieve about 50% committed patches, which would be similar to the previous

Re: [HACKERS] jsonb and nested hstore

2014-02-24 Thread Tomas Vondra
On 7.2.2014 00:47, Andrew Dunstan wrote: On 02/05/2014 10:36 AM, Teodor Sigaev wrote: Should I make new version of patch? Right now it's placed on github. May be Andrew wants to change something? Attached are updated patches. Apart from the things Teodor has fixed, this includes

Re: [HACKERS] Storing the password in .pgpass file in an encrypted format

2014-02-21 Thread Tomas Vondra
Hi, On 21 Únor 2014, 16:52, Christopher Browne wrote: On Fri, Feb 21, 2014 at 7:49 AM, firoz e v firoz...@huawei.com wrote: Hi, Is there a way to store the password in .pgpass file in an encrypted format (for example, to be used by pg_dump). Even though, there are ways to set the

Re: [HACKERS] Storing the password in .pgpass file in an encrypted format

2014-02-21 Thread Tomas Vondra
On 22.2.2014 00:02, Josh Berkus wrote: On 02/21/2014 09:11 AM, Tomas Vondra wrote: What I think might be useful and safe at the same time is encrypted .pgpass with tools asking for the encryption key. Think of it as a simple passord wallet - not really useful if you're connecting to a single

Re: [HACKERS] pg_stat_tmp files for dropped databases

2014-02-21 Thread Tomas Vondra
Hi, On 22.2.2014 01:13, Thom Brown wrote: Hi, I've noticed that files for dropped databases aren't removed from pg_stat_tmp. After a cluster-wide VACUUM ANALYSE, and restarting Postgres, all the old database pg_stat_tmp files remain. Shouldn't these be cleaned up? Yeah, that's a bug in

Re: [HACKERS] GIN improvements part2: fast scan

2014-02-09 Thread Tomas Vondra
On 3.2.2014 07:53, Oleg Bartunov wrote: Tomasa, it'd be nice if you use real data in your testing. One very good application of gin fast-scan is dramatic performance improvement of hstore/jsonb @ operator, see slides 57, 58

Re: [HACKERS] GIN improvements part2: fast scan

2014-02-09 Thread Tomas Vondra
On 9.2.2014 22:51, Erik Rijkers wrote: On Sun, February 9, 2014 22:35, Tomas Vondra wrote: On 3.2.2014 07:53, Oleg Bartunov wrote: PS. I used data delicious-rss-1250k.gz from http://randomwalker.info/data/delicious/ I'm working on extending the GIN testing to include this test (and I'll use

Re: [HACKERS] GIN improvements part2: fast scan

2014-02-03 Thread Tomas Vondra
Hi Oleg, On 3 Únor 2014, 7:53, Oleg Bartunov wrote: Tomasa, it'd be nice if you use real data in your testing. I'm using a mailing list archive (the benchmark is essentially a simple search engine on top of the archive, implemented using built-in full-text). So I think this is a quite 'real'

Re: [HACKERS] GIN improvements part2: fast scan

2014-02-03 Thread Tomas Vondra
Hi Alexander, On 3 Únor 2014, 15:31, Alexander Korotkov wrote: I found my patch 0005-Ternary-consistent-implementation.patch to be completely wrong. It introduces ternary consistent function to opclass, but don't uses it, because I forgot to include ginlogic.c change into patch. So, it

Re: [HACKERS] GIN improvements part2: fast scan

2014-02-03 Thread Tomas Vondra
On 3 Únor 2014, 17:08, Alexander Korotkov wrote: On Mon, Feb 3, 2014 at 7:24 PM, Tomas Vondra t...@fuzzy.cz wrote: On 3 Únor 2014, 15:31, Alexander Korotkov wrote: I found my patch 0005-Ternary-consistent-implementation.patch to be completely wrong. It introduces ternary consistent

Re: [HACKERS] GIN improvements part2: fast scan

2014-02-03 Thread Tomas Vondra
On 3 Únor 2014, 19:18, Alexander Korotkov wrote: On Mon, Feb 3, 2014 at 8:19 PM, Tomas Vondra t...@fuzzy.cz wrote: Sometimes test cases are not what we expect. For example: =# explain SELECT id FROM messages WHERE body_tsvector @@ to_tsquery('english','(5alpha1-initdb''d

Re: [HACKERS] GIN improvements part2: fast scan

2014-02-02 Thread Tomas Vondra
On 2.2.2014 11:45, Heikki Linnakangas wrote: On 01/30/2014 01:53 AM, Tomas Vondra wrote: (3) A file with explain plans for 4 queries suffering ~2x slowdown, and explain plans with 9.4 master and Heikki's patches is available here: http://www.fuzzy.cz/tmp/gin/queries.txt

Re: [HACKERS] GIN improvements part2: fast scan

2014-01-29 Thread Tomas Vondra
On 28.1.2014 08:29, Heikki Linnakangas wrote: On 01/28/2014 05:54 AM, Tomas Vondra wrote: Then I ran those scripts on: * 9.3 * 9.4 with Heikki's patches (9.4-heikki) * 9.4 with Heikki's and first patch (9.4-alex-1) * 9.4 with Heikki's and both patches (9.4-alex-2) It would

Re: [HACKERS] GIN improvements part2: fast scan

2014-01-26 Thread Tomas Vondra
On 26.1.2014 17:14, Heikki Linnakangas wrote: I would actually expect it to be fairly effective for that query, so that's a bit surprising. I added counters to see where the calls are coming from, and it seems that about 80% of the calls are actually coming from this little the feature I

Re: [HACKERS] GIN improvements part2: fast scan

2014-01-25 Thread Tomas Vondra
On 23.1.2014 17:22, Heikki Linnakangas wrote: I measured the time that query takes, and the number of pages hit, using explain (analyze, buffers true) patchestime (ms)buffers --- unpatched6501316 patch 10.521316 patches 1+20.501316

Re: [HACKERS] GIN improvements part2: fast scan

2014-01-25 Thread Tomas Vondra
Hi! On 25.1.2014 22:21, Heikki Linnakangas wrote: Attached is a new version of the patch set, with those bugs fixed. I've done a bunch of tests with all the 4 patches applied, and it seems to work now. I've done tests with various conditions (AND/OR, number of words, number of conditions) and I

Re: [HACKERS] GIN improvements part2: fast scan

2014-01-23 Thread Tomas Vondra
On 23.1.2014 17:22, Heikki Linnakangas wrote: On 01/14/2014 05:35 PM, Alexander Korotkov wrote: Attached version is rebased against last version of packed posting lists. Thanks! I think we're missing a trick with multi-key queries. We know that when multiple scan keys are used, they are

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-21 Thread Tomas Vondra
On 21.1.2014 22:21, Heikki Linnakangas wrote: On 01/21/2014 11:35 AM, Heikki Linnakangas wrote: On 01/21/2014 04:02 AM, Tomas Vondra wrote: On 20.1.2014 19:30, Heikki Linnakangas wrote: Attached is a yet another version, with more bugs fixed and more comments added and updated. I would

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-20 Thread Tomas Vondra
On 20.1.2014 19:30, Heikki Linnakangas wrote: Attached is a yet another version, with more bugs fixed and more comments added and updated. I would appreciate some heavy-testing of this patch now. If you could re-run the tests you've been using, that could be great. I've tested the WAL

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-14 Thread Tomas Vondra
On 14.1.2014 00:38, Tomas Vondra wrote: On 13.1.2014 18:07, Alexander Korotkov wrote: On Sat, Jan 11, 2014 at 6:15 AM, Tomas Vondra t...@fuzzy.cz mailto:t...@fuzzy.cz wrote: On 8.1.2014 22:58, Alexander Korotkov wrote: Thanks for reporting. Fixed version is attached. I've

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-13 Thread Tomas Vondra
On 13.1.2014 18:07, Alexander Korotkov wrote: On Sat, Jan 11, 2014 at 6:15 AM, Tomas Vondra t...@fuzzy.cz mailto:t...@fuzzy.cz wrote: On 8.1.2014 22:58, Alexander Korotkov wrote: Thanks for reporting. Fixed version is attached. I've tried to rerun the 'archie' benchmark

Re: [HACKERS] patch: make_timestamp function

2014-01-11 Thread Tomas Vondra
Hi, I've done a quick review of this patch: 1) patch applies fine to the current HEAD, with a few hunks offset by a few lines 2) the compilation fails because of duplicate OIDs in pg_proc, so I had to change 3969-3975 to 4033-4039, then it compiles fine 3) make installcheck works fine

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-10 Thread Tomas Vondra
On 8.1.2014 22:58, Alexander Korotkov wrote: Thanks for reporting. Fixed version is attached. I've tried to rerun the 'archie' benchmark with the current patch, and once again I got PANIC: could not split GIN page, didn't fit I reran it with '--enable-cassert' and with that I got TRAP:

Re: [HACKERS] writable FDWs / update targets confusion

2013-12-06 Thread Tomas Vondra
On 18.11.2013 09:28, Albe Laurenz wrote: Tom Lane wrote: Tom, could you show us a rope if there is one? What is it you actually need to fetch? IIRC, the idea was that most FDWs would do the equivalent of fetching the primary-key columns to use in an update. If that's what you need, then

Re: [HACKERS] strncpy is not a safe version of strcpy

2013-11-14 Thread Tomas Vondra
On 15 Listopad 2013, 0:07, David Rowley wrote: Hi All, As a bit of a background task, over the past few days I've been analysing the uses of strncpy in the code just to try and validate if it is the right function to be using. I've already seen quite a few places where their usage is

Re: [HACKERS] strncpy is not a safe version of strcpy

2013-11-14 Thread Tomas Vondra
On 15 Listopad 2013, 1:00, David Rowley wrote: On Fri, Nov 15, 2013 at 12:33 PM, Tomas Vondra t...@fuzzy.cz wrote: It is likely far better explained here -- http://www.courtesan.com/todd/papers/strlcpy.html For example , the following 2 lines in jsonfuncs.c memset(name, 0

[HACKERS] writable FDWs / update targets confusion

2013-11-12 Thread Tomas Vondra
Hi, I'm working on adding write support to one of my FDWs. Adding INSERT went pretty fine, but when adding DELETE/UPDATE I got really confused about how the update targets are supposed to work. My understanding of how it's supposed to work is this: (1) AddForeignUpdateTargets adds columns that

Re: [HACKERS] FDW: possible resjunk columns in AddForeignUpdateTargets

2013-11-12 Thread Tomas Vondra
On 8.11.2013 16:13, Albe Laurenz wrote: Tom Lane wrote: Albe Laurenz laurenz.a...@wien.gv.at writes: What I would like to do is add a custom resjunk column (e.g. a bytea) in AddForeignUpdateTargets that carries a row identifier from the scan state to the modify state. Would that be possible?

[HACKERS] strange behavior with C function and DEFAULT function parameters

2013-10-20 Thread Tomas Vondra
Hi, I ran into some pretty strange behavior of C-language function and default parameter values, both on 9.2 and 9.4devel. Consider for example this trivial C function: Datum show_bug(PG_FUNCTION_ARGS) { elog(WARNING, called ;-)); PG_RETURN_VOID(); } which is

Re: [HACKERS] strange behavior with C function and DEFAULT function parameters

2013-10-20 Thread Tomas Vondra
On 21.10.2013 02:38, Tomas Vondra wrote: Hi, I ran into some pretty strange behavior of C-language function and default parameter values, both on 9.2 and 9.4devel. Consider for example this trivial C function: Datum show_bug(PG_FUNCTION_ARGS) { elog(WARNING, called

[HACKERS] fdw_private and (List*) handling in FDW API

2013-10-18 Thread Tomas Vondra
Hi, I've been exploring the new FDW API in the past few days, and I'm slightly confused by the fdw_private fields. A few comments: 1) Generally all the API functions pass data using fields in the nodes (e.g. GetForeignRelSize uses baserel-fdw_private etc.), but PlanForeignModify simply returns

Re: [HACKERS] fdw_private and (List*) handling in FDW API

2013-10-18 Thread Tomas Vondra
On 18 Říjen 2013, 17:52, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: 2) Is there any particular reason why PlanForeignModify/BeginForeignModify require the fdw_private to be a List*, and not a generic pointer? That data has to be copiable by copyObject(), which a generic void

Re: [HACKERS] FDW API / flow charts for the docs?

2013-10-18 Thread Tomas Vondra
On 18.10.2013 23:35, Stephen Frost wrote: Tomas, * Tomas Vondra (t...@fuzzy.cz) wrote: My impression from that thread was that one of the requirements is reasonable versioning / diff support, and AFAIK that's not a good match for any GUI-based product. So while I like dia and I used

Re: [HACKERS] FDW API / flow charts for the docs?

2013-10-18 Thread Tomas Vondra
On 18.10.2013 23:52, Peter Eisentraut wrote: On 10/18/13 5:35 PM, Stephen Frost wrote: I can't see it being a major effort to get it from the wiki into the docs, though perhaps I'm being a bit over-optomistic wrt that. Hah! Consider that an image would have to work with the following

Re: [HACKERS] FDW API / flow charts for the docs?

2013-10-17 Thread Tomas Vondra
On 17 Říjen 2013, 5:32, Stephen Frost wrote: Alvaro, * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: Tomas Vondra wrote: Attached is the set of flow charts, showing the sequence of callbacks for all the supported commands (i.e. SELECT, INSERT, UPDATE, DELETE, ANALYZE). Wouldn't

[HACKERS] FDW API / flow charts for the docs?

2013-10-16 Thread Tomas Vondra
Hi, I've been experimenting with the new reworked FDW API to get familiar with it. The postgres_fdw is a great source of knowledge (huge thanks to Shigeru Hanada, KaiGai Kohei and everyone else who made this happen), but in the end I had to draw some flow charts in Dia, to understand how exactly

Re: [HACKERS] Improving avg performance for numeric

2013-10-14 Thread Tomas Vondra
On 24.9.2013 17:57, Pavel Stehule wrote: 2013/9/24 Robert Haas robertmh...@gmail.com mailto:robertmh...@gmail.com On Mon, Sep 23, 2013 at 4:15 PM, Tomas Vondra t...@fuzzy.cz mailto:t...@fuzzy.cz wrote: Seems ready for commiter to me. I'll wait a few days for others to comment

Re: [HACKERS] Re: [BUGS] BUG #7873: pg_restore --clean tries to drop tables that don't exist

2013-10-14 Thread Tomas Vondra
Hi, On 14.10.2013 23:44, Andres Freund wrote: On 2013-10-10 12:54:23 -0400, Andrew Dunstan wrote: On 09/19/2013 06:12 PM, Pavel Stehule wrote: 2013/9/16 Satoshi Nagayasu sn...@uptime.jp mailto:sn...@uptime.jp I'm looking at this patch, and I have a question here. Should DROP TRIGGER IF

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-12 Thread Tomas Vondra
On 12.10.2013 12:11, Alexander Korotkov wrote: On Sat, Oct 12, 2013 at 1:55 AM, Tomas Vondra t...@fuzzy.cz mailto:t...@fuzzy.cz wrote: Yup, this version fixed the issues. I haven't been able to do any benchmarks yet, all I have is some basic stats | HEAD

Re: [HACKERS] Re: custom hash-based COUNT(DISTINCT) aggregate - unexpectedly high memory consumption

2013-10-12 Thread Tomas Vondra
On 11.10.2013 13:42, Huchev wrote: gettimeofday(start, NULL); for (i = 0; i VALUES; i++) { state = XXH32_init(result); XXH32_update(state, i, 4); XXH32_digest(state); } gettimeofday(end, NULL); This code is using the update variant, which is only

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-11 Thread Tomas Vondra
On 10.10.2013 13:57, Heikki Linnakangas wrote: On 09.10.2013 02:04, Tomas Vondra wrote: On 8.10.2013 21:59, Heikki Linnakangas wrote: On 08.10.2013 17:47, Alexander Korotkov wrote: Hi, Tomas! On Sun, Oct 6, 2013 at 3:58 AM, Tomas Vondrat...@fuzzy.cz wrote: I've attempted to rerun

Re: [HACKERS] Re: custom hash-based COUNT(DISTINCT) aggregate - unexpectedly high memory consumption

2013-10-08 Thread Tomas Vondra
On 8 Říjen 2013, 6:23, Atri Sharma wrote: Hi Tomas, Consider the aspects associated with open addressing.Open addressing can quickly lead to growth in the main table.Also, chaining is a much cleaner way of collision resolution,IMHO. What do you mean by growth in the main table? Sorry, I

Re: [HACKERS] Re: custom hash-based COUNT(DISTINCT) aggregate - unexpectedly high memory consumption

2013-10-08 Thread Tomas Vondra
On 8 Říjen 2013, 11:42, Atri Sharma wrote: I've made some significant improvements in the chaining version (in the master branch), now getting about the memory consumption I've estimated. I agree, we can hope to reduce the memory consumption by making changes in the current chaining

<    6   7   8   9   10   11   12   13   14   >