Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

2014-03-20 Thread Jesper Krogh
On 15/03/14 20:27, Heikki Linnakangas wrote: That said, I didn't expect the difference to be quite that big when you're appending to the end of the table. When the new entries go to the end of the posting lists, you only need to recompress and WAL-log the last posting list, which is max 256

Re: [HACKERS] issue with gininsert under very high load

2014-02-13 Thread Jesper Krogh
On 14/02/14 00:49, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-02-13 16:15:42 -0500, Tom Lane wrote: Something like the attached? Can somebody who's seen this problem confirm this improves matters? Hm. Won't that possiby lead to the fast tuple list growing

Re: [HACKERS] GIN improvements part2: fast scan

2014-02-03 Thread Jesper Krogh
On 03/02/14 02:44, Tomas Vondra wrote: (2) The question is whether the new patch works fine on rare words. See this for comparison of the patches against HEAD: http://www.fuzzy.cz/tmp/gin/3-rare-words.png http://www.fuzzy.cz/tmp/gin/3-rare-words-new.png and this is the

Re: [HACKERS] Compression of full-page-writes

2013-10-13 Thread Jesper Krogh
On 11/10/13 19:06, Andres Freund wrote: On 2013-10-11 09:22:50 +0530, Amit Kapila wrote: I think it will be difficult to prove by using any compression algorithm, that it compresses in most of the scenario's. In many cases it can so happen that the WAL will also not be reduced and tps can also

Re: [HACKERS] [9.4 CF] Free VMs for Reviewers Testers

2013-07-09 Thread Jesper Krogh
The really, really big ones are useful even for pushing limits, such as cr1.8xlarge, with 32 CPUs and 244GiB memory. Current spot instance price (the heavily discounted can die at any time one) is $0.343/hr. Otherwise, it's 3.500/hr. Just to keep in mind cpus are similar throttled: One

Re: [HACKERS] Regarding GIN Fast Update Technique

2013-06-07 Thread Jesper Krogh
On 07/06/13 16:39, Tom Lane wrote: Amit Langote amitlangot...@gmail.com writes: Okay, aside from that case, what else would move those to the main structure? They (the entries in the unsorted pending list) are in the local memory (work_mem?) of the backend, right? No. If they were, it

Re: [HACKERS] Enabling Checksums

2012-11-11 Thread Jesper Krogh
On 12/11/12 05:55, Greg Smith wrote: The only guarantee I see that we can give for online upgrades is that after a VACUUM CHECKSUM sweep is done, and every page is known to both have a valid checksum on it and have its checksum bits set, *then* any page that doesn't have both set bits and a

Re: [HACKERS] Enabling Checksums

2012-11-08 Thread Jesper Krogh
On 09/11/12 02:01, Jeff Davis wrote: As I understand it, the main part of the remaining work to be done for the checksums patch (at least the first commit) is to have a better way to enable/disable them. For the sake of simplicity (implementation as well as usability), it seems like there is

Re: [HACKERS] Re: [WIP] Performance Improvement by reducing WAL for Update Operation

2012-10-25 Thread Jesper Krogh
Naturally, there are other compression and delta encoding schemes. Does anyone feel the need to explore further alternatives? We might eventually find the need for multiple, user-selectable, WAL compression strategies. I don't recommend taking that step yet. my currently implemented

Re: [HACKERS] Inserting heap tuples in bulk in COPY

2012-08-09 Thread Jesper Krogh
On 08/08/12 21:34, Robert Haas wrote: I think we need to implement buffering both to end of statement or end of transaction, not just one or the other. Another (not necessarily better) idea is to use a buffer that's part of the index, like the GIN fastupdate stuff, so that there's no particular

Re: [HACKERS] Qual evaluation cost estimates for GIN indexes

2012-02-16 Thread Jesper Krogh
Hi. First, thanks for looking at this. Except from GIN indexes and full-text-search being really good in our applications, this also points to those excact places where it can be improved. On 2012-02-17 00:15, Tom Lane wrote: I looked into the complaint here of poor estimation for GIN

Re: [HACKERS] index-only quals vs. security_barrier views

2012-02-11 Thread Jesper Krogh
On 2012-02-09 22:17, Jesper Krogh wrote: On 2012-02-09 21:09, Robert Haas wrote: That doesn't make sense to me. If you probe index A for rows where a = 1 and find that CTID (100,1) is such a row, and now want to return a column value b that is not present in that index, the fastest way to get

Re: [HACKERS] index-only quals vs. security_barrier views

2012-02-09 Thread Jesper Krogh
On 2012-02-09 18:02, Robert Haas wrote: I don't have any appetite for trying to do anything more with index-only scans for 9.2, though maybe someone else will think otherwise. But I would like very much to get KaiGai's leakproof stuff committed, and so it seems like a good idea to reconcile the

Re: [HACKERS] index-only quals vs. security_barrier views

2012-02-09 Thread Jesper Krogh
On 2012-02-09 21:09, Robert Haas wrote: That doesn't make sense to me. If you probe index A for rows where a = 1 and find that CTID (100,1) is such a row, and now want to return a column value b that is not present in that index, the fastest way to get the row is going to be to fetch block 100

Re: [HACKERS] Group commit, revised

2012-01-29 Thread Jesper Krogh
On 2012-01-29 01:48, Jeff Janes wrote: I ran three modes, head, head with commit_delay, and the group_commit patch shared_buffers = 600MB wal_sync_method=fsync optionally with: commit_delay=5 commit_siblings=1 pgbench -i -s40 for clients in 1 5 10 15 20 25 30 pgbench -T 30 -M prepared -c

Re: [HACKERS] Page Checksums + Double Writes

2011-12-22 Thread Jesper Krogh
On 2011-12-22 09:42, Florian Weimer wrote: * David Fetter: The issue is that double writes needs a checksum to work by itself, and page checksums more broadly work better when there are double writes, obviating the need to have full_page_writes on. How desirable is it to disable

Re: [HACKERS] Page Checksums

2011-12-20 Thread Jesper Krogh
On 2011-12-20 18:44, Simon Riggs wrote: On Mon, Dec 19, 2011 at 11:10 AM, Simon Riggssi...@2ndquadrant.com wrote: The only sensible way to handle this is to change the page format as discussed. IMHO the only sensible way that can happen is if we also support an online upgrade feature. I will

Re: [HACKERS] Page Checksums

2011-12-20 Thread Jesper Krogh
On 2011-12-19 02:55, Greg Stark wrote: On Sun, Dec 18, 2011 at 7:51 PM, Jesper Kroghjes...@krogh.cc wrote: I dont know if it would be seen as a half baked feature.. or similar, and I dont know if the hint bit problem is solvable at all, but I could easily imagine checksumming just skipping the

Re: [HACKERS] Postgres 9.1: Adding rows to table causing too much latency in other queries

2011-12-19 Thread Jesper Krogh
On 2011-12-19 18:08, Marti Raudsepp wrote: The query uses the gin index and the tsearch ranking function ts_rank_cd. Another thought -- have you read about the GIN fast updates feature? This existed in 9.0 too. Instead of updating the index directly, GIN appends all changes to a sequential

Re: [HACKERS] Page Checksums

2011-12-18 Thread Jesper Krogh
On 2011-12-18 11:19, Heikki Linnakangas wrote: The patch requires that full page writes be on in order to obviate this problem by never reading a torn page. Doesn't help. Hint bit updates are not WAL-logged. I dont know if it would be seen as a half baked feature.. or similar, and I dont

Re: [HACKERS] SP-GiST versus index-only scans

2011-12-14 Thread Jesper Krogh
On 2011-12-14 19:00, Tom Lane wrote: So the problem is that we have to either disallow such opclass designs, or support per-opclass rather than per-index-AM decisions about whether index-only scans are possible. Just a quick comment, for some queries like the famous select count(*) from table

Re: [HACKERS] SP-GiST versus index-only scans

2011-12-14 Thread Jesper Krogh
On 2011-12-14 19:48, Tom Lane wrote: Jesper Kroghjes...@krogh.cc writes: On 2011-12-14 19:00, Tom Lane wrote: So the problem is that we have to either disallow such opclass designs, or support per-opclass rather than per-index-AM decisions about whether index-only scans are possible. Just a

Re: [HACKERS] Is anybody actually using XLR_BKP_REMOVABLE?

2011-12-12 Thread Jesper Krogh
So: is there actually any such compression program out there? Would anybody really cry if this flag went away? Perhaps http://pglesslog.projects.postgresql.org/ Jesper

[HACKERS] Accounting for toast in query planner. (gin/gist indexes).

2011-11-30 Thread Jesper Krogh
Hi list. I have currently hit a problem which I dug into finding the cause for, in particular, searching in GIN indices seems in some situations to un-fairly favor Sequential Scans. Googling a bit I found this page: http://postgis.refractions.net/docs/ch06.html#id2635817 Describing the excact

[HACKERS] ginfastupdate.. slow

2011-09-15 Thread Jesper Krogh
Hi List. This is just an observation I'll try to reproduce it in a test set later. I've been trying to performancetune a database system which does a lot of updates on GIN indexes. I currently have 24 workers running executing quite cpu-intensive stored procedures that helps generate the body

Re: [HACKERS] tsvector concatenation - backend crash

2011-08-27 Thread Jesper Krogh
On 2011-08-26 23:02, Tom Lane wrote: AFAICT this is a red herring: the bug exists all the way back to where tsvector_concat was added, in 8.3. I think the reason that your test case happens to not crash before this commit is that it changed the sort ordering rules for lexemes. As you can see

[HACKERS] tsvector concatenation - backend crash

2011-08-25 Thread Jesper Krogh
Hi Attached SQL files gives (at least in my hands) a reliable backend crash with this stacktrace .. reproduced on both 9.0.4 and HEAD. I'm sorry I cannot provide a more trimmed down set of vectors the reproduces the bug, thus the obsfucated dataset. But even deleting single terms in the

Re: [HACKERS] tsvector concatenation - backend crash

2011-08-25 Thread Jesper Krogh
On 2011-08-26 05:28, Tom Lane wrote: Jesper Kroghjes...@krogh.cc writes: Attached SQL files gives (at least in my hands) a reliable backend crash with this stacktrace .. reproduced on both 9.0.4 and HEAD. I'm sorry I cannot provide a more trimmed down set of vectors the reproduces the bug,

Re: [HACKERS] mosbench revisited

2011-08-08 Thread Jesper Krogh
On 2011-08-08 15:29, Robert Haas wrote: On Sat, Aug 6, 2011 at 2:16 PM, Dimitri Fontainedimi...@2ndquadrant.fr wrote: Robert Haasrobertmh...@gmail.com writes: It would be nice if the Linux guys would fix this problem for us, but I'm not sure whether they will. For those who may be curious,

Re: [HACKERS] WIP: Fast GiST index build

2011-06-25 Thread Jesper Krogh
On 2011-06-06 09:42, Heikki Linnakangas wrote: took about 15 hours without the patch, and 2 hours with it. That's quite dramatic. With the precense of robust consumer-class SSD-drives that can be found in sizes where they actually can fit many database usage scenarios. A PostgreSQL version is

Re: [HACKERS] FOR KEY LOCK foreign keys

2011-06-20 Thread Jesper Krogh
On 2011-06-20 22:11, Noah Misch wrote: On Sun, Jun 19, 2011 at 06:30:41PM +0200, Jesper Krogh wrote: I hope this hasn't been forgotten. But I cant see it has been committed or moved into the commitfest process? If you're asking about that main patch for $SUBJECT rather than those

Re: [HACKERS] FOR KEY LOCK foreign keys

2011-06-19 Thread Jesper Krogh
I hope this hasn't been forgotten. But I cant see it has been committed or moved into the commitfest process? Jesper On 2011-03-11 16:51, Noah Misch wrote: On Fri, Feb 11, 2011 at 02:13:22AM -0500, Noah Misch wrote: Automated tests would go a long way toward building confidence that this

Re: [HACKERS] pg_upgrade using appname to lock out other users

2011-06-14 Thread Jesper Krogh
On 2011-06-15 05:01, Bruce Momjian wrote: You might remember we added a postmaster/postgres -b switch to indicate binary upgrade mode. The attached patch prevents any client without an application_name of 'binary-upgrade' from connecting to the cluster while it is binary upgrade mode. This

Re: [HACKERS] crash-safe visibility map, take five

2011-05-10 Thread Jesper Krogh
On 2011-05-10 14:48, Robert Haas wrote: We could avoid all of this complexity - and the possibility of pinning the visibility map page needlessly - by locking the heap buffer first and then pinning the visibility map page if the heap page is all-visible. However, that would involve holding the

Re: [HACKERS] the big picture for index-only scans

2011-05-10 Thread Jesper Krogh
On 2011-05-11 01:54, Greg Stark wrote: To be fair about 3/4 of them were actually complaining about the lack of some global materialized cache of the aggregate value. Covering index-only scans are only going to be a linear speedup no matter how large the factor it's not going to turn select

Re: [HACKERS] Unlogged tables, persistent kind

2011-04-25 Thread Jesper Krogh
On 2011-04-25 20:00, Leonardo Francalanci wrote: The amount of data loss on a big table will be 1% of the data loss caused by truncating the whole table. If that 1% is random (not time/transaction related), usually you'd rather have an empty table. In other words: is a table that is not

Re: [HACKERS] Evaluation of secondary sort key.

2011-04-18 Thread Jesper Krogh
On 2011-04-18 11:00, Greg Stark wrote: On Mon, Apr 18, 2011 at 6:25 AM, Jesper Kroghjes...@krogh.cc wrote: Getting the value for the first sortkey and carrying on a closure for the rest would mostly (very often) be optimal ? Well that might depend. The input data to the function might be much

Re: [HACKERS] Evaluation of secondary sort key.

2011-04-17 Thread Jesper Krogh
On 2011-04-09 18:54, Tom Lane wrote: I think that would be a positive disimprovement. The current design guarantees that volatile sort expressions are evaluated exactly once, in the order the rows are read from the data source. There would be no guarantees at all, either as to the number of

[HACKERS] Locking when concurrent updated of foreign references

2011-04-11 Thread Jesper Krogh
Hi. This seem a bit strange to me. In short: 2 tables, one with has a foreign key to the other one; CREATE TABLE test (id SERIAL primary key, data text); CREATE TABLE testref(id SERIAL primary key, test_id integer references test(id) not null, data text); INSERT INTO test(data)

Re: [HACKERS] Locking when concurrent updated of foreign references

2011-04-11 Thread Jesper Krogh
On 2011-04-11 20:18, Jesper Krogh wrote: Hi. This seem a bit strange to me. In short: Not any more I.. I guess what made me a bit confused was that a update table set key = value would acually block out changes on tables referencing this tuple even if the referenced column wasn't effected

Re: [HACKERS] Locking when concurrent updated of foreign references

2011-04-11 Thread Jesper Krogh
On 2011-04-11 23:30, Alvaro Herrera wrote: Excerpts from Jesper Krogh's message of lun abr 11 17:07:33 -0300 2011: But when the locking is done row-level then it is correct to do it that way. It would allthough be nice with a weaker locklevel for that kind of updates (I have no clue if that is

Re: [HACKERS] k-neighbourhood search in databases

2011-04-10 Thread Jesper Krogh
On 2011-04-10 12:18, Oleg Bartunov wrote: Wow, custom solution for 2008 still much faster Denali 2011 solution. Also, what's about not spatial data types ? In our approach, we can provide knn for any datatype, which has GiST index and distance method. Can you share some insight about how it

[HACKERS] Evaluation of secondary sort key.

2011-04-09 Thread Jesper Krogh
This seems like a place where there is room for improvement. 2011-04-09 15:18:08.016 testdb=# select id from test1 where id 3 order by id; id 1 2 (2 rows) Time: 0.328 ms 2011-04-09 15:18:11.936 testdb=# CREATE or Replace FUNCTION testsort(id integer) returns integer as $$ BEGIN

Re: [HACKERS] Evaluation of secondary sort key.

2011-04-09 Thread Jesper Krogh
How would PostgreSQL know that sorting by id leaves no ambiguity for the next key to address? It wouldn't But it could postpone evaluation until ambiguity was actually met. Jesper -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Evaluation of secondary sort key.

2011-04-09 Thread Jesper Krogh
On 2011-04-09 20:00, David Fetter wrote: Given the horrors query generators perpetrate, it might be worth dropping provably redundant ORDER BYs on the floor at planning time. Well, many people often add a secondary sort-key to their SQL for the only purpose of obtainting a consistent result in

Re: [HACKERS] Evaluation of secondary sort key.

2011-04-09 Thread Jesper Krogh
On 2011-04-09 18:54, Tom Lane wrote: I think that would be a positive disimprovement. The current design guarantees that volatile sort expressions are evaluated exactly once, in the order the rows are read from the data source. There would be no guarantees at all, either as to the number of

[HACKERS] Visibillity testing - some numbers on current performance.

2011-04-05 Thread Jesper Krogh
Hi. I initially set out to put some numbers on why the visibillity map was important for select count(*), primarily to give some feedback to Simon Riggs stating: Your tests and discussion remind me that I haven't yet seen any tests that show that index-only scans would be useful for performance.

Re: [HACKERS] crash-safe visibility map, take four

2011-03-23 Thread Jesper Krogh
On 2011-03-22 21:43, Robert Haas wrote: I took a crack at implementing the first approach described above, which seems to be by far the simplest idea we've come up with to date. Patch attached. It doesn't seem to be that complicated, which could mean either that it's not that complicated or

Re: [HACKERS] really lazy vacuums?

2011-03-17 Thread Jesper Krogh
Robert Haas wrote: Right. Really-lazy vacuum could freeze tuples. Unlike regular vacuum, it can also sensibly be done incrementally. One thing I was thinking about is counting the number of times that we fetched a tuple that was older than RecentGlobalXmin and had a committed xmin and an

Re: [HACKERS] really lazy vacuums?

2011-03-17 Thread Jesper Krogh
On 2011-03-17 15:02, Robert Haas wrote: On Thu, Mar 17, 2011 at 4:17 AM, Jesper Kroghjes...@krogh.cc wrote: Is it obvious that the visibillity map bits should track complete pages and not individual tuples? If the visibillity map tracks at page-level the benefit would fall on slim tables where

Re: [HACKERS] Read uncommitted ever possible?

2011-03-10 Thread Jesper Krogh
On 2011-03-10 18:00, Bruce Momjian wrote: Dirty reads are unlikely to be implemented. We do have a TODO item and wiki page about how to allow index scans without heap access: http://wiki.postgresql.org/wiki/Index-only_scans I think we (the company I work for) would help co-sponsor such a

Re: [HACKERS] Read uncommitted ever possible?

2011-03-10 Thread Jesper Krogh
On 2011-03-10 19:25, Bruce Momjian wrote: Sure, anyone can add text to that wiki; you create a community account here: http://www.postgresql.org/community/signup Suggestion: Add this url to the login box on the wiki. -- Jesper -- Sent via pgsql-hackers mailing list

Re: [HACKERS] tsearch Parser Hacking

2011-02-17 Thread Jesper Krogh
On 16 Feb 2011, at 23:22, David E. Wheeler da...@kineticode.com wrote: On Feb 14, 2011, at 11:44 PM, Oleg Bartunov wrote: IMO, sooner or later we need to trash that code and replace it with something a bit more modification-friendly. We thought about configurable parser, but AFAIR, we

[HACKERS] Estimates not taking null_frac element into account with @@ operator? (8.4 .. git-head)

2011-02-17 Thread Jesper Krogh
to align the behaviour Thanks. -- Jesper From 4b4be27864f683a9b3464b86d6cfa567f8ab6bd2 Mon Sep 17 00:00:00 2001 From: Jesper Krogh j...@novozymes.com Date: Thu, 17 Feb 2011 22:21:52 +0100 Subject: [PATCH] Take null_frac into account in calculating selectivity for @@ operator. This makes behaviour

Re: [HACKERS] Estimates not taking null_frac element into account with @@ operator? (8.4 .. git-head)

2011-02-17 Thread Jesper Krogh
On 2011-02-17 23:20, Tom Lane wrote: Jesper Kroghjes...@krogh.cc writes: When something evaluates to null isn't included in the result, shouldn't the query-planner then take the null_frac into account when computing the estimate? The proposed patch seems wrong to me: if we're estimating on

Re: [HACKERS] wildcard search support for pg_trgm

2011-01-24 Thread Jesper Krogh
On 2011-01-24 16:34, Alexander Korotkov wrote: Hi! On Mon, Jan 24, 2011 at 3:07 AM, Jan Urbańskiwulc...@wulczer.org wrote: I see two issues with this patch. First of them is the resulting index size. I created a table with 5 copies of /usr/share/dict/american-english in it and a gin index on

Re: [HACKERS] crash-safe visibility map, take three

2011-01-05 Thread Jesper Krogh
On 2010-11-30 05:57, Robert Haas wrote: Last week, I posted a couple of possible designs for making the visibility map crash-safe, which did not elicit much comment. Since this is an important prerequisite to index-only scans, I'm trying again. The logic seems to be: * If the visibillity map

Re: [HACKERS] crash-safe visibility map, take three

2011-01-05 Thread Jesper Krogh
On 2011-01-06 03:10, Robert Haas wrote: On Wed, Jan 5, 2011 at 3:22 PM, Jesper Kroghjes...@krogh.cc wrote: Given a crash-safe visibility map, what purpuse does the PD_ALL_VISIBLE bit serve? If we modify a page on which PD_ALL_VISIBLE isn't set, we don't attempt to update the visibility map.

[HACKERS] The cost of visibillity testing? (gin-search)

2010-12-21 Thread Jesper Krogh
Hi Hackers. I have a feeling that GIN is cheating on the visibillity checks: test=# set enable_seqscan = off; SET Time: 0.129 ms test=# select count(id) from fts_test where fts @@ to_tsquery('core'); count 158827 (1 row) Time: 95.530 ms test=# explain select count(id) from fts_test

Re: [HACKERS] The cost of visibillity testing? (gin-search)

2010-12-21 Thread Jesper Krogh
On 2010-12-21 21:28, Andres Freund wrote: On Tuesday 21 December 2010 20:25:16 Jesper Krogh wrote: What have I missed in the logic? A reproducible testcase ;-) Yes, I did a complete dump/restore of the dataset and the numbers looked like expected. So table bloat seems

[HACKERS] window function count(*) and limit

2010-10-23 Thread Jesper Krogh
Hi. I have been puzzled about the evaluation order when using window functions and limit. jk=# select * from testtable; id | value +--- 1 | 1 2 | 2 3 | 3 4 | 4 5 | 5 6 | 6 7 | 7 8 | 8 9 | 9 10 |10 (10 rows) jk=# select

Re: [HACKERS] window function count(*) and limit

2010-10-23 Thread Jesper Krogh
On 2010-10-23 18:42, Tom Lane wrote: Jesper Kroghjes...@krogh.cc writes: I have been puzzled about the evaluation order when using window functions and limit. It's basically FROM - WHERE - window functions - LIMIT. I expected it to either count to 3 or blow up and tell me

[HACKERS] Make tuples_per_page pr. table configureable.

2010-09-21 Thread Jesper Krogh
Hi. This is a follow up and updated patch on several old discussions: http://archives.postgresql.org/pgsql-hackers/2009-07/msg01065.php http://archives.postgresql.org/pgsql-admin/2010-04/msg00164.php http://archives.postgresql.org/pgsql-hackers/2009-06/msg00831.php First patch:

Re: [HACKERS] Make tuples_per_page pr. table configureable.

2010-09-21 Thread Jesper Krogh
On 2010-09-22 04:33, Itagaki Takahiro wrote: On Wed, Sep 22, 2010 at 3:44 AM, Jesper Kroghjes...@krogh.cc wrote: Command to set tuples_per_page is: ALTER TABLEtablename set (tuples_per_page = X) where 1= X= 32. The tuples_per_page means *minimal* number of tuples in a page, right?

Re: [HACKERS] Configuring synchronous replication

2010-09-17 Thread Jesper Krogh
On 2010-09-17 10:09, Heikki Linnakangas wrote: I think it makes most sense to set sync vs. async in the master, and the level of synchronicity in the slave. Although I have sympathy for the argument that it's simpler if you configure it all from the master side as well. Just a comment as a

Re: [HACKERS] SHOW TABLES

2010-07-15 Thread Jesper Krogh
On 2010-07-15 18:07, Marc G. Fournier wrote: On Thu, 15 Jul 2010, Thom Brown wrote: If it's only a psql problem, why implement it as SQL? Is it just so we're not adding keywords specifically to psql? In that case, it shouldn't support QUIT. Personally, I think this is somethign that

Re: [HACKERS] pessimal trivial-update performance

2010-07-05 Thread Jesper Krogh
On 2010-07-04 06:11, Tom Lane wrote: Robert Haasrobertmh...@gmail.com writes: CREATE OR REPLACE FUNCTION update_tab() RETURNS void AS $$ BEGIN INSERT INTO tab VALUES (0); FOR i IN 1..10 LOOP UPDATE tab SET x = x + 1; END LOOP; END $$ LANGUAGE

Re: [HACKERS] pessimal trivial-update performance

2010-07-05 Thread Jesper Krogh
On 2010-07-05 12:11, Pierre C wrote: The problem can generally be written as tuples seeing multiple updates in the same transaction? I think that every time PostgreSQL is used with an ORM, there is a certain amount of multiple updates taking place. I have actually been reworking

Re: [HACKERS] Admission Control

2010-06-28 Thread Jesper Krogh
On 2010-06-25 22:44, Robert Haas wrote: On Fri, Jun 25, 2010 at 3:52 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Heck, I think an even *more* trivial admission control policy which limits the number of active database transactions released to execution might solve a lot of

Re: [HACKERS] Admission Control

2010-06-28 Thread Jesper Krogh
On 2010-06-28 21:24, Kevin Grittner wrote: Jesper Kroghjes...@krogh.cc wrote: Sorry if I'm asking silly questions, but how does transactions and connection pooler's interact? That depends a great deal on the pooler and its configuration, as well as your client architecture. Our

Re: [HACKERS] tsvector pg_stats seems quite a bit off.

2010-05-31 Thread Jesper Krogh
On 2010-05-30 20:02, Jan Urbański wrote: Here's a patch against recent git, but should apply to 8.4 sources as well. It would be interesting to measure the memory and time needed to analyse the table after applying it, because we will be now using a lot bigger bucket size and I haven't done any

[HACKERS] bitmap-index-scan faster than seq-scan on full-table-scan (gin index)

2010-05-31 Thread Jesper Krogh
Hi. The test data a set of generated terms using this perl-script http://shrek.krogh.cc/~jesper/build-test.pl and http://shrek.krogh.cc/~jesper/words.txt I have generated a test dataset with an average tsvector length of around 250 and 200.000 tuples in the dataset. Conceptually searching for

Re: [HACKERS] bitmap-index-scan faster than seq-scan on full-table-scan (gin index)

2010-05-31 Thread Jesper Krogh
On 2010-05-31 22:09, Tom Lane wrote: Jesper Kroghjes...@krogh.cc writes: Conceptually searching for the full dataset would always be fastest solved by a seq-scan. The query planner enforces this so much, so not even enable_seqscan=off can convince it to to something else. ... Would it be

Re: [HACKERS] tsvector pg_stats seems quite a bit off.

2010-05-31 Thread Jesper Krogh
On 2010-05-31 20:38, Tom Lane wrote: Jesper Kroghjes...@krogh.cc writes: Just a small follow up. I tried out the patch (or actually a fresh git checkout) and it now gives very accurate results for both upper and lower end of the MCE-histogram with a lower cutoff that doesn't approach 2.

Re: [HACKERS] tsvector pg_stats seems quite a bit off.

2010-05-30 Thread Jesper Krogh
On 2010-05-29 15:56, Jan Urbański wrote: On 29/05/10 12:34, Jesper Krogh wrote: On 2010-05-28 23:47, Jan Urbański wrote: On 28/05/10 22:22, Tom Lane wrote: Now I tried to substitute some numbers there, and so assuming the English language has ~1e6 words H(W) is around 6.5. Let's

Re: [HACKERS] tsvector pg_stats seems quite a bit off.

2010-05-29 Thread Jesper Krogh
On 2010-05-28 23:47, Jan Urbański wrote: On 28/05/10 22:22, Tom Lane wrote: The idea that I was toying with is to assume a Zipfian distribution of the input (with some reasonable parameter), and use that to estimate what the frequency of the K'th element will be, where K is the target

Re: [HACKERS] tsvector pg_stats seems quite a bit off.

2010-05-29 Thread Jesper Krogh
On 2010-05-28 04:47, Tom Lane wrote: Cranking up the stats target actually makes it worse not better, since low-frequency items are then more likely to get into the MCV list I should have been more precise in the wording. Cranking up the stats target gave me overall a better plan, but that

[HACKERS] Statistics for tsvector wildcards. term*

2010-05-29 Thread Jesper Krogh
Hi. There seems to be an unimplemented area around getting statistics for wildcard searches done. Wildcards anchored to the left can be matched up by the gin-index and the ts_match_vq operator: testdb=# select to_tsvector('project') @@ to_tsquery('proj:*'); ?column? -- t (1 row)

Re: [HACKERS] Idea for getting rid of VACUUM FREEZE on cold pages

2010-05-27 Thread Jesper Krogh
On 27/05/2010, at 20.00, Josh Berkus j...@agliodbs.com wrote: Well, maybe I'm confused here, but arranging things so that we NEVER have to visit the page after initially writing it seems like it's setting the bar almost impossibly high. That is the use case, though. What I've encountered

Re: [HACKERS] Idea for getting rid of VACUUM FREEZE on cold pages

2010-05-26 Thread Jesper Krogh
On 27/05/2010, at 02.48, Robert Haas robertmh...@gmail.com wrote: On Wed, May 26, 2010 at 8:06 PM, Tom Lane t...@sss.pgh.pa.us wrote: Josh Berkus j...@agliodbs.com writes: How does that get us out of reading and writing old pages, though? Yeah. Neither PD_ALL_VISIBLE nor the visibility

Re: [HACKERS] tsvector pg_stats seems quite a bit off.

2010-05-25 Thread Jesper Krogh
On 26/05/2010, at 01.16, Jan Urbański wulc...@wulczer.org wrote: On 19/05/10 21:01, Jesper Krogh wrote: The document base is arount 350.000 documents and I have set the statistics target on the tsvector column to 1000 since the 100 seems way of. So for tsvectors the statistics target means

Re: [HACKERS] pg_upgrade - link mode and transaction-wraparound data loss

2010-05-19 Thread Jesper Krogh
On 2010-05-18 18:57, Bruce Momjian wrote: jes...@krogh.cc wrote: Hi I tried running pg_upgrade from the current snapshot of postgresql and upgrading from 8.4.4 to the snapshot version. Everything seem to look fine in the process and all that came out was only ok's but when I tried a simple

[HACKERS] tsvector pg_stats seems quite a bit off.

2010-05-19 Thread Jesper Krogh
Hi. I am working on getting full-text-search to work and have come across something I think look a bit strange. The document base is arount 350.000 documents and I have set the statistics target on the tsvector column to 1000 since the 100 seems way of. # ANALYZE verbose reference

Re: [HACKERS] pg_upgrade - link mode and transaction-wraparound data loss

2010-05-18 Thread Jesper Krogh
On 2010-05-18 20:52, Bruce Momjian wrote: This line above looks very odd because I didn't think the template0 datfrozenxid could be advanced. Can I see the output of this query: SELECT datname, datfrozenxid, datallowconn FROM pg_database; Only from the old database: data=#

[HACKERS] BYTEA / DBD::Pg change in 9.0 beta

2010-05-18 Thread Jesper Krogh
Hi. I'm trying to do a test move of one of our applications onto 9.0beta1. We use storable and serializes data into a bytea column in the database. This script uses that: #!/usr/bin/perl use strict; use warnings; use Storable; use DBI; use DBD::Pg; use Data::Dumper; my $dbh =

Re: [HACKERS] pg_upgrade - link mode and transaction-wraparound data loss

2010-05-18 Thread Jesper Krogh
On 2010-05-18 21:56, Bruce Momjian wrote: Jesper Krogh wrote: On 2010-05-18 20:52, Bruce Momjian wrote: This line above looks very odd because I didn't think the template0 datfrozenxid could be advanced. Can I see the output of this query: SELECT datname, datfrozenxid

Re: [HACKERS] BYTEA / DBD::Pg change in 9.0 beta

2010-05-18 Thread Jesper Krogh
On 2010-05-18 23:12, Alex Hunsaker wrote: set bytea_output 'escape'; That was it. Knowing what the problem was I had no problem finding it in the release notes. May I ask whats the reason is for breaking the compatibillity? -- Jesper -- Sent via pgsql-hackers mailing list

Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta

2010-05-05 Thread Jesper Krogh
On 2010-05-03 23:09, Bruce Momjian wrote: Robert Haas wrote: On Sun, May 2, 2010 at 3:45 PM, Dimitri Fontainedfonta...@hi-media.com wrote: Now you tell me how awful this idea really is :) I'm not sure I can count that high. :-) While I can't improve on Robert's

Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta

2010-05-05 Thread Jesper Krogh
On 2010-05-06 01:45, Bruce Momjian wrote: Jesper Krogh wrote: On 2010-05-03 23:09, Bruce Momjian wrote: Robert Haas wrote: On Sun, May 2, 2010 at 3:45 PM, Dimitri Fontainedfonta...@hi-media.com wrote: Now you tell me how awful this idea really

Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta

2010-05-05 Thread Jesper Krogh
On 2010-05-06 06:41, Alvaro Herrera wrote: Excerpts from Jesper Krogh's message of jue may 06 00:32:09 -0400 2010: Q: I read you pdf, why isn't statistics copied over? It seems to be the last part missing from doing an upgrade in a few minutes. Seems fraught with peril, and a bit

[HACKERS] Using GIN/Gist to search the union of two indexes?

2010-03-05 Thread Jesper Krogh
Hi. How complicated would it be to make postgresql-fts search the union of several GIN/Gist indexes. The use-case is that you have two tables: tablea(id serial, tableb_id int, text tsvector); and tableb(id serial, text tsvector); and indices on both tsvectors. The typical query would join the

[HACKERS] Make TOAST_TUPLES_PER_PAGE configurable per table.

2010-02-01 Thread Jesper Krogh
data then this patch can enable you to teach postgresql to take advantage of that. In my situation I would estimate that the production set would be able to drop a couple of GB from main memory (leaving room for more index-pages and such). Thanks in advance. -- Jesper Krogh diff -rc

Re: [HACKERS] Make TOAST_TUPLES_PER_PAGE configurable per table.

2010-02-01 Thread Jesper Krogh
Tom Lane wrote: Jesper Krogh jes...@krogh.cc writes: This patch enables users to set TOAST_TUPLES_PER_PAGE with ALTER TABLE table SET (tuples_per_page = X); .. currently with 1 = X = 32; It's not clear to me that fiddling with that is useful unless the toast tuple size also changes