Re: [HACKERS] [NOVICE] WHERE clause not used when index is used

2016-03-01 Thread Jeff Janes
On Tue, Mar 1, 2016 at 7:40 AM, Tom Lane wrote: > Tobias Florek writes: >> When creating an index to use for an ORDER BY clause, a simple query >> starts to return more results than expected. See the following detailed >> log. > > Ugh. That is *badly* broken. I thought maybe it had something t

Re: [HACKERS] Random inconsistencies in GiST support function declarations

2016-03-01 Thread Jeff Janes
On Mon, Jan 18, 2016 at 2:29 PM, Tom Lane wrote: > > Fixing the pg_proc entries in HEAD seems like no big deal, but some of > the errors are in contrib modules. If we wanted to be really clean > about that, we'd have to bump those modules' extension versions, which > is a pain in the rear. Since

[HACKERS] pg_basebackup compression TODO item

2016-03-03 Thread Jeff Janes
Since SSL compression seems to be a busted flush, I would like to see pg_basebackup be able to do compression on the server end, not just the client end, in order to spare network bandwidth. Any comments on how hard this would be, or why we don't want it? Cheers, Jeff -- Sent via pgsql-hacker

Re: [HACKERS] More stable query plans via more predictable column statistics

2016-03-07 Thread Jeff Janes
On Mon, Mar 7, 2016 at 3:17 AM, Shulgin, Oleksandr wrote: > On Fri, Mar 4, 2016 at 7:27 PM, Robert Haas wrote: >> >> On Thu, Mar 3, 2016 at 2:48 AM, Shulgin, Oleksandr >> wrote: >> > On Wed, Mar 2, 2016 at 7:33 PM, Alvaro Herrera >> > >> > wrote: >> >> Shulgin, Oleksandr wrote: >> >> >> >> > Al

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-08 Thread Jeff Janes
On Tue, Mar 8, 2016 at 5:30 AM, Robert Haas wrote: > On Tue, Mar 8, 2016 at 7:26 AM, Masahiko Sawada wrote: >> Regarding pg_visibility module, I'd like to share some bugs and >> propose to add a relation type condition to each functions. > > OK, thanks. > >> Including it, I've attached remaining

Re: [HACKERS] multivariate statistics v11

2016-03-08 Thread Jeff Janes
On Tue, Mar 8, 2016 at 12:13 PM, Tomas Vondra wrote: > Hi, > > attached is v11 of the patch - this is mostly a cleanup of v10, removing > redundant code, adding missing comments, removing obsolete FIXME/TODOs > and so on. Overall this shaves ~20kB from the patch (not a primary > objective, though)

Re: [HACKERS] multivariate statistics v14

2016-03-09 Thread Jeff Janes
On Wed, Mar 9, 2016 at 7:02 AM, Tomas Vondra wrote: > Hi, > > thanks for the feedback. Attached is v14 of the patch series, fixing > most of the points you've raised. Hi Tomas, Applied to aa09cd242fa7e3a694a31f, I still get the seg faults in make check if I configure without --enable-cassert.

Re: [HACKERS] pgcrypto: add s2k-count

2016-03-09 Thread Jeff Janes
On Tue, Mar 8, 2016 at 4:09 PM, Alvaro Herrera wrote: > Jeff Janes wrote: >> pgcrypto supports s2k-mode for key-stretching during symmetric >> encryption, and even defaults to s2k-mode=3, which means configurable >> iterations. But it doesn't support s2k-count to actual

Re: [HACKERS] multivariate statistics v14

2016-03-09 Thread Jeff Janes
On Wed, Mar 9, 2016 at 9:21 AM, Tomas Vondra wrote: > Hi, > > On Wed, 2016-03-09 at 08:45 -0800, Jeff Janes wrote: >> On Wed, Mar 9, 2016 at 7:02 AM, Tomas Vondra >> wrote: >> > Hi, >> > >> > thanks for the feedback. Attached is v14 of the pat

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-12 Thread Jeff Janes
On Thu, Mar 10, 2016 at 11:25 PM, Peter Geoghegan wrote: > On Thu, Mar 10, 2016 at 11:18 PM, Fabien COELHO wrote: >> I can only concur! >> >> The "Performance Tips" chapter (II.14) is more user/query oriented. The >> "Server Administration" bool (III) does not discuss this much. > > That's defini

Re: [HACKERS] multivariate statistics v14

2016-03-12 Thread Jeff Janes
On Wed, Mar 9, 2016 at 9:21 AM, Tomas Vondra wrote: > Hi, > > On Wed, 2016-03-09 at 08:45 -0800, Jeff Janes wrote: >> On Wed, Mar 9, 2016 at 7:02 AM, Tomas Vondra >> wrote: >> > Hi, >> > >> > thanks for the feedback. Attached is v14 of the pat

Re: [HACKERS] memory leak in GIN

2016-03-13 Thread Jeff Janes
On Fri, Mar 11, 2016 at 11:40 PM, Jaime Casanova wrote: > Hi, > > On the spanish list, Felipe de Jesús Molina Bravo, reported a few days > back that a query that worked well in 9.4 consume all memory in 9.5. > With the self contained test he provided us i reproduced the problem > in 9.5 and 9.6dev

Re: [HACKERS] Weighted Stats

2016-03-18 Thread Jeff Janes
On Tue, Mar 15, 2016 at 8:36 AM, David Fetter wrote: > > Please find attached a patch that uses the float8 version to cover the > numeric types. Is there a well-defined meaning for having a negative weight? If no, should it be disallowed? I don't know what I was expecting, but not this: selec

Re: [HACKERS] Fuzzy substring searching with the pg_trgm extension

2016-03-18 Thread Jeff Janes
On Mon, Mar 14, 2016 at 9:27 AM, Artur Zakirov wrote: > On 14.03.2016 18:48, David Steele wrote: >> >> Hi Jeff, >> >> On 2/25/16 5:00 PM, Jeff Janes wrote: >> >>> But, It doesn't sound like I am going to win that debate. Given that, >>>

Re: [HACKERS] Idle In Transaction Session Timeout, revived

2016-03-18 Thread Jeff Janes
On Wed, Mar 16, 2016 at 8:32 AM, Robert Haas wrote: > > Committed with slight changes to the docs, and I added a flag variable > instead of relying on IdleInTransactionSessionTimeout not changing at > an inopportune time. Thanks for committing, this should be a useful feature. I get a pretty str

[HACKERS] logger process infinite loop

2016-03-18 Thread Jeff Janes
While testing some patches on my laptop, I noticed my knee getting uncomfortably warm. It turns out I has accumulating deranged logging processes, needing kill -9 to get rid of them. The culprit is: commit c4901a1e03a7730e4471fd1143f1caf79695493d Author: Andres Freund Date: Fri Mar 18 11:43:5

Re: [HACKERS] snapshot too old, configured by time

2016-03-18 Thread Jeff Janes
On Thu, Mar 17, 2016 at 2:15 PM, Kevin Grittner wrote: > New patch just to merge in recent commits -- it was starting to > show some bit-rot. Tests folded in with main patch. I'm not sure if this is operating as expected. I set the value to 1min. I set up a test like this: pgbench -i pgbench

Re: [HACKERS] logger process infinite loop

2016-03-18 Thread Jeff Janes
On Fri, Mar 18, 2016 at 10:39 PM, Andres Freund wrote: > Hi, > > On 2016-03-18 21:59:01 -0700, Jeff Janes wrote: >> While testing some patches on my laptop, I noticed my knee getting >> uncomfortably warm. It turns out I has accumulating deranged logging >> processes,

Re: [HACKERS] Weighted Stats

2016-03-19 Thread Jeff Janes
On Fri, Mar 18, 2016 at 11:34 PM, David Fetter wrote: > On Fri, Mar 18, 2016 at 06:12:12PM -0700, Jeff Janes wrote: >> Also, I think it might not give the correct answer even without >> negative weights: >> >> create table foo as select floor(random()*1)::int

Re: [HACKERS] pgbench stats per script & other stuff

2016-03-19 Thread Jeff Janes
On Sat, Mar 19, 2016 at 11:34 AM, Alvaro Herrera wrote: > Jeff Janes wrote: >> On Sat, Mar 19, 2016 at 8:41 AM, Alvaro Herrera >> wrote: >> > I pushed your 25, with some additional minor tweaks. I hope I didn't >> > break anything; please test. >> >

Re: [HACKERS] pgbench stats per script & other stuff

2016-03-19 Thread Jeff Janes
On Fri, Jul 17, 2015 at 6:50 AM, Fabien wrote: > > This patch adds per-script statistics & other improvements to pgbench > > Rationale: Josh asked for the per-script stats:-) > > Some restructuring is done so that all stats (-l --aggregate-interval > --progress --per-script-stats, latency & lag...

Re: [HACKERS] multivariate statistics v14

2016-03-21 Thread Jeff Janes
On Sun, Mar 20, 2016 at 4:34 PM, Tomas Vondra wrote: > > > D'oh. Thanks for reporting. Attached is v16, hopefully fixing the few > remaining whitespace issues. Hi Tomas, I'm trying out v16 against a common problem, where postgresql thinks it is likely top stop early during a "order by (index exp

[HACKERS] trivial typo in vacuum progress doc

2016-03-21 Thread Jeff Janes
"skipped blocks are included this total" The word "in" is missing Cheers, Jeff diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml new file mode 100644 index cb22afb..105d541 *** a/doc/src/sgml/monitoring.sgml --- b/doc/src/sgml/monitoring.sgml *** SELECT pg_stat

Re: [HACKERS] Draft release notes for next week's releases

2016-03-26 Thread Jeff Janes
ate, but: + + + Correctly handle wraparound cases in the pg_subtrans + startup logic for hot standby (Jeff Janes) + + This applies to all recovery scenarios, whether they are hot standby or just plain-old automatic crash recovery. (However, it does only matter when prepare

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-12 Thread Jeff Janes
On Tue, Jan 12, 2016 at 8:59 AM, Anastasia Lubennikova wrote: > 08.01.2016 00:12, David Rowley: > > On 7 January 2016 at 06:36, Jeff Janes wrote: >> > But now I see the reason to create non-unique index with included columns - > lack of suitable opclass on column "b&

Re: [HACKERS] Fuzzy substring searching with the pg_trgm extension

2016-01-13 Thread Jeff Janes
On Sat, Dec 26, 2015 at 9:12 PM, Jeff Janes wrote: > On Fri, Dec 18, 2015 at 11:43 AM, Artur Zakirov > wrote: >> Hello. >> >> PostgreSQL has a contrib module named pg_trgm. It is used to the fuzzy text >> search. It provides some functions and operators for determ

Re: [HACKERS] Fuzzy substring searching with the pg_trgm extension

2016-01-15 Thread Jeff Janes
On Fri, Dec 18, 2015 at 11:43 AM, Artur Zakirov wrote: > Hello. > > PostgreSQL has a contrib module named pg_trgm. It is used to the fuzzy text > search. It provides some functions and operators for determining the > similarity of the given texts using trigram matching. > > At the moment, in pg_tr

Re: [HACKERS] GIN pending list clean up exposure to SQL

2016-01-15 Thread Jeff Janes
On Sun, Jan 10, 2016 at 4:24 AM, Julien Rouhaud wrote: > On 29/12/2015 00:30, Jeff Janes wrote: >> On Wed, Nov 25, 2015 at 9:29 AM, Jeff Janes wrote: >>> >>> I'll prepare a patch for core for the January commitfest, and see if >>> it flies. If not, ther

Re: [HACKERS] Truncating/vacuuming relations on full tablespaces

2016-01-17 Thread Jeff Janes
On Fri, Jan 15, 2016 at 11:16 AM, Tom Lane wrote: > I believe that there would be ramifications for some of the index AMs > too. For example, if left to its own devices GIN would consider VACUUM > to include flushing its pending-list pages, which more than likely will > increase not reduce the t

Re: [HACKERS] Proposal: speeding up GIN build with parallel workers

2016-01-17 Thread Jeff Janes
On Fri, Jan 15, 2016 at 3:29 PM, Peter Geoghegan wrote: > On Fri, Jan 15, 2016 at 2:38 PM, Constantin S. Pan wrote: >> I have a draft implementation which divides the whole process between >> N parallel workers, see the patch attached. Instead of a full scan of >> the relation, I give each worker

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-21 Thread Jeff Janes
On Tue, Jan 19, 2016 at 9:08 AM, Anastasia Lubennikova wrote: > > > 18.01.2016 01:02, David Rowley пишет: > > On 14 January 2016 at 08:24, David Rowley > wrote: >> >> I will try to review the omit_opclass_4.0.patch soon. > > > Hi, as promised, here's my review of the omit_opclass_4.0.patch patch.

Re: [HACKERS] Combining Aggregates

2016-01-22 Thread Jeff Janes
On Wed, Jan 20, 2016 at 11:06 AM, Robert Haas wrote: > On Wed, Jan 20, 2016 at 7:38 AM, David Rowley > wrote: >> Agreed. So I've attached a version of the patch which does not have any of >> the serialise/deserialise stuff in it. > > I re-reviewed this and have committed most of it with only mino

Re: [HACKERS] Combining Aggregates

2016-01-23 Thread Jeff Janes
On Fri, Jan 22, 2016 at 4:53 PM, David Rowley wrote: > > It seems that I must have mistakenly believed that non-existing > columns for previous versions were handled using the power of magic. > Turns out that I was wrong, and they need to be included as dummy > columns in the queries for previous

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-24 Thread Jeff Janes
On Fri, Jan 22, 2016 at 7:19 AM, Anastasia Lubennikova wrote: > > Done. I hope that my patch is close to the commit too. > Thanks for the update. I've run into this problem: create table foobar (x text, w text); create unique index foobar_pkey on foobar (x) including (w); alter table foobar add

Re: [HACKERS] GIN pending list clean up exposure to SQL

2016-01-24 Thread Jeff Janes
On Wed, Jan 20, 2016 at 6:17 AM, Fujii Masao wrote: > On Sat, Jan 16, 2016 at 7:42 AM, Julien Rouhaud > wrote: >> On 15/01/2016 22:59, Jeff Janes wrote: >>> On Sun, Jan 10, 2016 at 4:24 AM, Julien Rouhaud >>> wrote: >> >> All looks fine to me, I

[HACKERS] brin_summarize_new_values error checking

2016-01-24 Thread Jeff Janes
In reviewing one of my patches[1], Fujii-san has pointed out that I didn't include checks for being in recovery, or for working on another backend's temporary index. I think that brin_summarize_new_values in 9.5.0 commits those same sins. In its case, I don't think those are critical, as they just

Re: [HACKERS] GIN pending list clean up exposure to SQL

2016-02-07 Thread Jeff Janes
On Wed, Jan 27, 2016 at 12:54 PM, Julien Rouhaud wrote: > On 27/01/2016 10:27, Fujii Masao wrote: >> >> Thanks for updating the patch! It looks good to me. >> >> Based on your patch, I just improved the doc. For example, I added >> the following note into the doc. >> >> +These functions cannot

[HACKERS] Bug in StartupSUBTRANS

2016-02-09 Thread Jeff Janes
While testing the crash resilience of the recent 2-part-commit improvements, I've run into a problem where sometimes after a crash the recovery process creates zeroed files in pg_subtrans until it exhausts all disk space. Looking at the code, it looks like it does not anticipate that the xid might

[HACKERS] pgcrypto: add s2k-count

2016-02-09 Thread Jeff Janes
pgcrypto supports s2k-mode for key-stretching during symmetric encryption, and even defaults to s2k-mode=3, which means configurable iterations. But it doesn't support s2k-count to actually set those iterations to be anything other than the default. If you are interested in key-stretching, the de

Re: [HACKERS] postgres_fdw join pushdown (was Re: Custom/Foreign-Join-APIs)

2016-02-09 Thread Jeff Janes
On Tue, Feb 9, 2016 at 11:16 AM, Robert Haas wrote: > On Tue, Feb 9, 2016 at 8:39 AM, Ashutosh Bapat > wrote: >> Thanks Jeevan for your review and comments. PFA the patch which fixes those. > > Committed with a couple more small adjustments. I'm getting a compiler warning which I think is coming

[HACKERS] Seg fault in pgbench

2016-02-12 Thread Jeff Janes
If I give pgbench an empty file, I get a segfault. $ touch empty.sql $ src/bin/pgbench/pgbench -T 60 -f empty.sql starting vacuum...end. Segmentation fault (core dumped) This has been since this commit: commit 8bea3d2219844887e170471f223ba100b3c17571 Author: Alvaro Herrera Date: Wed Jan 27 02

Re: [HACKERS] Seg fault in pgbench

2016-02-12 Thread Jeff Janes
On Fri, Feb 12, 2016 at 12:22 PM, Alvaro Herrera wrote: > Jeff Janes wrote: >> If I give pgbench an empty file, I get a segfault. >> >> $ touch empty.sql >> $ src/bin/pgbench/pgbench -T 60 -f empty.sql >> starting vacuum...end. >> Segmentation fault (co

Re: [HACKERS] Bug in StartupSUBTRANS

2016-02-13 Thread Jeff Janes
On Tue, Feb 9, 2016 at 10:33 AM, Simon Riggs wrote: > On 9 February 2016 at 18:42, Jeff Janes wrote: >> >> While testing the crash resilience of the recent 2-part-commit >> improvements, I've run into a problem where sometimes after a crash >> the recovery

Re: [HACKERS] Remove or weaken hints about "effective resolution of sleep delays is 10 ms"?

2016-02-16 Thread Jeff Janes
On Tue, Feb 16, 2016 at 12:06 AM, Robert Haas wrote: > On Wed, Feb 10, 2016 at 5:15 PM, Andres Freund wrote: >> Hi, >> >> Several places in our docs have blurbs like >>> Note that on many systems, the effective resolution of sleep delays is >>> 10 milliseconds; setting wal_writer_delay to a value

[HACKERS] Deferrable check constraints

2016-02-20 Thread Jeff Janes
I recently wished for deferrable check constraints while doing some crash-recovery stress testing. I don't know how important they would be for real-world cases, but the SQL standard does seem to require them, so I think they would be desirable just for that reason. There isn't an entry for this

Re: [HACKERS]WIP: Covering + unique indexes.

2016-02-25 Thread Jeff Janes
On Thu, Feb 11, 2016 at 8:46 AM, Anastasia Lubennikova wrote: > 02.02.2016 15:50, Anastasia Lubennikova: > > As promised, here's the new version of the patch "including_columns_4.0". > I fixed all issues except some points mentioned below. Thanks for the update patch. I get a compiler warning:

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2016-02-25 Thread Jeff Janes
On Wed, Feb 24, 2016 at 8:51 AM, Teodor Sigaev wrote: > Thank you for remembering this problem, at least for me. > >>> Well, turns out there's a quite significant difference, actually. The >>> index sizes I get (quite stable after multiple runs): >>> >>> 9.5 : 2428 MB >>> 9.6 + alone clean

Re: [HACKERS] Fuzzy substring searching with the pg_trgm extension

2016-02-25 Thread Jeff Janes
On Fri, Jan 29, 2016 at 6:15 AM, Teodor Sigaev wrote: >> The behavior of this function is surprising to me. >> >> select substring_similarity('dog' , 'hotdogpound') ; >> >> substring_similarity >> -- >> 0.25 >> > Substring search was desined to search simil

[HACKERS] distinct estimate of a hard-coded VALUES list

2016-08-18 Thread Jeff Janes
I have a query which contains a where clause like: aid =ANY(VALUES (1),(45),(87), <6948 more>, (447)) for example: perl -le 'print "explain (analyze) select sum(abalance) from pgbench_accounts where aid=ANY(VALUES "; print join ",", map "($_)", sort {$a<=>$b} map int(rand(500)), 1..6952

Re: [HACKERS] sslmode=require fallback

2016-08-19 Thread Jeff Janes
On Sat, Jul 30, 2016 at 11:18 AM, Bruce Momjian wrote: > On Fri, Jul 29, 2016 at 11:27:06AM -0400, Peter Eisentraut wrote: > > On 7/29/16 11:13 AM, Bruce Momjian wrote: > > > Yes, I am thinking of a case where Postgres is down but a malevolent > > > user starts a Postgres server on 5432 to gather

Re: [HACKERS] distinct estimate of a hard-coded VALUES list

2016-08-20 Thread Jeff Janes
On Thu, Aug 18, 2016 at 2:25 PM, Tom Lane wrote: > Jeff Janes writes: > > So even though it knows that 6952 values have been shoved in the bottom, > it > > thinks only 200 are going to come out of the aggregation. This seems > like > > a really lousy estimate. In mo

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2016-08-23 Thread Jeff Janes
Hi Amit, Thanks for working on this. When building with --enable-cassert, I get compiler warning: hash.c: In function 'hashbucketcleanup': hash.c:722: warning: 'new_bucket' may be used uninitialized in this function After an intentionally created crash, I get an Assert triggering: TRAP: Faile

[HACKERS] recent compiler warnings

2016-08-23 Thread Jeff Janes
Sorry for starting a new thread, I can't find the correct one to reply to. Using: gcc version 4.8.3 20140627 [gcc-4_8-branch revision 212064] (SUSE Linux) commit ed0097e4f9e6b1 has introduced two compiler warnings: gistutil.c: In function 'gistproperty': gistutil.c:855:16: warning: variable 'rd_

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2016-08-24 Thread Jeff Janes
On Tue, Aug 23, 2016 at 10:05 PM, Amit Kapila wrote: > On Wed, Aug 24, 2016 at 2:37 AM, Jeff Janes wrote: > > > > > After an intentionally created crash, I get an Assert triggering: > > > > TRAP: FailedAssertion("!(((freep)[(bitmapbit)/32] & > >

Re: [HACKERS] distinct estimate of a hard-coded VALUES list

2016-08-28 Thread Jeff Janes
On Mon, Aug 22, 2016 at 10:19 AM, Robert Haas wrote: > On Sat, Aug 20, 2016 at 4:58 PM, Tom Lane wrote: > > Jeff Janes writes: > >> On Thu, Aug 18, 2016 at 2:25 PM, Tom Lane wrote: > >>> It does know it, what it doesn't know is how many duplicates there are

Re: [HACKERS] Notice lock waits

2016-08-31 Thread Jeff Janes
On Tue, Aug 9, 2016 at 5:17 PM, Jim Nasby wrote: > On 8/5/16 12:00 PM, Jeff Janes wrote: > >> So I created a new guc, notice_lock_waits, which acts like >> log_lock_waits but sends the message as NOTICE so it will show up on >> interactive connections like psql. >&g

Re: [HACKERS] restoration after crash slowness, any way to improve?

2016-09-01 Thread Jeff Janes
On Wed, Aug 31, 2016 at 6:26 PM, Joshua D. Drake wrote: > -hackers, > > So this is more of a spit balling thread than anything. As I understand > it, if we have a long running transaction or a large number of wal logs and > we crash, we then have to restore those logs on restart to the last known

Re: [HACKERS] Hash Indexes

2016-09-07 Thread Jeff Janes
On Thu, Sep 1, 2016 at 8:55 PM, Amit Kapila wrote: > > I have fixed all other issues you have raised. Updated patch is > attached with this mail. > I am finding the comments (particularly README) quite hard to follow. There are many references to an "overflow bucket", or similar phrases. I thi

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2016-09-08 Thread Jeff Janes
On Wed, Sep 7, 2016 at 3:29 AM, Ashutosh Sharma wrote: > > Thanks to Ashutosh Sharma for doing the testing of the patch and > > helping me in analyzing some of the above issues. > > Hi All, > > I would like to summarize the test-cases that i have executed for > validating WAL logging in hash inde

Re: [HACKERS] High-CPU consumption on information_schema (only) query

2016-09-08 Thread Jeff Janes
On Wed, Sep 7, 2016 at 4:37 PM, Robins Tharakan wrote: > > Hi, > > An SQL (with only information_schema related JOINS) when triggered, runs with max CPU (and never ends - killed after 2 days). > - It runs similarly (very slow) on a replicated server that acts as a read-only slave. > - Top shows

Re: [HACKERS] feature request: explain "with details" option

2016-09-10 Thread Jeff Janes
On Thu, Sep 8, 2016 at 10:40 AM, Roger Pack wrote: > My apologies if this was already requested before... > > I think it would be fantastic if postgres had an "explain the explain" > option: > Today's explain tells us what loops and scans were used, and relative > costs, etc. It doesn't seem to

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2016-09-11 Thread Jeff Janes
On Thu, Sep 8, 2016 at 12:09 PM, Jeff Janes wrote: > I plan to do testing using my own testing harness after changing it to > insert a lot of dummy tuples (ones with negative values in the pseudo-pk > column, which are never queried by the core part of the harness) and > deleting th

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2016-09-11 Thread Jeff Janes
On Sun, Sep 11, 2016 at 7:40 PM, Amit Kapila wrote: > On Mon, Sep 12, 2016 at 7:00 AM, Jeff Janes wrote: > > On Thu, Sep 8, 2016 at 12:09 PM, Jeff Janes > wrote: > > > >> > >> I plan to do testing using my own testing harness after changing it to > >

Re: [HACKERS] Hash Indexes

2016-09-13 Thread Jeff Janes
On Wed, Sep 7, 2016 at 9:32 PM, Amit Kapila wrote: > On Wed, Sep 7, 2016 at 11:49 PM, Jeff Janes wrote: > > On Thu, Sep 1, 2016 at 8:55 PM, Amit Kapila > wrote: > >> > >> > >> I have fixed all other issues you have raised. Updated patch is > &g

Re: [HACKERS] pageinspect: Hash index support

2016-09-14 Thread Jeff Janes
On Tue, Aug 30, 2016 at 10:06 AM, Alvaro Herrera wrote: > Jesper Pedersen wrote: > > Hi, > > > > Attached is a patch that adds support for hash indexes in pageinspect. > > > > The functionality (hash_metap, hash_page_stats and hash_page_items) > follows > > the B-tree functions. > > I suggest tha

Re: [HACKERS] Hash Indexes

2016-09-14 Thread Jeff Janes
On Tue, Sep 13, 2016 at 9:31 AM, Jeff Janes wrote: > === > > +Vacuum acquires cleanup lock on bucket to remove the dead tuples and or > tuples > +that are moved due to split. The need for cleanup lock to remove dead > tuples > +is to ensure that scans' returns c

Re: [HACKERS] Hash Indexes

2016-09-14 Thread Jeff Janes
On Tue, May 10, 2016 at 5:09 AM, Amit Kapila wrote: > > > Although, I don't think it is a very good idea to take any performance > data with WIP patch, still I couldn't resist myself from doing so and below > are the performance numbers. To get the performance data, I have dropped > the primary

Re: [HACKERS] Hash Indexes

2016-09-16 Thread Jeff Janes
On Thu, Sep 15, 2016 at 7:23 AM, Andres Freund wrote: > Hi, > > On 2016-05-10 17:39:22 +0530, Amit Kapila wrote: > > For making hash indexes usable in production systems, we need to improve > > its concurrency and make them crash-safe by WAL logging them. > > One earlier question about this is wh

Re: [HACKERS] README of hash index

2016-09-16 Thread Jeff Janes
On Fri, Sep 16, 2016 at 4:20 AM, Amit Kapila wrote: > Currently README of hash module contain algorithms written in below form. > > The insertion algorithm is rather similar: > > pin meta page and take buffer content lock in shared mode > loop: > compute bucket number for target hash key > releas

Re: [HACKERS] Hash Indexes

2016-09-19 Thread Jeff Janes
On Sun, Sep 18, 2016 at 11:44 PM, Amit Kapila wrote: > On Mon, Sep 19, 2016 at 11:20 AM, Mark Kirkwood > wrote: > > On 17/09/16 06:38, Andres Freund wrote: > > > > While I see the point of what you are saying here, I recall all previous > > discussions about has indexes tended to go a bit like t

Re: [HACKERS] pageinspect: Hash index support

2016-09-20 Thread Jeff Janes
On Tue, Sep 20, 2016 at 5:40 AM, Jesper Pedersen wrote: > On 09/20/2016 03:19 AM, Michael Paquier wrote: > >> You did not get right the comments from Alvaro upthread. The following >> functions are added with this patch: >> function hash_metap(text) >> function hash_metap_bytea(bytea) >> funct

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2016-09-20 Thread Jeff Janes
On Thu, Sep 15, 2016 at 11:42 PM, Amit Kapila wrote: > > Okay, Thanks for pointing out the same. I have fixed it. Apart from > that, I have changed _hash_alloc_buckets() to initialize the page > instead of making it completely Zero because of problems discussed in > another related thread [1].

Re: [HACKERS] pageinspect: Hash index support

2016-09-21 Thread Jeff Janes
On Tue, Sep 20, 2016 at 11:14 PM, Michael Paquier wrote: > > + > + The type information will be 'm' for a metadata > page, > + 'v' for an overflow page, > 'b' for a bucket page, > + 'i' for a bitmap page, and > 'u' for an unused page. > + > > Other functions don't go in

Re: [HACKERS] Hash Indexes

2016-09-21 Thread Jeff Janes
On Thu, Sep 15, 2016 at 7:13 AM, Robert Haas wrote: > On Thu, Sep 15, 2016 at 1:41 AM, Amit Kapila > wrote: > > I think it is possible without breaking pg_upgrade, if we match all > > items of a page at once (and save them as local copy), rather than > > matching item-by-item as we do now. We a

Re: [HACKERS] Hash Indexes

2016-09-21 Thread Jeff Janes
On Wed, Sep 21, 2016 at 12:44 PM, Geoff Winkless wrote: > On 21 September 2016 at 13:29, Robert Haas wrote: > > I'd be curious what benefits people expect to get. > > An edge case I came across the other day was a unique index on a large > string: postgresql popped up and told me that I couldn't

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2016-09-21 Thread Jeff Janes
On Tue, Sep 20, 2016 at 10:27 PM, Amit Kapila wrote: > On Tue, Sep 20, 2016 at 10:24 PM, Jeff Janes wrote: > > On Thu, Sep 15, 2016 at 11:42 PM, Amit Kapila > > wrote: > >> > >> > >> Okay, Thanks for pointing out the same. I have fixed

Re: [HACKERS] pageinspect: Hash index support

2016-09-26 Thread Jeff Janes
On Mon, Sep 26, 2016 at 10:39 AM, Jesper Pedersen < [email protected]> wrote: > Hi, > > On 09/23/2016 12:10 AM, Peter Eisentraut wrote: > >> >> > - As of very recently, we don't need to move pageinspect--1.5.sql to >> pageinspect--1.6.sql anymore. Just add pageinspect--1.5--1.6.sql. >>

Re: [HACKERS] pageinspect: Hash index support

2016-09-26 Thread Jeff Janes
On Tue, Sep 20, 2016 at 12:19 AM, Michael Paquier wrote: > > Note: the patch checks if a superuser is calling the new functions, > which is a good thing. > If we only have the bytea functions and the user needs to supply the raw pages themselves, rather than having the function go get the raw pa

Re: [HACKERS] Cache Hash Index meta page.

2016-09-26 Thread Jeff Janes
On Tue, Sep 13, 2016 at 12:55 PM, Mithun Cy wrote: > On Thu, Sep 8, 2016 at 11:21 PM, Jesper Pedersen < > [email protected]> wrote: >> >> > For the archives, this patch conflicts with the WAL patch [1]. >> >> > [1] https://www.postgresql.org/message-id/CAA4eK1JS%2BSiRSQBzEFp >> nsSmxZKin

Re: [HACKERS] Possible SSL improvements for a newcomer to tackle

2017-10-04 Thread Jeff Janes
On Mon, Oct 2, 2017 at 9:33 PM, Tom Lane wrote: > > It's possible that we could adopt some policy like "if the root.crt file > exists then default to verify" ... but that seems messy and unreliable, > so I'm not sure it would really add any security. > That is what we do. If root.crt exists, we

Re: [HACKERS] Possible SSL improvements for a newcomer to tackle

2017-10-04 Thread Jeff Janes
On Tue, Oct 3, 2017 at 6:44 AM, Tom Lane wrote: > Magnus Hagander writes: > > On Tue, Oct 3, 2017 at 6:33 AM, Tom Lane wrote: > >> I'm not an SSL expert, so insert appropriate grain of salt, but AIUI the > >> question is what are you going to verify against? > > > One way to do it would be to d

Re: [HACKERS] postgres_fdw super user checks

2017-10-04 Thread Jeff Janes
On Thu, Sep 14, 2017 at 1:08 PM, Robert Haas wrote: > On Thu, Sep 14, 2017 at 2:33 PM, Jeff Janes wrote: > > I think that foreign tables ought to behave as views do, where they run > as > > the owner rather than the invoker. No one has talked me out of it, but > no > &

Re: [HACKERS] postgres_fdw super user checks

2017-10-05 Thread Jeff Janes
On Thu, Oct 5, 2017 at 6:44 AM, Robert Haas wrote: > On Wed, Oct 4, 2017 at 6:13 PM, Jeff Janes wrote: > > OK. And if you want the first one, you can wrap it in a view currently, > but > > if it were changed I don't know what you would do if you want the 2nd one >

[HACKERS] bgwriter_lru_maxpages range in postgresql.conf

2017-10-27 Thread Jeff Janes
With v10, the upper limit on bgwriter_lru_maxpages was changed from 1000 to INT_MAX / 2, but the postgresql.conf.sample was not updated. #bgwriter_lru_maxpages = 100# 0-1000 max buffers written/round I don't see any precedence for including INT_MAX-type limits in the sample config file, s

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

2017-11-03 Thread Jeff Janes
Hi Alvaro, With this v3 patch (assuming this is the one you just committed as ec42a1dcb30de235b252f6d4) am now getting make check failures. brin_summarize_range is returning unexpected values. CentOS6, PostgreSQL 11devel on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

2017-11-03 Thread Jeff Janes
On Fri, Nov 3, 2017 at 1:34 PM, Tom Lane wrote: > Jeff Janes writes: > > With this v3 patch (assuming this is the one you just committed > > as ec42a1dcb30de235b252f6d4) am now getting make check failures. > > There's a followup commit already :-( > >

Re: [HACKERS] pg_basebackup --progress output for batch execution

2017-11-09 Thread Jeff Janes
On Fri, Sep 29, 2017 at 4:00 PM, Martin Marques < [email protected]> wrote: > Hi, > > Some time ago I had to work on a system where I was cloning a standby > using pg_basebackup, that didn't have screen or tmux. For that reason I > redirected the output to a file and ran it with nohup

Re: [HACKERS] Better tracking of free space during SP-GiST index build

2016-09-28 Thread Jeff Janes
On Wed, Sep 28, 2016 at 10:48 AM, Robert Haas wrote: > On Sun, Sep 25, 2016 at 3:28 PM, Tomas Vondra > wrote: > > Sure, that would be useful. > > > > I think it would be useful to make repository of such data sets, so that > > patch authors & reviewers can get a reasonable collection of data set

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

2016-09-29 Thread Jeff Janes
On Thu, Sep 29, 2016 at 8:33 AM, Peter Eisentraut < [email protected]> wrote: > On 9/26/16 10:34 PM, Michael Paquier wrote: > > I thought that as long as the error string is shown to the user, it > > does not matter much if errno is still saved or not. All the callers > > of durable

Re: [HACKERS] Notice lock waits

2016-09-29 Thread Jeff Janes
On Wed, Sep 28, 2016 at 11:57 PM, Haribabu Kommi wrote: > > > On Sat, Aug 6, 2016 at 3:00 AM, Jeff Janes wrote: > >> One time too many, I ran some minor change using psql on a production >> server and was wondering why it was taking so much longer than it did >>

Re: [HACKERS] Hash Indexes

2016-10-03 Thread Jeff Janes
On Thu, Sep 29, 2016 at 5:14 PM, Robert Haas wrote: > On Thu, Sep 29, 2016 at 8:07 PM, Peter Geoghegan wrote: > > On Wed, Sep 28, 2016 at 8:06 PM, Andres Freund > wrote: > >> On 2016-09-28 15:04:30 -0400, Robert Haas wrote: > >>> Andres already > >>> stated that he things working on btree-over-

Re: [HACKERS] Cache Hash Index meta page.

2016-10-04 Thread Jeff Janes
On Fri, Jul 22, 2016 at 3:02 AM, Mithun Cy wrote: > I have created a patch to cache the meta page of Hash index in > backend-private memory. This is to save reading the meta page buffer every > time when we want to find the bucket page. In “_hash_first” call, we try > to read meta page buffer twi

Re: [HACKERS] Autovacuum launcher process launches worker process at high frequency

2016-10-05 Thread Jeff Janes
On Wed, Oct 5, 2016 at 7:28 AM, Masahiko Sawada wrote: > Hi all, > > I found the kind of strange behaviour of the autovacuum launcher > process when XID anti-wraparound vacuum. > > Suppose that a database (say test_db) whose age of frozenxid is about > to reach max_autovacuum_max_age has three ta

Re: [HACKERS] Fast AT ADD COLUMN with DEFAULTs

2016-10-05 Thread Jeff Janes
On Wed, Oct 5, 2016 at 3:29 PM, Andres Freund wrote: > On 2016-10-05 15:23:05 -0700, Vitaly Burovoy wrote: > > On 10/5/16, Andres Freund wrote: > > > On 2016-10-05 11:58:33 -0700, Serge Rielau wrote: > > >> Dear Hackers, > > >> I’m working on a patch that expands PG’s ability to add columns to a

Re: [HACKERS] VACUUM's ancillary tasks

2016-10-06 Thread Jeff Janes
On Sat, Oct 1, 2016 at 1:34 PM, Vik Fearing wrote: > > Sure, I could handle each case separately, but the goal of this patch > (as hinted at in the Subject) is to generalize all the different tasks > we've been giving to VACUUM. The only missing piece is what the first > patch addresses; which i

Re: [HACKERS] pgbench vs. wait events

2016-10-06 Thread Jeff Janes
On Thu, Oct 6, 2016 at 11:38 AM, Robert Haas wrote: > Hi, > > I decided to do some testing on hydra (IBM-provided community > resource, POWER, 16 cores/64 threads, kernel 3.2.6-3.fc16.ppc64) using > the newly-enhanced wait event stuff to try to get an idea of what > we're waiting for during pgben

Re: [HACKERS] VACUUM's ancillary tasks

2016-10-06 Thread Jeff Janes
going to > > benefit from index-only-scans, and I don't want to pay the cost of them > > getting vacuumed just because of some iOn Thu, Oct 6, 2016 at 3:56 PM, > Jeff Janes wrote: > >> Sure, I could handle each case separately, but the goal of this patch > > nserts,

Re: [HACKERS] pgbench vs. wait events

2016-10-07 Thread Jeff Janes
On Thu, Oct 6, 2016 at 11:38 AM, Robert Haas wrote: > > Next, I tried lowering the scale factor to something that fits in > shared buffers. Here are the results at scale factor 300: > > 14 Lock| tuple > 22 LWLockTranche | lock_manager > 39 LWLockNamed | WALBuf

Re: [HACKERS] Question about pg_control usage

2016-10-07 Thread Jeff Janes
On Fri, Oct 7, 2016 at 8:24 AM, Anastasia Lubennikova < [email protected]> wrote: > Hi, hackers! > > I am examining various global variables in ShmemVariableCache, > pg_control and pg_controldata. To force and debug xid wraparound, > I've implemented a function, that allows to set next

Re: [HACKERS] pgbench vs. wait events

2016-10-08 Thread Jeff Janes
On Fri, Oct 7, 2016 at 1:28 PM, Robert Haas wrote: > On Fri, Oct 7, 2016 at 11:51 AM, Jeff Janes wrote: > > What happens if you turn fsync off? Once a xlog file is fully written, > it > > is immediately fsynced, even if the backend is holding WALWriteLock or > > wal

  1   2   3   4   5   6   7   8   9   10   >