Re: [HACKERS] Determine state of cluster (HA)

2017-10-17 Thread Joshua D. Drake
On 10/16/2017 07:31 PM, Craig Ringer wrote: On 17 October 2017 at 01:02, Joshua D. Drake <j...@commandprompt.com> wrote: On 10/15/2017 07:39 PM, Craig Ringer wrote: - Get info about master. We should finish merging recovery.conf into postgresql.conf. Definitely. There's a patc

Re: [HACKERS] Determine state of cluster (HA)

2017-10-16 Thread Joshua D. Drake
On 10/16/2017 03:55 AM, Magnus Hagander wrote: On Mon, Oct 16, 2017 at 4:39 AM, Craig Ringer <cr...@2ndquadrant.com <mailto:cr...@2ndquadrant.com>> wrote: On 13 October 2017 at 08:50, Joshua D. Drake <j...@commandprompt.com <mailto:j...@commandprompt.co

Re: [HACKERS] Determine state of cluster (HA)

2017-10-16 Thread Joshua D. Drake
On 10/15/2017 07:39 PM, Craig Ringer wrote: On 13 October 2017 at 08:50, Joshua D. Drake <j...@commandprompt.com> wrote: -Hackers, I had a long call with a firm developing front end proxy/cache/HA for Postgres today. Essentially the software is a replacement for PGPool in entirety bu

Re: [HACKERS] Determine state of cluster (HA)

2017-10-13 Thread Joshua D. Drake
On 10/12/2017 05:50 PM, Joshua D. Drake wrote: -Hackers, Bumping this. I had a long call with a firm developing front end proxy/cache/HA for Postgres today. Essentially the software is a replacement for PGPool in entirety but also supports analytics etc... When I was asking them about

[HACKERS] Determine state of cluster (HA)

2017-10-12 Thread Joshua D. Drake
-Hackers, I had a long call with a firm developing front end proxy/cache/HA for Postgres today. Essentially the software is a replacement for PGPool in entirety but also supports analytics etc... When I was asking them about pain points they talked about the below and I was wondering if this

Re: [HACKERS] Columnar storage support

2017-10-09 Thread Joshua D. Drake
On 10/09/2017 01:03 PM, legrand legrand wrote: Is there a chance that pluggable storage permits creation of a columnar rdbms as monetDB in PostgreSQL ? Thanks un advance for thé answer The extension C-Store from Citus is probably what you are looking for. jD -- Sent from:

Re: [HACKERS] search path security issue?

2017-10-05 Thread Joshua D. Drake
On 10/05/2017 02:54 PM, David G. Johnston wrote: On Thu, Oct 5, 2017 at 2:37 PM, Joshua D. Drake <j...@commandprompt.com <mailto:j...@commandprompt.com>>wrote: I get being able to change my search_path on the fly but it seems odd that as user foo I can change my default

[HACKERS] search path security issue?

2017-10-05 Thread Joshua D. Drake
-hackers, Please see the below: """ postgres=# create user foo; CREATE ROLE postgres=# create schema foo; CREATE SCHEMA postgres=# alter role foo set search_path to 'foo'; ALTER ROLE postgres=# \q jd@jd-wks:~$ psql -U foo postgres psql (9.6.5) Type "help" for help. postgres=> show search_path;

Re: [HACKERS] 64-bit queryId?

2017-10-02 Thread Joshua D. Drake
On 10/01/2017 04:22 PM, Robert Haas wrote: On Sun, Oct 1, 2017 at 3:48 PM, Greg Stark wrote: Well these kinds of monitoring systems tend to be used by operations people who are a lot more practical and a lot less worried about theoretical concerns like that. +1, well said.

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Joshua D. Drake
On 09/25/2017 11:31 AM, Alvaro Hernandez wrote: Whether or not they are included in a managed environment is generally based on two things: 1. Safety (why RDS doesn't allow certain C extensions) 2. Community/Popularity (Exactly why RDS has PostGIS)     A. Demand with a

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Joshua D. Drake
On 09/25/2017 10:43 AM, Andres Freund wrote: On 2017-09-25 10:38:52 -0700, Joshua D. Drake wrote: On 09/25/2017 10:32 AM, Petr Jelinek wrote: On 25/09/17 19:26, Tom Lane wrote: Alvaro Hernandez <a...@ongres.com> writes: There is already about 3 million output plugins out there so I

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Joshua D. Drake
On 09/25/2017 10:32 AM, Petr Jelinek wrote: On 25/09/17 19:26, Tom Lane wrote: Alvaro Hernandez writes: There is already about 3 million output plugins out there so I think we did reasonable job there. The fact that vast majority of that are various json ones gives

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Joshua D. Drake
On 09/25/2017 10:19 AM, Petr Jelinek wrote: On 25/09/17 18:48, Alvaro Hernandez wrote:     In my opinion, logical decoding plugins that don't come with core are close to worthless (don't get me wrong): I respectfully disagree. As do I. - They very unlikely will be installed in

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Joshua D. Drake
On 09/25/2017 10:15 AM, Gregory Brail wrote: Yes. I'm advocating something "built-in" to Postgres. Any or all of those are likely a great starting point. I don't see a benefit to having this "in postgres". The whole reason we have built out a mature and extensible product is so that not

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Joshua D. Drake
On 09/25/2017 09:59 AM, Gregory Brail wrote: However, I can't find any docs for the output format of pgoutput, which is going to make it less likely for people to be able to consume it. Is anyone working on docs? I know that it's a painful process. I also think that a JSON-format (or

Re: [HACKERS] SCRAM in the PG 10 release notes

2017-09-21 Thread Joshua D. Drake
On 09/21/2017 07:51 AM, Peter Eisentraut wrote: On 9/20/17 15:52, Jeff Janes wrote: I think that the addition of a link to > https://wiki.postgresql.org/wiki/List_of_drivers would be appropriate. I don't have any

Re: [HACKERS] PG 10 release notes

2017-09-19 Thread Joshua D. Drake
On 09/19/2017 09:32 AM, 'Bruce Momjian' wrote: On Tue, Sep 19, 2017 at 12:30:01PM -0400, Tom Lane wrote: "'Bruce Momjian'" writes: On Tue, Sep 19, 2017 at 12:22:39PM -0400, Tom Lane wrote: We don't normally release-note documentation changes. If this wasn't purely a

Re: [HACKERS] Is it time to kill support for very old servers?

2017-09-18 Thread Joshua D. Drake
On 09/18/2017 04:54 AM, Robert Haas wrote: On Mon, Sep 18, 2017 at 7:17 AM, Andres Freund wrote: Private: Not so much. Well, as much as the Internet is actually private: https://ilccyberreport.files.wordpress.com/2013/06/nsa11.jpg JD ;) -- Command Prompt, Inc. ||

Re: [HACKERS] Proposal: global index

2017-08-24 Thread Joshua D. Drake
On 08/24/2017 10:52 AM, Adam Brusselback wrote: My understanding is that global indexes allow foreign keys to work naturally with partitioned tables, or tables in an inheritance hierarchy. That is pretty big IMO, as it allows you to partition a table without making a trade-off in your

Re: [HACKERS] Updating line length guidelines

2017-08-20 Thread Joshua D. Drake
On 08/20/2017 09:32 AM, Andres Freund wrote: On 2017-08-20 09:29:39 -0700, Joshua D. Drake wrote: On 08/20/2017 07:49 AM, Andres Freund wrote: Hi, We currently still have the guideline that code should fit into an 80 character window. But an increasing amount of the code, and code submissions

Re: [HACKERS] Updating line length guidelines

2017-08-20 Thread Joshua D. Drake
On 08/20/2017 07:49 AM, Andres Freund wrote: Hi, We currently still have the guideline that code should fit into an 80 character window. But an increasing amount of the code, and code submissions, don't adhere to that (e.g. copy.c, which triggered me to write this email). And I mean outside of

Re: [HACKERS] On Complex Source Code Reading Strategy

2017-07-27 Thread Joshua D. Drake
On 07/27/2017 04:45 PM, Tom Lane wrote: Peter Geoghegan writes: 2. Start somewhere. I have no idea where that should be, but it has to be some particular place that seems interesting to you. Don't forget to start with the available documentation, ie

Re: [HACKERS] Create language syntax is not proper in pg_dumpall and not working using pg_upgrade

2017-07-25 Thread Joshua D. Drake
On 07/25/2017 10:24 AM, Andres Freund wrote: On 2017-07-25 13:18:25 -0400, Robert Haas wrote: On Tue, Jul 25, 2017 at 1:13 PM, Andres Freund wrote: On 2017-07-25 13:10:11 -0400, Robert Haas wrote: On Tue, Jul 25, 2017 at 1:06 PM, Tom Lane wrote: Is

Re: [HACKERS] autovacuum can't keep up, bloat just continues to rise

2017-07-24 Thread Joshua D. Drake
On 07/23/2017 12:03 PM, Joshua D. Drake wrote: As you can see even with aggressive vacuuming, over a period of 36 hours life gets increasingly miserable. The largest table is: postgres=# select pg_size_pretty(pg_total_relation_size('bmsql_order_line')); pg_size_pretty

Re: [HACKERS] autovacuum can't keep up, bloat just continues to rise

2017-07-23 Thread Joshua D. Drake
Hello, I changed the test to run for 6 hours at a time regardless of number of transactions. I also changed the du command to only look at the database (previously wal logs were included). This is the clearest indication of the problem I have been able to produce. Again, this is with 128

[HACKERS] possible effective_io_concurrency performance regression

2017-07-21 Thread Joshua D. Drake
-hackers, While updating my Postgres performance curriculum I was doing some testing with effective_io_concurrency and I may have found a regression. I am aware that the parameter only works under certain conditions. However, what I appear to have found is that if it is set to anything but

Re: [HACKERS] autovacuum can't keep up, bloat just continues to rise

2017-07-21 Thread Joshua D. Drake
On 07/20/2017 08:58 PM, Joshua D. Drake wrote: On 07/19/2017 07:57 PM, Tom Lane wrote: Peter Geoghegan <p...@bowt.ie> writes: Test 1: 55G/srv/main TPS:955 Test 2: 112G/srv/main TPS:531 (Not sure what happened here, long checkpoint?) Test 3: 109G/srv/main TPS:

Re: [HACKERS] autovacuum can't keep up, bloat just continues to rise

2017-07-21 Thread Joshua D. Drake
On 07/20/2017 11:54 PM, Sokolov Yura wrote: On 2017-07-21 06:58, Joshua D. Drake wrote: On 07/19/2017 07:57 PM, Tom Lane wrote: Peter Geoghegan <p...@bowt.ie> writes: -- Command Prompt, Inc. || http://the.postgres.company/ || @cmdpromptinc PostgreSQL Centered full stack s

Re: [HACKERS] autovacuum can't keep up, bloat just continues to rise

2017-07-20 Thread Joshua D. Drake
On 07/19/2017 07:57 PM, Tom Lane wrote: Peter Geoghegan writes: My argument for the importance of index bloat to the more general bloat problem is simple: any bloat that accumulates, that cannot be cleaned up, will probably accumulate until it impacts performance quite

Re: [HACKERS] autovacuum can't keep up, bloat just continues to rise

2017-07-20 Thread Joshua D. Drake
On 07/20/2017 06:28 AM, Stephen Frost wrote: It's not clear off-hand how much that would improve this case, as the database size appears to pretty quickly get beyond the OS memory size (and only in the first test is the DB starting size less than system memory to begin with). FYI, I will be

[HACKERS] autovacuum can't keep up, bloat just continues to rise

2017-07-19 Thread Joshua D. Drake
Hello, At PGConf US Philly last week I was talking with Jim and Jan about performance. One of the items that came up is that PostgreSQL can't run full throttle for long periods of time. The long and short is that no matter what, autovacuum can't keep up. This is what I have done: Machine:

Re: [HACKERS] Revisiting NAMEDATALEN

2017-07-03 Thread Joshua D. Drake
On 07/03/2017 11:31 AM, Emrul wrote: Hi hackers, This question came up again on Reddit: https://www.reddit.com/r/PostgreSQL/comments/6kyyev/i_have_hit_the_table_name_length_limit_a_number/ and I thought I'd echo it here. I totally am on board with short, descriptive names and a good

Re: [HACKERS] Optional message to user when terminating/cancelling backend

2017-06-26 Thread Joshua D. Drake
On 06/26/2017 07:15 AM, Joel Jacobson wrote: +1 On Tue, Jun 20, 2017 at 8:54 PM, Alvaro Herrera wrote: Unless you have a lot of users running psql manually, I don't see how this is actually very useful or actionable. What would the user do with the information?

Re: [HACKERS] TAP backpatching policy

2017-05-31 Thread Joshua D. Drake
On 05/31/2017 07:49 AM, Robert Haas wrote: On Tue, May 30, 2017 at 9:12 PM, Craig Ringer wrote: Thoughts? Backpatch new TAP methods, etc, into back branches routinely? [...] When customers start to doubt that, then they become reluctant to apply new minor versions

Re: [HACKERS] TAP backpatching policy

2017-05-31 Thread Joshua D. Drake
On 05/30/2017 09:52 PM, Stephen Frost wrote: Tom, Um ... but we still have 2 live pre-9.4 branches. If your proposal doesn't extend to back-porting all of this stuff as far as 9.2, I don't see what this is really buying. We'd still need version cutoff checks in the tests. I don't believe

Re: [HACKERS] Potential issue with alter system

2017-05-04 Thread Joshua D. Drake
On 05/04/2017 12:49 PM, Tom Lane wrote: "Joshua D. Drake" <j...@commandprompt.com> writes: So I did this: If you have other entries you want to keep in the postgresql.auto.conf file, you could get away with manually editing it to remove the newline. Got it. Th

[HACKERS] Potential issue with alter system

2017-05-04 Thread Joshua D. Drake
Folks, So I did this: postgres=# alter system set archive_command to 'rsynv -av %p postgres@52.3.141.224:/data/archive/%f '; Note the new line. It properly created in postgresql.auto.conf: archive_command = 'rsynv -av %p postgres@52.3.141.224:/data/archive/%f ' (note the new line) I

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Joshua D. Drake
On 05/02/2017 05:13 AM, Tom Lane wrote: Robert Haas writes: On Thu, Apr 20, 2017 at 7:46 AM, Petr Jelinek wrote: DROP SUBSCRIPTION mysub NODROP SLOT; Having said that, I doubt that anyone would argue that CREATE USER is anything but

Re: [HACKERS] PG 10 release notes

2017-05-01 Thread Joshua D. Drake
On 05/01/2017 05:40 AM, Robert Haas wrote: On Tue, Apr 25, 2017 at 9:56 PM, Bruce Momjian wrote: First, I don't think RFC references belong in the release notes, let alone RFC links. Why not? I think RFC references are a great thing to include in the release notes, and

Re: [HACKERS] RFC: ALTER SYSTEM [...] COMMENT

2017-04-26 Thread Joshua D. Drake
On 04/26/2017 10:31 AM, Tom Lane wrote: "Joshua D. Drake" <j...@commandprompt.com> writes: I wouldn't fight hard to change it but really if we think about it, what makes more sense from usability perspective? CREATE TABLE foo() COMMENT IS I think it's likely to be imposs

Re: [HACKERS] RFC: ALTER SYSTEM [...] COMMENT

2017-04-26 Thread Joshua D. Drake
On 04/26/2017 10:14 AM, Stephen Frost wrote: JD, Having COMMENT ON accept a general query whose result is then cast to text and stored as the comment would allow this to be done, eg: COMMENT ON table IS (pg_get_comment('table') || ' new text'); Dig it, although we probably want the

[HACKERS] RFC: ALTER SYSTEM [...] COMMENT

2017-04-26 Thread Joshua D. Drake
-hackers, We have had ALTER SYSTEM for some time now. It is awesome to be able to make changes that can be system wide without ever having to hit a shell but it does lack a feature that seems like an oversight, the ability to comment. Problem we are trying to solve: Having documentation

Re: [HACKERS] parallel "return query" is no good

2017-03-23 Thread Joshua D. Drake
On 03/23/2017 10:03 AM, Robert Haas wrote: On Thu, Mar 23, 2017 at 12:50 PM, Robert Haas wrote: Commit 7aea8e4f2daa4b39ca9d1309a0c4aadb0f7ed81b allowed a parallel plan to be generated when for a RETURN QUERY or RETURN QUERY EXECUTE statement in a PL/pgsql block. As it

Re: [HACKERS] Removing binaries (was: createlang/droplang deprecated)

2017-03-19 Thread Joshua D. Drake
On 03/19/2017 01:01 PM, Tom Lane wrote: Andreas Karlsson writes: As for if we should have backwards compatibility for the old names I am leaning weakly for providing it in the case of createuser. I can see end users being pissed off that the createuser command is suddenly

Re: [HACKERS] Removing binaries (was: createlang/droplang deprecated)

2017-03-19 Thread Joshua D. Drake
On 03/18/2017 01:12 PM, Magnus Hagander wrote: Magnus Hagander > writes: 2017-03-18 14:00 GMT+01:00 Peter Eisentraut >: createuser, dropuser - definitely pollutes the

[HACKERS] temp_buffers vs temp vs local and explain

2017-03-16 Thread Joshua D. Drake
-hackers, I was reviewing an explain plan today and with some help from Andrew G, I got a lot more information than I deserved. It did however bring up quite a usability issue that I think we should consider. Let's review the following two lines: Sort Method: external merge Disk: 19352kB

Re: [HACKERS] Odd listen_addresses behavior

2017-03-15 Thread Joshua D. Drake
On 03/15/2017 12:57 PM, Tom Lane wrote: I'm suspicious that you have an override of listen_addresses somewhere --- for instance, the "-i" postmaster command line switch effectively is --listen-addresses='*'. Yep postgresql.auto.conf. Thanks, sorry for the noise. JD -- Command Prompt,

[HACKERS] Odd listen_addresses behavior

2017-03-15 Thread Joshua D. Drake
-hackers, I found this today: jd@jd-wks:~/snap/postgresql96/common/data$ /snap/postgresql96/19/usr/bin/pg_ctl -D data stop pg_ctl: directory "data" does not exist jd@jd-wks:~/snap/postgresql96/common/data$ cd .. jd@jd-wks:~/snap/postgresql96/common$ /snap/postgresql96/19/usr/bin/pg_ctl -D

[HACKERS] Should we eliminate or reduce HUP from docs?

2017-03-10 Thread Joshua D. Drake
Hello, I am a bad speaker, I am writing a talk three weeks before the conference (as opposed to on the plane). I noticed in the docs we still reference the passing of SIGHUP for reloading conf file but we now have pg_reload_conf(); It seems the use of pg_reload_conf() would provide a better

Re: [HACKERS] Replication vs. float timestamps is a disaster

2017-02-27 Thread Joshua D. Drake
On 02/22/2017 02:45 PM, Tom Lane wrote: Andres Freund writes: On 2017-02-22 08:43:28 -0500, Tom Lane wrote: (To be concrete, I'm suggesting dropping --disable-integer-datetimes in HEAD, and just agreeing that in the back branches, use of replication protocol with

Re: [HACKERS] I propose killing PL/Tcl's "modules" infrastructure

2017-02-25 Thread Joshua D. Drake
On 02/25/2017 10:14 AM, Tom Lane wrote: Now, we could try to fix this bug, and add the regression test coverage that the code clearly lacks, and upgrade the documentation about it from its currently very sad state. But I think the right answer is just to remove the feature altogether. It's

Re: [HACKERS] Documentation improvements for partitioning

2017-02-23 Thread Joshua D. Drake
On 02/23/2017 09:27 AM, Peter Geoghegan wrote: On Thu, Feb 23, 2017 at 8:08 AM, Simon Riggs wrote: * "Good work so far, but there is still a very significant amount of work to do." There is always more work to do, so why say so? I think that the implication is that

Re: [HACKERS] Documentation improvements for partitioning

2017-02-15 Thread Joshua D. Drake
On 02/15/2017 06:10 AM, Simon Riggs wrote: On 13 February 2017 at 05:21, Amit Langote wrote: If I issue DROP TABLE elsewhere, it doesn't refuse to drop because it has indexes, sequences etc on it. So why should it just because it has partitions? Because

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-26 Thread Joshua D. Drake
-Hackers, From the field. I do not care what you chose, I care that: 1. It is consistent 2. It is readable/understandable 3. It is documented 4. It is done wholesale (because of usability) That's it. So whatever meets that criteria, let's go for it. That may mean that certain commands look a

Re: [HACKERS] Checksums by default?

2017-01-26 Thread Joshua D. Drake
On 01/25/2017 05:25 PM, Peter Geoghegan wrote: On Wed, Jan 25, 2017 at 1:22 PM, Peter Geoghegan wrote: I understand that my experience with storage devices is unusually narrow compared to everyone else here. That's why I remain neutral on the high level question of whether or

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Joshua D. Drake
On 01/25/2017 11:41 AM, Tom Lane wrote: Stephen Frost writes: Would you say that most user's databases run fast enough with checksums enabled? Or more than most, maybe 70%? 80%? In today's environment, I'd probably say that it's more like 90+%. It would be nice if

Re: [HACKERS] Checksums by default?

2017-01-24 Thread Joshua D. Drake
On 01/21/2017 09:09 AM, Tom Lane wrote: Stephen Frost writes: As for checksums, I do see value in them and I'm pretty sure that the author of that particular feature did as well, or we wouldn't even have it as an option. You seem to be of the opinion that we might as well

Re: [HACKERS] Packages: Again

2017-01-20 Thread Joshua D. Drake
On 01/17/2017 09:26 AM, Robert Haas wrote: On Fri, Jan 13, 2017 at 7:24 PM, Peter Geoghegan wrote: MERGE isn't UPSERT, and isn't even in competition with UPSERT as a feature. I've written reams of text explaining why this is so in precise detail, ... Hello, This is the

Re: [HACKERS] Packages: Again

2017-01-12 Thread Joshua D. Drake
On 01/12/2017 03:35 PM, Craig Ringer wrote: So far that's all "that'd be nice, but isn't a technical barrier" stuff. Package variables for example. When _do_ you _need_ them? For what? (I'm aware of some uses but "when you need them" helps us not at all). Well my answer would be,

Re: [HACKERS] Packages: Again

2017-01-12 Thread Joshua D. Drake
On 01/11/2017 04:12 PM, Craig Ringer wrote: What aspects / features of packages were the key issues? Unfortunately we didn't get too far into it because the webinar was about Postgres specifically. That said, I have been doing some followup. Here is some of it: because packages[1] o

Re: [HACKERS] Retiring from the Core Team

2017-01-12 Thread Joshua D. Drake
On 01/12/2017 07:48 AM, Jonathan Katz wrote: On Jan 11, 2017, at 7:29 PM, Josh Berkus wrote: It's been a long, fun ride, and I'm proud of the PostgreSQL we have today: both the database, and the community. Thank you for sharing it with me. Thank you for all of your

Re: [HACKERS] Packages: Again

2017-01-11 Thread Joshua D. Drake
On 01/11/2017 11:32 AM, Pavel Stehule wrote: We have a schemas instead - the PostgreSQL schema is close to Oracle packages. No. It isn't. A Package is essentially a class with dependencies. It has nothing to do with schemas outside of being named qualified. For example:

[HACKERS] Packages: Again

2017-01-11 Thread Joshua D. Drake
-hackers, I know we have talked about this before but today it was impressed upon me rather firmly. I presented a Webinar: Postgres for Oracle People. The attendees were 90% pl/pgsql developers. 330 people registered for an event that was only allowed to host 100 people. The webinar went on

Re: [HACKERS] RustgreSQL

2017-01-10 Thread Joshua D. Drake
On 01/10/2017 08:12 AM, Robert Haas wrote: Really? What language would you pick in a vacuum? The Linux kernel is written in C, too, for pretty much the same reasons: it's the canonical language for system software. I don't deny that there may be some newer languages out which could

Re: [HACKERS] Make pg_basebackup -x stream the default

2016-12-15 Thread Joshua D. Drake
On 12/15/2016 10:23 AM, Peter Eisentraut wrote: On 11/8/16 12:45 PM, Magnus Hagander wrote: Per some discussions with a number of different people at pgconfeu, here is a patch that changes the default mode of pg_basebackup to be streaming the wal, as this is what most users would want -- and

Re: pg_authid.rolpassword format (was Re: [HACKERS] Password identifiers, protocol aging and SCRAM protocol)

2016-12-14 Thread Joshua D. Drake
On 12/14/2016 11:41 AM, Stephen Frost wrote: * Heikki Linnakangas (hlinn...@iki.fi) wrote: On 14 December 2016 20:12:05 EET, Bruce Momjian wrote: On Wed, Dec 14, 2016 at 11:27:15AM +0100, Magnus Hagander wrote: Storing plaintext passwords has been bad form for just about

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2016-11-28 Thread Joshua D. Drake
On 11/28/2016 11:40 AM, Jim Nasby wrote: With current limits, the most bgwriter can do (with 8k pages) is 1000 pages * 100 times/sec = 780MB/s. It's not hard to exceed that with modern hardware. Should we increase the limit on bgwriter_lru_maxpages? Considering a single SSD can do 70% of that

Re: [HACKERS] Mail thread references in commits

2016-11-19 Thread Joshua D. Drake
On 11/17/2016 01:02 PM, Andrew Dunstan wrote: I love seeing references to email threads in commit messages. It would make them a lot friendlier though if a full http URL were included instead of just a Message-ID, i.e. instead of put

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

2016-11-10 Thread Joshua D. Drake
On 11/10/2016 09:33 AM, David Steele wrote: On 11/10/16 10:28 AM, Stephen Frost wrote: diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c [...] + if (log_checkpoints) + ereport(LOG, (errmsg("checkpoint

Re: [HACKERS] Issues with building snap packages and psql

2016-10-27 Thread Joshua D. Drake
On 10/26/2016 12:10 PM, Joshua D. Drake wrote: On 10/26/2016 11:33 AM, Robert Haas wrote: On Wed, Oct 26, 2016 at 2:22 PM, Joshua D. Drake It is also inconsistent with how other programs behave. For example if psql uses readline, and you change the value of $HOME, then readline uses $HOME

Re: [HACKERS] pg_hba_file_settings view patch

2016-10-26 Thread Joshua D. Drake
On 10/26/2016 12:54 PM, Josh Berkus wrote: On 10/26/2016 12:24 PM, Tom Lane wrote: Robert Haas writes: FWIW, I'm -1 on using JSON here. I don't believe that we should start using JSON all over the place just because we can. If we do that, we'll end up with a mishmash

Re: [HACKERS] Issues with building snap packages and psql

2016-10-26 Thread Joshua D. Drake
On 10/26/2016 11:33 AM, Robert Haas wrote: On Wed, Oct 26, 2016 at 2:22 PM, Joshua D. Drake <j...@commandprompt.com> wrote: After some further investigation and collaboration with #postgresql it looks like what we have here is an actual bug or at least improper implementation. Apparent

Re: [HACKERS] Issues with building snap packages and psql

2016-10-26 Thread Joshua D. Drake
Hello, After some further investigation and collaboration with #postgresql it looks like what we have here is an actual bug or at least improper implementation. Apparently, we use getpwuid on the euid to locate the 'home directory'. This is incorrect (as well as inconsistent to our own

Re: [HACKERS] Issues with building snap packages and psql

2016-10-26 Thread Joshua D. Drake
On 10/26/2016 10:26 AM, Andres Freund wrote: On October 26, 2016 8:09:11 PM GMT+03:00, "Joshua D. Drake" <j...@commandprompt.com> wrote: postgres=# \! export export HOME='/home/jd/snap/postgresql96/1' That doesn't do much. Isn't that just spawning a shell in which you s

Re: [HACKERS] Issues with building snap packages and psql

2016-10-26 Thread Joshua D. Drake
On 10/26/2016 10:16 AM, Tom Lane wrote: "Joshua D. Drake" <j...@commandprompt.com> writes: Is psql getting the home directory from /etc/passwd? Yes, of course. If so what can we do about that? Fix your /etc/passwd entry? Seems unlikely that psql is the only progr

[HACKERS] Issues with building snap packages and psql

2016-10-26 Thread Joshua D. Drake
Hello, CMD has been working on building snap packages. It has been an adventure and we are very close to having them complete. In fact they are complete enough that we have published them in the snap repo. jd@jd-wks:~$ snap find postgres NameVersion Developer Notes Summary

[HACKERS] Does it make sense to add a -q (quiet) flag to initdb?

2016-10-25 Thread Joshua D. Drake
Hello, Per: https://www.commandprompt.com/blog/can_i_make_initdb_quiet/ This was a question that was asked on #postgresql. Obviously we found a work around but I wonder if it makes sense to add a -q to solve some of these issues? (I could see it being useful with automation). Sincerely, JD

Re: [HACKERS] Remove autovacuum GUC?

2016-10-20 Thread Joshua D. Drake
Hello, What about a simpler solution to all of this. Let's just remove it from postgresql.conf. Out of sight. If someone needs to test they can but a uneducated user won't immediately know what to do about that "autovacuum process" and when they look it up the documentation is exceedingly

Re: [HACKERS] Remove autovacuum GUC?

2016-10-20 Thread Joshua D. Drake
On 10/20/2016 09:24 AM, Robert Haas wrote: On Thu, Oct 20, 2016 at 11:53 AM, Joshua D. Drake <j...@commandprompt.com> wrote: The right answer isn't the answer founded in the reality for many if not most of our users. I think that's high-handed nonsense. Sure, there are some unsophist

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-10-20 Thread Joshua D. Drake
On 10/20/2016 09:12 AM, Stephen Frost wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: Robert Haas writes: That said, I'd also like to see a --force or similar option or mechanism put in place to reduce the risk of users trashing their system because they think

Re: [HACKERS] Disable autovacuum guc?

2016-10-20 Thread Joshua D. Drake
On 10/20/2016 08:54 AM, Josh Berkus wrote: On 10/20/2016 06:34 AM, Joshua D. Drake wrote: On 10/19/2016 07:22 PM, Josh Berkus wrote: On 10/19/2016 06:27 PM, Joshua D. Drake wrote: Hrm, true although that is by far a minority of our users. What if we made it so we disabled the autovacuum guc

Re: [HACKERS] Remove autovacuum GUC?

2016-10-20 Thread Joshua D. Drake
On 10/20/2016 07:12 AM, Robert Haas wrote: On Wed, Oct 19, 2016 at 9:24 PM, Joshua D. Drake <j...@commandprompt.com> wrote: Setting autovacuum=off is at least useful for testing purposes and I've used it that way. On the other hand, I haven't seen a customer disable this unintenti

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Joshua D. Drake
On 10/20/2016 06:39 AM, Alvaro Herrera wrote: Bruce Momjian wrote: Also, it seems indirect indexes would be useful for indexing columns that are not updated frequently on tables that are updated frequently, and whose primary key is not updated frequently. That's quite a logic problem for

Re: [HACKERS] Disable autovacuum guc?

2016-10-20 Thread Joshua D. Drake
On 10/19/2016 07:22 PM, Josh Berkus wrote: On 10/19/2016 06:27 PM, Joshua D. Drake wrote: Hello, After all these years, we are still regularly running into people who say, "performance was bad so we disabled autovacuum". I am not talking about once in a while, it is often. I wou

[HACKERS] Remove autovacuum GUC?

2016-10-20 Thread Joshua D. Drake
Hello, After all these years, we are still regularly running into people who say, "performance was bad so we disabled autovacuum". I am not talking about once in a while, it is often. I would like us to consider removing the autovacuum option. Here are a few reasons: 1. It does not hurt

[HACKERS] Disable autovacuum guc?

2016-10-19 Thread Joshua D. Drake
Hello, After all these years, we are still regularly running into people who say, "performance was bad so we disabled autovacuum". I am not talking about once in a while, it is often. I would like us to consider removing the autovacuum option. Here are a few reasons: 1. It does not hurt

Re: [HACKERS] Indirect indexes

2016-10-18 Thread Joshua D. Drake
On 10/18/2016 11:28 AM, Alvaro Herrera wrote: Vacuuming presents an additional challenge: in order to remove index items from an indirect index, it's critical to scan the PK index first and collect the PK values that are being removed. Then scan the indirect index and remove any items that

[HACKERS] restoration after crash slowness, any way to improve?

2016-08-31 Thread Joshua D. Drake
-hackers, So this is more of a spit balling thread than anything. As I understand it, if we have a long running transaction or a large number of wal logs and we crash, we then have to restore those logs on restart to the last known good transaction. No problem. I recently ran a very long

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-08-29 Thread Joshua D. Drake
On 08/29/2016 10:00 AM, Daniel Verite wrote: Let's imagine that pg_xlog is named wal instead. How does that help our user with the disk space problem? Does that point to a path of resolution? I don't see it. What do we think that user's next move will be? After all, WAL means Write Ahead *Log*.

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-08-29 Thread Joshua D. Drake
On 08/29/2016 08:07 AM, Tom Lane wrote: "Joshua D. Drake" <j...@commandprompt.com> writes: Also as a note to the idea that we make break things for external user space; the next version being v10 is the exact time to do that. Let's please drop this meme that "v10 is

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-08-29 Thread Joshua D. Drake
On 08/29/2016 06:42 AM, Daniel Verite wrote: Aside from that, we might also question how much of the excuse "pg_xlog looked like it was just deletable logs" is a lie made up after the fact, because anybody wrecking a database is not against deflecting a bit of the blame to the software, that's

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-08-29 Thread Joshua D. Drake
On 08/29/2016 12:04 AM, Magnus Hagander wrote: On Mon, Aug 29, 2016 at 2:45 AM, Jim Nasby > wrote: On 8/26/16 4:08 PM, Andres Freund wrote: Splitting of ephemeral data seems to have a benefit, the rest seems more

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-08-29 Thread Joshua D. Drake
On 08/27/2016 11:11 AM, Tom Lane wrote: Alvaro Herrera writes: I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that. I think that would make sense if we were to relocate

Re: [HACKERS] pg_dump with tables created in schemas created by extensions

2016-08-26 Thread Joshua D. Drake
On 08/26/2016 04:15 PM, Tomas Vondra wrote: On 08/27/2016 12:37 AM, Tom Lane wrote: =?UTF-8?B?TWFydMOtbiBNYXJxdcOpcw==?= writes: Looking at this issue today, I found that we are not setting a dependency for an index created inside an extension. Surely the index has a

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-08-26 Thread Joshua D. Drake
On 08/26/2016 09:28 AM, Tom Lane wrote: Magnus Hagander writes: On Aug 26, 2016 5:54 PM, "Peter Eisentraut" < peter.eisentr...@2ndquadrant.com> wrote: If we're going to do some renaming, then I suggest we do a mini-file-system structure under $PGDATA, like $PGDATA/etc

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-08-26 Thread Joshua D. Drake
On 08/25/2016 07:39 PM, Michael Paquier wrote: Hi all, I am relaunching $subject as 10 development will begin soon. As far as I know, there is agreement that we can do something here. Among the different proposals I have found: - pg_clog renamed to pg_commit_status, pg_xact or pg_commit -

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-08-26 Thread Joshua D. Drake
On 08/26/2016 03:48 AM, Magnus Hagander wrote: Same reason I'm also +1 for Stephens suggestion to put all things that should not be in a base backup into the same directory. That may break things now, but it will simplify things down the road. And doing it at the same time as renaming these

Re: [HACKERS] [GENERAL] C++ port of Postgres

2016-08-16 Thread Joshua D. Drake
On 08/16/2016 09:33 AM, Robert Haas wrote: On Tue, Aug 16, 2016 at 10:47 AM, Jim Nasby wrote: On 8/16/16 2:52 AM, Gavin Flower wrote: I agree with your statement that one of our biggest problems is getting more developers interested in working on PostgreSQL. Even

Re: [HACKERS] No longer possible to query catalogs for index capabilities?

2016-08-11 Thread Joshua D. Drake
On 08/11/2016 10:46 AM, Kevin Grittner wrote: On Wed, Aug 10, 2016 at 5:14 PM, Tom Lane wrote: Kevin Grittner writes: But a DBA who has a problem doesn't care what the truth will be in a year or two -- the interest is in *right now* on one particular

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-10 Thread Joshua D. Drake
On 08/10/2016 09:04 AM, Stephen Frost wrote: * Joshua D. Drake (j...@commandprompt.com) wrote: +1 for Robert here, removing async commit is a non-starter. It is PostgreSQL performance 101 that you disable synchronous commit unless you have a specific data/business requirement that needs

  1   2   3   4   5   6   7   8   9   10   >