Re: [HACKERS] Mac OS: invalid byte sequence for encoding "UTF8"

2016-02-10 Thread Teodor Sigaev
It seems that *scanf() with %s format occures only here: - check.c - get_bin_version() - server.c - get_major_server_version() - filemap.c - isRelDataFile() - pg_backup_directory.c - _LoadBlobs() - xlog.c - do_pg_stop_backup() - mac.c - macaddr_in() I think here sscanf() do not works with the

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread Stephen Frost
* David Steele (da...@pgmasters.net) wrote: > On 2/10/16 9:44 AM, Stephen Frost wrote: > > Hrmmm. If that's the case then perhaps you're right. I liked the > > general idea of not having to maintain a TCP connection during the > > entire backup (TCP connections can be annoyingly finicky in

Re: [HACKERS] Tracing down buildfarm "postmaster does not shut down" failures

2016-02-10 Thread Tom Lane
Noah Misch writes: >>> That's reasonable. If you would like higher-fidelity data, I can run loops >>> of >>> "pg_ctl -w start; make installcheck; pg_ctl -t900 -w stop", and I could run >>> that for HEAD and 9.2 simultaneously. A day of logs from that should show >>> clearly

Re: [HACKERS] extend pgbench expressions with functions

2016-02-10 Thread Michael Paquier
On Tue, Feb 9, 2016 at 5:06 AM, Alvaro Herrera wrote: > Fabien COELHO wrote: >> >> Hello, >> >> >>v23 attached, which does not change the message but does the other fixes. >> > >> >This doesn't apply anymore >> >> Indeed, but the latest version was really v25. >> >> >--

Re: [HACKERS] extend pgbench expressions with functions

2016-02-10 Thread Fabien COELHO
Hello Michaël, + parameter% of the time. Nitpick: double space here. Ok. + default: + return false; } In evalFunc(), the default case in switch for the operator functions should never be reached. Adding for example Assert(0) is

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread Magnus Hagander
On Wed, Feb 10, 2016 at 4:38 PM, David Steele wrote: > On 2/10/16 7:46 AM, Magnus Hagander wrote: > > Per discussion at the developer meeting in Brussels, here's a patch that > > makes some updates to the backup APIs, to support non-exclusive backups > > without using

Re: [HACKERS] Mac OS: invalid byte sequence for encoding "UTF8"

2016-02-10 Thread Artur Zakirov
On 09.02.2016 20:13, Tom Lane wrote: I do not like this patch much. It is basically "let's stop using sscanf() because it seems to have a bug on one platform". There are at least two things wrong with that approach: 1. By my count there are about 80 uses of *scanf() in our code. Are we going

Re: [HACKERS] Optimization for updating foreign tables in Postgres FDW

2016-02-10 Thread Thom Brown
On 10 February 2016 at 08:00, Rushabh Lathia wrote: > Fujita-san, I am attaching update version of the patch, which added > the documentation update. > > Once we finalize this, I feel good with the patch and think that we > could mark this as ready for committer. I find

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread Andres Freund
Hi, On 2016-02-10 13:46:05 +0100, Magnus Hagander wrote: > Per discussionat the developer meeting in Brussels, here's a patch that > makes some updates to the backup APIs, to support non-exclusive backups > without using pg_basebackup. Thanks for following through with this! > * If the client

Re: [HACKERS] Fuzzy substring searching with the pg_trgm extension

2016-02-10 Thread Artur Zakirov
On 02.02.2016 15:45, Artur Zakirov wrote: On 01.02.2016 20:12, Artur Zakirov wrote: I have changed the patch: 1 - trgm2.data was corrected, duplicates were deleted. 2 - I have added operators <<-> and <->> with GiST index supporting. A regression test will pass only with the patch

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread Stephen Frost
* Andres Freund (and...@anarazel.de) wrote: > On 2016-02-10 13:46:05 +0100, Magnus Hagander wrote: > > * If the client disconnects with a non-exclusive backup running, the backup > > is automatically aborted. This is the same thing that pg_basebackup does. > > To use these non-exclusive backups

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread Stephen Frost
* Magnus Hagander (mag...@hagander.net) wrote: > On Wed, Feb 10, 2016 at 2:46 PM, Andres Freund wrote: > > On 2016-02-10 13:46:05 +0100, Magnus Hagander wrote: > > > Per discussionat the developer meeting in Brussels, here's a patch that > > > makes some updates to the backup

Re: [HACKERS] [PATCH] Refactoring of LWLock tranches

2016-02-10 Thread Robert Haas
On Wed, Feb 10, 2016 at 1:32 AM, Amit Kapila wrote: > The reason for using centralized way is that we need to request > named tranches before initialization of shared memory and as far as > I can see, currently there is no way in the subsystems where they can > issue such

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread David Steele
On 2/10/16 7:46 AM, Magnus Hagander wrote: > Per discussion at the developer meeting in Brussels, here's a patch that > makes some updates to the backup APIs, to support non-exclusive backups > without using pg_basebackup. <...> This sounds like a great idea and I have signed up to review. > * A

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread Magnus Hagander
On Wed, Feb 10, 2016 at 2:46 PM, Andres Freund wrote: > Hi, > > On 2016-02-10 13:46:05 +0100, Magnus Hagander wrote: > > Per discussionat the developer meeting in Brussels, here's a patch that > > makes some updates to the backup APIs, to support non-exclusive backups > >

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread David Steele
On 2/10/16 9:44 AM, Stephen Frost wrote: > * Magnus Hagander (mag...@hagander.net) wrote: >> On Wed, Feb 10, 2016 at 2:46 PM, Andres Freund wrote: >>> On 2016-02-10 13:46:05 +0100, Magnus Hagander wrote: * If the client disconnects with a non-exclusive backup running, the

Re: [HACKERS] Tracing down buildfarm "postmaster does not shut down" failures

2016-02-10 Thread Andrew Dunstan
On 02/09/2016 11:21 PM, Andrew Dunstan wrote: The idea I was toying with is that previous filesystem activity (making the temp install, the server's never-fsync'd writes, etc) has built up a bunch of dirty kernel buffers, and at some point the kernel goes nuts writing all that data. So

Re: [HACKERS] [PROPOSAL] Improvements of Hunspell dictionaries support

2016-02-10 Thread Teodor Sigaev
I duplicate the patch here. it's very good thing to update disctionaries to support modern versions. And thank you for improving documentation. Also I've impressed by long description in spell.c header. Som notices about code: 1 struct SPELL. Why do you remove union p? You leave comment

Re: [HACKERS] old bug in full text parser

2016-02-10 Thread Mike Rylander
On Wed, Feb 10, 2016 at 4:28 AM, Oleg Bartunov wrote: > It looks like there is a very old bug in full text parser (somebody pointed > me on it), which appeared after moving tsearch2 into the core. The problem > is in how full text parser process hyphenated words. Our

Re: [HACKERS] Mac OS: invalid byte sequence for encoding "UTF8"

2016-02-10 Thread Tom Lane
Artur Zakirov writes: > I agree that previous patch is wrong. Instead of using new > parse_ooaffentry() function maybe better to use sscanf() with %ls > format. The %ls format is used to read a wide character string. No, that way is going to give you worse portability

[HACKERS] Some refactoring of index structures .

2016-02-10 Thread Anastasia Lubennikova
Hi, hackers. Long story short, I'd like to do some refactoring of the code related to indexes. I work on patch which provides INCLUDING columns functional [1]. The patch was reviewed again and again, I fixed many issues, but we still don't sure enough that all occurrences of

Re: [HACKERS] [COMMITTERS] pgsql: Code cleanup in the wake of recent LWLock refactoring.

2016-02-10 Thread Craig Ringer
On 11 February 2016 at 00:21, Robert Haas wrote: > > > If there's a strong feeling that we should keep the old APIs around, > we can do that, but I think that (1) if we don't remove them now, we > probably never will and (2) they are vile APIs. Yep. Delete 'em. Things

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread David Steele
On 2/10/16 11:01 AM, Andres Freund wrote: > On 2016-02-10 16:50:26 +0100, Magnus Hagander wrote: >>> I would be happy to see the time-stamp returned from the >>> pg_start_backup() function as well. It's a bigger change, but once >>> pg_start_backup() returns multiple columns it will be easier to

Re: [HACKERS] Tracing down buildfarm "postmaster does not shut down" failures

2016-02-10 Thread Andres Freund
On 2016-02-09 22:27:07 -0500, Tom Lane wrote: > The idea I was toying with is that previous filesystem activity (making > the temp install, the server's never-fsync'd writes, etc) has built up a > bunch of dirty kernel buffers, and at some point the kernel goes nuts > writing all that data. So

Re: [HACKERS] [COMMITTERS] pgsql: Code cleanup in the wake of recent LWLock refactoring.

2016-02-10 Thread Robert Haas
On Wed, Feb 10, 2016 at 11:08 AM, Heikki Linnakangas wrote: > On 10/02/16 17:12, Robert Haas wrote: >> Code cleanup in the wake of recent LWLock refactoring. >> >> As of commit c1772ad9225641c921545b35c84ee478c326b95e, there's no >> longer any way of requesting additional LWLocks

Re: [HACKERS] old bug in full text parser

2016-02-10 Thread Tom Lane
Oleg Bartunov writes: > It looks like there is a very old bug in full text parser (somebody > pointed me on it), which appeared after moving tsearch2 into the core. The > problem is in how full text parser process hyphenated words. Our original > idea was to report

Re: [HACKERS] proposal: schema PL session variables

2016-02-10 Thread Pavel Stehule
2016-02-09 23:31 GMT+01:00 Jim Nasby : > On 2/8/16 10:02 AM, Pavel Stehule wrote: > >> >> I think it would make sense to implement the interface in at least >> one of our other supported PLs. I'm not entirely clear how well this >> will match up with, say,

Re: [HACKERS] [COMMITTERS] pgsql: Code cleanup in the wake of recent LWLock refactoring.

2016-02-10 Thread Tom Lane
Robert Haas writes: > On Wed, Feb 10, 2016 at 11:08 AM, Heikki Linnakangas wrote: >> (Sorry if this was discussed already, I haven't been paying attention) >> >> LWLockAssign() is used by extensions. Are we OK with just breaking them, >> requiring them to

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread Andres Freund
On 2016-02-10 11:06:01 -0500, David Steele wrote: > That makes sense. The backup_label "as is" could be output at the > beginning but if we want to add the minimum recovery point it would need > to be output at the end. > > It seems like tablespace_map could still be output at the beginning,

Re: [HACKERS] Patch: fix lock contention for HASHHDR.mutex

2016-02-10 Thread Robert Haas
On Wed, Feb 10, 2016 at 3:24 AM, Aleksander Alekseev wrote: >> Basically, the burden for you to impose a new coding rule on everybody >> who uses shared hash tables in the future is very high. > > I fixed an issue you described. Number of spinlocks doesn't depend of >

Re: [HACKERS] Mac OS: invalid byte sequence for encoding "UTF8"

2016-02-10 Thread Artur Zakirov
On 10.02.2016 18:51, Teodor Sigaev wrote: Hmm. Here src/backend/access/transam/xlog.c read_tablespace_map() using %s in scanf looks suspisious. I don't fully understand but it looks like it tries to read oid as string. So, it should be safe in usial case Next, _LoadBlobs() reads filename

Re: [HACKERS] proposal: schema PL session variables

2016-02-10 Thread Pavel Stehule
Hi Would it make sense to explicitly import variables in function definitions? > > CREATE SESSION VARIABLE foo integer; > CREATE SESSION VARIABLE my_schema.bar text; > SET SESSION VARIABLE foo to 4; > SET SESSION VARIABLE my_schema.bar to 'hi mom'; > > CREATE FUNCTION my_func (p_param text)

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread Andres Freund
On 2016-02-10 16:50:26 +0100, Magnus Hagander wrote: > > I would be happy to see the time-stamp returned from the > > pg_start_backup() function as well. It's a bigger change, but once > > pg_start_backup() returns multiple columns it will be easier to add more > > columns in the future. > > > >

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread David Steele
On 2/10/16 10:50 AM, Magnus Hagander wrote: > On Wed, Feb 10, 2016 at 4:38 PM, David Steele > This information is handy for automating selection of a backup when > doing time-based PITR (or so the user can manually select). For > exclusive backups it is possible

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-02-10 Thread David Steele
On 2/10/16 11:12 AM, Andres Freund wrote: > On 2016-02-10 11:06:01 -0500, David Steele wrote: >> That makes sense. The backup_label "as is" could be output at the >> beginning but if we want to add the minimum recovery point it would need >> to be output at the end. >> >> It seems like

Re: [HACKERS] proposal: schema PL session variables

2016-02-10 Thread Pavel Stehule
2016-02-09 20:55 GMT+01:00 David G. Johnston : > On Tue, Feb 9, 2016 at 11:32 AM, Corey Huinker > wrote: > >> >> Oh, and I suggest we call them SESSION variables rather than SCHEMA >> variables, to reinforce the idea of how long the values in

Re: [HACKERS] Way to check whether a particular block is on the shared_buffer?

2016-02-10 Thread Robert Haas
On Tue, Feb 9, 2016 at 6:35 PM, Kouhei Kaigai wrote: > Unfortunately, it was not sufficient. > > Due to the assumption, the buffer page to be suspended does not exist > when a backend process issues a series P2P DMA command. (If block would > be already loaded to the shared

Re: [HACKERS] Improve docs wrt catalog object ACLs

2016-02-10 Thread Tom Lane
Stephen Frost writes: > The way permissions on catalog objects are handled isn't discussed at > all in the documentation. Barring objections, I'll commit and > back-patch the attached to improve that situation in the next day or so. I have no objection to the goal, but I do

Re: [HACKERS] proposal: schema PL session variables

2016-02-10 Thread Jim Nasby
On 2/10/16 11:25 AM, Pavel Stehule wrote: Oh, and I suggest we call them SESSION variables rather than SCHEMA variables, to reinforce the idea of how long the values in the variables live. A session variable is in a sense a 1x1 temp table, whose definition persists across

Re: [HACKERS] proposal: schema PL session variables

2016-02-10 Thread Pavel Stehule
>> I didn't propose SESSION variables - now there are some workarounds how >> to anybody can emulate it, so this feature can wait. What we need is >> safe session variables with limited access. And the border can be >> defined by schema scope. So the keyword SCHEMA has sense, and it is >>

Re: [HACKERS] proposal: schema PL session variables

2016-02-10 Thread Pavel Stehule
2016-02-10 20:25 GMT+01:00 Jim Nasby : > On 2/10/16 1:17 PM, Pavel Stehule wrote:It is too simple and too like > workaround :) I can do it this in plpgsql > >> extension probably. >> > > I think it's something people will definitely want. If we don't have it, > then

[HACKERS] Improve docs wrt catalog object ACLs

2016-02-10 Thread Stephen Frost
Greetings, The way permissions on catalog objects are handled isn't discussed at all in the documentation. Barring objections, I'll commit and back-patch the attached to improve that situation in the next day or so. Thanks! Stephen From ad8e663893ea906238a9c0346bf8791eafe3d333 Mon Sep 17

Re: [HACKERS] proposal: PL/Pythonu - function ereport

2016-02-10 Thread Pavel Stehule
Hi > >> Actually I imagine that if there's no agreement between author and first >> reviewer, there might not *be* a committer in the first place. Perhaps >> try to get someone else to think about it and make a decision. It is >> possible that some other committer is able to decide by

Re: [HACKERS] proposal: schema PL session variables

2016-02-10 Thread Jim Nasby
On 2/10/16 1:04 PM, Pavel Stehule wrote: BTW, if all that's desired here are session variables for plpgsql, I think it makes a lot more sense to start with implementing per-function session variables. That's a lot simpler design-wise and is something we should have anyway. You

Re: [HACKERS] proposal: PL/Pythonu - function ereport

2016-02-10 Thread Jim Nasby
On 2/10/16 12:44 PM, Pavel Stehule wrote: FWIW, I'd think it's better to not break backwards compatibility, but I'm also far from a python expert. It might well be worth adding a plpython GUC to control the behavior so that there's a migration path forward, or maybe do something

Re: [HACKERS] proposal: function parse_ident

2016-02-10 Thread Jim Nasby
On 2/10/16 12:26 PM, Jim Nasby wrote: I editorialized the docs and some of the comments. In particular, I documented behavior of not truncating, and recommended casting to name[] if user cares about that. Added a unit test to verify that works. BTW, I saw mention in the thread about not

Re: [HACKERS] proposal: schema PL session variables

2016-02-10 Thread Jim Nasby
On 2/10/16 11:33 AM, Pavel Stehule wrote: I don't think that really makes sense. I would expect schema variables to be exposed to a function as variables or attributes, either in the global namespace for that PL, or as an attribute of some object (ie the plpy object in

Re: [HACKERS] proposal: schema PL session variables

2016-02-10 Thread Pavel Stehule
2016-02-10 20:10 GMT+01:00 Jim Nasby : > On 2/10/16 1:04 PM, Pavel Stehule wrote: > >> >> BTW, if all that's desired here are session variables for plpgsql, I >> think it makes a lot more sense to start with implementing >> per-function session variables.

Re: [HACKERS] Tracing down buildfarm "postmaster does not shut down" failures

2016-02-10 Thread Tom Lane
Andrew Dunstan writes: >> Yeah. It's faintly possible that a kernel upgrade will help. > Another data point. I have another RPi2B that is running Debian Wheezy > rather than the Fedora remix. I'm running the same test on it we ran > yesterday on axolotl. It seems to be

Re: [HACKERS] proposal: function parse_ident

2016-02-10 Thread Jim Nasby
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested NEEDS CATVERSION BUMP I editorialized the docs and some of the

[HACKERS] eXtensible Transaction Manager API (v2)

2016-02-10 Thread Konstantin Knizhnik
Hi, PostgresProffesional cluster teams wants to propose new version of eXtensible Transaction Manager API. Previous discussion concerning this patch can be found here: http://www.postgresql.org/message-id/f2766b97-555d-424f-b29f-e0ca0f6d1...@postgrespro.ru The API patch itself is small

Re: [HACKERS] proposal: schema PL session variables

2016-02-10 Thread Jim Nasby
On 2/10/16 11:54 AM, Pavel Stehule wrote: 2016-02-09 23:41 GMT+01:00 Jim Nasby >: The other big thing you get is public vs private. You can sorta-kinda-almost simulate that with permissions in simple cases, but it ultimately

Re: [HACKERS] proposal: schema PL session variables

2016-02-10 Thread Jim Nasby
On 2/10/16 1:17 PM, Pavel Stehule wrote: 2016-02-10 20:10 GMT+01:00 Jim Nasby >: On 2/10/16 1:04 PM, Pavel Stehule wrote: BTW, if all that's desired here are session variables for plpgsql, I

Re: [HACKERS] proposal: schema PL session variables

2016-02-10 Thread Pavel Stehule
>> >> The schema variables are private by design. It can be enhanced in >> future, but now it is out my scope. If you need public access to these >> variables, you can use a functions. The access to functions can be >> controlled by a rights. We can introduce a private (schema limited) >> function

[HACKERS] Moving responsibility for logging "database system is shut down"

2016-02-10 Thread Tom Lane
In our recent investigations of slow shutdown on some buildfarm members, it became clear that "database system is shut down" can get logged many seconds before the postmaster actually exits; the main cause of delay is that the stats collector's final data dump can take awhile. This is pretty

Re: [HACKERS] Optimization for updating foreign tables in Postgres FDW

2016-02-10 Thread Robert Haas
On Wed, Feb 10, 2016 at 3:00 AM, Rushabh Lathia wrote: > Fujita-san, I am attaching update version of the patch, which added > the documentation update. > > Once we finalize this, I feel good with the patch and think that we > could mark this as ready for committer. It

Re: [HACKERS] old bug in full text parser

2016-02-10 Thread Oleg Bartunov
On Wed, Feb 10, 2016 at 7:21 PM, Tom Lane wrote: > Oleg Bartunov writes: > > It looks like there is a very old bug in full text parser (somebody > > pointed me on it), which appeared after moving tsearch2 into the core. > The > > problem is in how full

Re: [HACKERS] pl/pgSQL, get diagnostics and big data

2016-02-10 Thread Andreas 'ads' Scherbaum
Hello, thanks for reviewing the patch! On 09.02.2016 20:32, Christian Ullrich wrote: - Are there portability issues/Will it work on Windows/BSD etc.: No, it will not work correctly on Windows when built with MSVC, although it may work with MinGW. +++

Re: [HACKERS] old bug in full text parser

2016-02-10 Thread Oleg Bartunov
On Wed, Feb 10, 2016 at 7:45 PM, Mike Rylander wrote: > On Wed, Feb 10, 2016 at 4:28 AM, Oleg Bartunov > wrote: > > It looks like there is a very old bug in full text parser (somebody > pointed > > me on it), which appeared after moving tsearch2 into

Re: [HACKERS] [PATCH v4] GSSAPI encryption support

2016-02-10 Thread Robbie Harwood
Hello friends, For your consideration, here is a new version of GSSAPI encryption support. For those who prefer, it's also available on my github: https://github.com/frozencemetery/postgres/commit/c92275b6605d7929cda5551de47a4c60aab7179e Some thoughts: - The overall design is different this

Re: [HACKERS] Moving responsibility for logging "database system is shut down"

2016-02-10 Thread Robert Haas
On Wed, Feb 10, 2016 at 3:13 PM, Tom Lane wrote: > In our recent investigations of slow shutdown on some buildfarm members, > it became clear that "database system is shut down" can get logged many > seconds before the postmaster actually exits; the main cause of delay > is

Re: [HACKERS] Mac OS: invalid byte sequence for encoding "UTF8"

2016-02-10 Thread Tom Lane
I wrote: > Artur Zakirov writes: >> I think this is not a bug. It is a normal behavior. In Mac OS sscanf() >> with the %s format reads the string one character at a time. The size of >> letter 'х' is 2. And sscanf() separate it into two wrong characters. > That

Re: [HACKERS] Mac OS: invalid byte sequence for encoding "UTF8"

2016-02-10 Thread Larry Rosenman
On 2016-02-10 16:19, Tom Lane wrote: I wrote: Artur Zakirov writes: I think this is not a bug. It is a normal behavior. In Mac OS sscanf() with the %s format reads the string one character at a time. The size of letter 'х' is 2. And sscanf() separate it into two

Re: [HACKERS] Moving responsibility for logging "database system is shut down"

2016-02-10 Thread Tom Lane
Robert Haas writes: > On Wed, Feb 10, 2016 at 3:13 PM, Tom Lane wrote: >> So I propose the attached patch. Any objections? Should this get >> back-patched? It's arguably a bug, though surely a minor one, that >> the message comes out when it does. >

Re: [HACKERS] Mac OS: invalid byte sequence for encoding "UTF8"

2016-02-10 Thread Tom Lane
Larry Rosenman writes: > On 2016-02-10 16:19, Tom Lane wrote: >> I looked into the OS X sources, and found that indeed you are right: >> *scanf processes the input a byte at a time, and applies isspace() to >> each byte separately, even when the locale is such that that's a >>

Re: [HACKERS] proposal: schema PL session variables

2016-02-10 Thread Pavel Stehule
Hi 2016-02-09 23:41 GMT+01:00 Jim Nasby : > On 2/9/16 4:13 PM, Corey Huinker wrote: > >> >> We're not going to get source compatibility without implementing >> packages, and there's no enthusiasm for that. It's been stated a few >> times before by some that the only

Re: [HACKERS] Tracing down buildfarm "postmaster does not shut down" failures

2016-02-10