Re: [HACKERS] wrapping in extended mode doesn't work well with default pager

[Pavel Stehule]

Hello

With this patch it works perfect

Thank you

Regards

Pavel


2014-05-13 21:33 GMT+02:00 Sergey Muraviov :

> Please check this patch.
>
>
> 2014-05-12 22:56 GMT+04:00 Sergey Muraviov :
>
> Hi.
>>
>> I'll try to fix it tomorrow.
>>
>>
>> 2014-05-12 18:42 GMT+04:00 Tom Lane :
>>
>> Greg Stark  writes:
>>> > On Mon, May 12, 2014 at 2:12 PM, Greg Stark  wrote:
>>> >> Hm, there was an off by one error earlier in some cases, maybe we
>>> >> fixed it by breaking other case. Will investigate.
>>>
>>> > Those spaces are coming from the ascii wrapping indicators. i.e. the
>>> periods in:
>>>
>>> Ah.  I wonder whether anyone will complain that the format changed?
>>>
>>> > Apparently we used to print those with border=1 in normal mode but in
>>> > expanded mode we left out the space for those on the outermost edges
>>> > since there was no need for them. If we put them in for wrapped mode
>>> > then we'll be inconsistent if we don't for nonwrapped mode though. And
>>> > if we don't put them in for wrapped mode then there's no way to
>>> > indicate wrapping versus newlines.
>>>
>>> Barring anyone complaining that the format changed, I'd say the issue
>>> is not that you added them but that the accounting for line length
>>> fails to include them.
>>>
>>> regards, tom lane
>>>
>>
>>
>>
>> --
>> Best regards,
>> Sergey Muraviov
>>
>
>
>
> --
> Best regards,
> Sergey MuraviovH
>

Re: [HACKERS] regression tests fail for different block sizes

[Tom Lane]

Jeff Davis  writes:
> In master, 4 tests fail due to plan changes with blocksize 32K. The
> failures started creeping in around 9.0.

AFAIR, that's been true since the dark ages, not since 9.0.

> I don't see a clean way to make the plans consistent with 8K and 32K
> pages, so I was about to write this off as "we don't care much".

Indeed.  There are any number of parameters that you can change that will
break the regression tests by changing plan choices.  If we tried to
cover all such cases we'd end up with lobotomized tests that detect
fewer issues, not more.

> Then I ran with block size 1KB, and I got what appears to be a genuine
> failure related to range types and/or spgist.

This might be worth investigating, but it doesn't mean that we need
automated regression tests covering the case.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] regression tests fail for different block sizes

[Jeff Davis]

In master, 4 tests fail due to plan changes with blocksize 32K. The
failures started creeping in around 9.0.

I don't see a clean way to make the plans consistent with 8K and 32K
pages, so I was about to write this off as "we don't care much".

Then I ran with block size 1KB, and I got what appears to be a genuine
failure related to range types and/or spgist. I'm still investigating
that, which will be a separate discussion, but the fact that different
block sizes can catch real errors makes me think we should properly
support them in the regression tests.

Any ideas how we can make the plans consistent enough when the blocksize
varies?

Regards,
Jeff Davis




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] gettimeofday is at the end of its usefulness?

[Jeff Janes]

On Tuesday, May 13, 2014, Tom Lane  wrote:

> A recent question from Tim Kane prompted me to measure the overhead
> costs of EXPLAIN ANALYZE, which I'd not checked in awhile.  Things
> are far worse than I thought.  On my current server (by no means
> lavish hardware: Xeon E5-2609 @2.40GHz) a simple seqscan can run
> at something like 110 nsec per row:
>
> regression=# create table foo as select x as f1 from
> generate_series(1,100) x;
> SELECT 100
> regression=# vacuum foo;
> VACUUM
> regression=# explain analyze select * from foo;
>   QUERY PLAN
>
> ---
>  Seq Scan on foo  (cost=0.00..14425.00 rows=100 width=4) (actual
> time=0.053..111.720 rows=100 loops=1)
>  Planning time: 0.222 ms
>  Execution time: 166.682 ms
> (3 rows)
>
> (and, btw, this is a debug build --- without assert and memory
> context checks it'd be faster.)
>
> The problem with this number is that a simple test program shows that
> gettimeofday() requires about 40 nsec on this hardware.  That means
> that two-thirds of the above timing measurement is overhead.
>

I'm all for finding something better if we can, but in the mean time this
is certainly not unexpected, and isn't it exactly what "explain
(analyze,timing off)" was invented for?

Cheers,

Jeff

Re: [HACKERS] 9.5: UPDATE/DELETE .. ORDER BY .. LIMIT ..

[Amit Kapila]

On Tue, May 13, 2014 at 8:04 PM, Tom Lane  wrote:
> Amit Kapila  writes:
>> How about sorting step, are you thinking to have MergeAppend
>> node for it beneath ModifyTable?
>
> Well yeah, that's pretty much the point.

IIUC, the way new design will work is that for new tuple we will now
get tableoid+TID, modified column values as an input (for inheritance
tables we will get this for all child tables as well) for ModifyTable
and get old tuple (which in current case will be provided by MergeAppend
or in general by some scan node) from some node beneath the
ModifyTable.  It then matches the tableoid from old tuple with appropriate
tableoid incase of child tables and then form the new tuple for that
tableoid using old tuple and modified column values.
In this case can we safely assume that we will always get tableoid from
old tuple, ideally it should be there but just not sure and another minor
point is won't we get TID from old tuple (tuple we get from node beneath
ModifyTable), what's the need to pass for new tuple?

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] sepgsql: label regression test failed

[Sergey Muraviov]

Hi.

Some regression tests for sepgsql still not work on Fedora 20:

== running regression test queries==
test label... FAILED
test dml  ... ok
test ddl  ... FAILED
test alter... FAILED
test misc ... ok

==
 3 of 5 tests failed.
==

$ sestatus
SELinux status: enabled
SELinuxfs mount:/sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode:   enforcing
Mode from config file:  enforcing
Policy MLS status:  enabled
Policy deny_unknown status: allowed
Max kernel policy version:  29

$ uname -i -o -r
3.14.3-200.fc20.x86_64 x86_64 GNU/Linux

$ /usr/local/pgsql/bin/postgres --version
postgres (PostgreSQL) 9.4beta1

PS
I've got this compiler warning:
 relation.c: In function ‘sepgsql_relation_drop’:
relation.c:472:25: warning: ‘tclass’ may be used uninitialized in this
function [-Wmaybe-uninitialized]
  sepgsql_avc_check_perms(&object,
 ^


2013-12-25 0:34 GMT+04:00 Kohei KaiGai :

> Hello,
>
> It seems to me changes in the base security policy on Fedora affected to
> the regression test. Our test cases for sepgsql_setcon() utilizes the MCS
> rules, that prevents domain transition from narrow categories to wider
> ones,
> to control the success cases and failure cases.
>
> However, its coverage was changed. It was applied all the domains in the
> system, thus "unconfined_t" domain had been enforced by MCS rules.
> But now, it shall be applied only domains with "mcs_constrained_type"
> attribute.
>
> [kaigai@vmlinux tmp]$ diff -up old/policy/mcs new/policy/mcs
>   :
>  
>   :
>  mlsconstrain process { transition dyntransition }
> -   (( h1 dom h2 ) or ( t1 == mcssetcats ));
> +   (( h1 dom h2 ) or ( t1 != mcs_constrained_type ));
>
> Probably, we need to define a domain by ourselves for regression test to
> ensure
> the test stability, not using the system "unconfined" domain that has
> different
> meaning by release.
>
> I'll make a patch. Please wait for a while.
>
> Thanks for your test & reports.
>
> 2013/12/18 Sergey Muraviov :
> > # semodule -l | grep sepgslq
> > sepgsql-regtest 1.07
> >
> > Full list of modules is in attachment.
> >
> >
> > 2013/12/18 Kohei KaiGai 
> >>
> >> Could you show me semodule -l on your environment?
> >> I believe security policy has not been changed between F19 and F20...
> >>
> >> Thanks,
> >>
> >> 2013/12/18 Sergey Muraviov :
> >> > Hi
> >> >
> >> > I've tried to test postgres 9.3.2 and 9.4devel with selinux on Fedora
> 20
> >> > and
> >> > met with a label regression test failure.
> >> >
> >> > PS
> >> > I've got some warning during build process.
> >> >
> >> > --
> >> > Best regards,
> >> > Sergey Muraviov
> >> >
> >> >
> >> > --
> >> > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> >> > To make changes to your subscription:
> >> > http://www.postgresql.org/mailpref/pgsql-hackers
> >> >
> >>
> >>
> >>
> >> --
> >> KaiGai Kohei 
> >
> >
> >
> >
> > --
> > Best regards,
> > Sergey Muraviov
>
>
>
> --
> KaiGai Kohei 
>



-- 
Best regards,
Sergey Muraviov


regression.diffs
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal for CSN based snapshots

[Rajeev rastogi]

On 13 May 2014 14:06, Heikki Linnakangas

> >> >The core of the design is to store the LSN of the commit record in
> >> >pg_clog. Currently, we only store 2 bits per transaction there,
> >> >indicating if the transaction committed or not, but the patch will
> >> >expand it to 64 bits, to store the LSN. To check the visibility of
> >> >an XID in a snapshot, the XID's commit LSN is looked up in pg_clog,
> >> >and compared with the snapshot's LSN.
> > Isn't it will be bit in-efficient to look in to pg_clog to read XID's
> > commit LSN for every visibility check?
> 
> Maybe. If no hint bit is set on the tuple, you have to check the clog
> anyway to determine if the tuple is committed. And if for XIDs older
> than xmin or newer than xmax, you don't need to check pg_clog. But it's
> true that for tuples with hint bit set, and xmin < XID < xmax, you have
> to check the pg_clog in the new system, when currently you only need to
> do a binary search of the local array in the snapshot. My gut feeling
> is that it won't be significantly slower in practice. If it becomes a
> problem, some rearrangement pg_clog code might help, or you could build
> a cache of XID->CSN mappings that you've alread looked up in
> SnapshotData. So I don't think that's going to be a show-stopper.

Yes definitely it should not be not show-stopper. This can be optimized later 
by method 
as you mentioned  and also by some cut-off technique based on which we can 
decide that a XID beyond a certain range will be always visible, and thereby
avoiding look-up in pg_clog.

Thanks and Regards,
Kumar Rajeev Rastogi


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] gettimeofday is at the end of its usefulness?

[Tom Lane]

Greg Stark  writes:
> I actually think it would be more interesting if we could measure the
> overhead and adjust for it.

Actually, that's quite a good thought.  The overhead should be a pretty
stable number on any given machine, so in theory we could do this to
high precision.  And the numbers I just showed say that on current
x86_64 platforms, the *best we could possibly hope for* in terms of
direct overhead reduction is about 4x.  Which is something, but it
hardly makes the problem vanish.

I have a vague feeling that we discussed subtract-the-overhead once before
and thought it wasn't necessary yet.  Maybe it's time.

But we also need to be using something that gives better than 1usec
resolution.  So I'm still thinking we should use clock_gettime() where
available, and look for alternative APIs where not.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal for CSN based snapshots

[Amit Kapila]

On Tue, May 13, 2014 at 1:59 PM, Heikki Linnakangas
 wrote:
> On 05/13/2014 09:44 AM, Amit Kapila wrote:
>>
>> To accomplish this won't XID-CSN map table be required and how will
>> it be maintained (means when to clear and add a entry to that map table)?
>
>
> Not sure I understand. The clog is a mapping from XID to CSN.

The case I was referring is for xmin < XID < xmax, which you have
mentioned below that you are planing to directly refer pg_clog.  This
is I think one of the main place where new design can have impact
on performance, but as you said it is better to first do the implementation
based on pg_clog rather than directly jumping to optimize by maintaining
XID to CSN mapping.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Typo in release notes

[Sergey Muraviov]

Hi.

Here's a patch that fixes it.

-- 
Best regards,
Sergey Muraviov
diff --git a/doc/src/sgml/release-9.4.sgml b/doc/src/sgml/release-9.4.sgml
index cabfbdd..0b6f383 100644
--- a/doc/src/sgml/release-9.4.sgml
+++ b/doc/src/sgml/release-9.4.sgml
@@ -1587,7 +1587,7 @@

 Add variables plpgsql.extra_warnings
-and plpgsql.extra_errorsto enable additional PL/pgSQL
+and plpgsql.extra_errors to enable additional PL/pgSQL
 warnings and errors (Marko Tiikkaja, Petr Jelinek)

 

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] gettimeofday is at the end of its usefulness?

[Greg Stark]

On Tue, May 13, 2014 at 11:58 PM, Tom Lane  wrote:
> I also tried a loop around a bare "rdtsc" assembly instruction, finding
> that that instruction takes about 10nsec.  That would be a nice
> improvement over gettimeofday, except that using that directly would
> involve dealing with cross-CPU skew, which seems like no fun at all.
> And I don't really want to get into finding equivalents for non-Intel
> architectures, either.

I always assumed the kernel used rdtsc to implement some of the high
performance timers. It can save the current time in a mapped page when
it schedules a process and then in the vdso syscall (ie in user-space)
it can use rdtsc to calculate the offset needed to adjust that
timestamp to the current time. This seems consistent with your
calculations that showed the 40ns overhead with +/- 10ns precision.

I actually think it would be more interesting if we could measure the
overhead and adjust for it. I don't think people are really concerned
with how long EXPLAIN ANALYZE takes to run if they could get accurate
numbers out of it.

Other profiling tools I poked at in the past ran a tight loop around
the profiling code to estimate the time it actually took and then
subtracted that from all the measurements. I think that might work for
the actual clock_gettime overhead. If we did that then we could call
it twice and measure the time spent in the rest of the EXPLAIN ANALYZE
code and subtract that plus the time for the two clock_gettimes from
the run-time...

-- 
greg


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 9.4 beta1 crash on Debian sid/i386

[Tom Lane]

Christoph Berg  writes:
> Building 9.4 beta1 on Debian sid/i386 fails during the regression
> tests. amd64 works fine, as does i386 on the released distributions.

It would appear that something is wrong with check_stack_depth(),
and/or getrlimit(RLIMIT_STACK) is lying to us about the available stack.
None of that logic has changed in awhile.  You might try checking what
the max_stack_depth GUC gets set to by default in each build, and how that
compares to what "ulimit -s" has to say.  If it looks sane, try tracing
through check_stack_depth.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] gettimeofday is at the end of its usefulness?

[Tom Lane]

Peter Geoghegan  writes:
> On Tue, May 13, 2014 at 3:58 PM, Tom Lane  wrote:
>> There's also a CLOCK_REALTIME_COARSE selector, which is noticeably faster
>> --- about 10nsec for me --- but the output appears to only advance once
>> every millisecond, so it's probably useless for our purposes.  The other
>> selectors mentioned in the Linux man page are considerably slower than
>> CLOCK_REALTIME for me, suggesting that they actually call into the kernel.

> What Linux kernel version is in use here?

Ah, sorry, I should have specified.  This is RHEL6.5, current kernel
version 2.6.32-431.17.1.el6.x86_64.

> Apparently, as I think
> you've stated another way, more recent versions have VDSO for this,
> which can make a big difference. This article seems like a sensible
> guide to all of this:
> https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_MRG/1.3/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-General_System_Tuning-gettimeofday_speedup.html

This appears to be talking about RHEL5, which is quite a bit older
(and, I'd guess, trailing edge for anybody who might deploy PG 9.5).
I did confirm that /proc/sys/kernel/vsyscall64 exists and has a default
setting of 1 on RHEL6.  Setting it to 0 causes gettimeofday to take
150ns, which probably represents the time for a trivial kernel call.
The MRG extension described on the linked page doesn't seem to be
implemented in stock RHEL6 (setting vsyscall64 to 2 is allowed but
doesn't change behavior compared to 1).  However, if I'm reading it
right, all that does is make gettimeofday behave like
clock_gettime(CLOCK_REALTIME_COARSE).

> CLOCK_REALTIME_COARSE seemingly influences precision in a way that
> allows user space applications to decide on their precision/cost
> trade-off, rather than being forced to use the system default (that
> procfs surfaces) through gettimeofday():
> http://lwn.net/Articles/342018/

Yeah, I think these are the same implementations exposed to apps in two
different ways, one being a system-wide switch affecting gettimeofday()
while the other allows the app source code to say which one it wants.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] gettimeofday is at the end of its usefulness?

[Peter Geoghegan]

On Tue, May 13, 2014 at 3:58 PM, Tom Lane  wrote:
> There's also a CLOCK_REALTIME_COARSE selector, which is noticeably faster
> --- about 10nsec for me --- but the output appears to only advance once
> every millisecond, so it's probably useless for our purposes.  The other
> selectors mentioned in the Linux man page are considerably slower than
> CLOCK_REALTIME for me, suggesting that they actually call into the kernel.

What Linux kernel version is in use here? Apparently, as I think
you've stated another way, more recent versions have VDSO for this,
which can make a big difference. This article seems like a sensible
guide to all of this:
https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_MRG/1.3/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-General_System_Tuning-gettimeofday_speedup.html

CLOCK_REALTIME_COARSE seemingly influences precision in a way that
allows user space applications to decide on their precision/cost
trade-off, rather than being forced to use the system default (that
procfs surfaces) through gettimeofday():
http://lwn.net/Articles/342018/

I can see a benefit in exposing this trade-off to Postgres code
directly. I still think that a correlated reference period will prove
useful, and while there are a number of ways to amortize the cost of
repeatedly (coarsely) getting the wall time in the ordinary course of
choosing victim buffers, it would be nice to do this too.
-- 
Peter Geoghegan


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

[Craig Ringer]

On 05/14/2014 07:06 AM, Thomas Munro wrote:
> Hi
> 
> A couple of years ago I posted an outline of a plan [1] and an
> initial patch [2] for implementing SKIP LOCKED DATA.  I have
> recently come back to this idea, rebased the patch and added a
> simple isolation test -- please see attached.

Simon did some similar work a while ago, so I've cc'd him for his input.



-- 
 Craig Ringer   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] gettimeofday is at the end of its usefulness?

[Tom Lane]

A recent question from Tim Kane prompted me to measure the overhead
costs of EXPLAIN ANALYZE, which I'd not checked in awhile.  Things
are far worse than I thought.  On my current server (by no means
lavish hardware: Xeon E5-2609 @2.40GHz) a simple seqscan can run
at something like 110 nsec per row:

regression=# create table foo as select x as f1 from generate_series(1,100) 
x;
SELECT 100
regression=# vacuum foo;
VACUUM
regression=# explain analyze select * from foo;
  QUERY PLAN
---
 Seq Scan on foo  (cost=0.00..14425.00 rows=100 width=4) (actual 
time=0.053..111.720 rows=100 loops=1)
 Planning time: 0.222 ms
 Execution time: 166.682 ms
(3 rows)

(and, btw, this is a debug build --- without assert and memory
context checks it'd be faster.)

The problem with this number is that a simple test program shows that
gettimeofday() requires about 40 nsec on this hardware.  That means
that two-thirds of the above timing measurement is overhead.

To add insult to injury, gettimeofday's output cannot be more precise than
1 microsecond, making its use for measuring sub-microsecond intervals at
best stochastic.

I looked around a bit and found that recent versions of POSIX have a
function clock_gettime() that's a bit more modern than gettimeofday():
at least, the output struct provides nsec rather than usec available
precision.  I benchmarked this, with the CLOCK_REALTIME selector, and
found that it also requires about 40nsec, while the output is actually
good to perhaps 10nsec precision.  (I base this on seeing no duplicate
readings in a tight loop, so that the value is certainly getting advanced
more often than once every 40 nsec.)

There's also a CLOCK_REALTIME_COARSE selector, which is noticeably faster
--- about 10nsec for me --- but the output appears to only advance once
every millisecond, so it's probably useless for our purposes.  The other
selectors mentioned in the Linux man page are considerably slower than
CLOCK_REALTIME for me, suggesting that they actually call into the kernel.

I also tried a loop around a bare "rdtsc" assembly instruction, finding
that that instruction takes about 10nsec.  That would be a nice
improvement over gettimeofday, except that using that directly would
involve dealing with cross-CPU skew, which seems like no fun at all.
And I don't really want to get into finding equivalents for non-Intel
architectures, either.

Anyway it looks like clock_gettime() might be worth using on Linux
just for the more precise output.  It doesn't seem to exist on OS X
though, and I have no idea about elsewhere.

I'm curious if anybody has ideas about other things we might do for
portable high-precision timing.

regards, tom lane

#include 
#include 
#include 

int main(int argc, char **argv)
{
	long thistime, lasttime = 0;
	int nbogus = 0;
	int ndistinct = 0;
	int i;

	for (i=0; i<10; i++)
	{
#if 1
		struct timespec tv;
		clock_gettime(CLOCK_REALTIME, &tv);
		thistime = (long) tv.tv_sec * 10 + tv.tv_nsec;
#else
		struct timeval tv;
		gettimeofday(&tv, NULL);
		thistime = (long) tv.tv_sec * 100 + tv.tv_usec;
#endif

		if (thistime < lasttime)
			nbogus++;
		else if (thistime > lasttime)
			ndistinct++;
		lasttime = thistime;
	}

	printf("%d bogus readings\n", nbogus);
	printf("%d distinct readings\n", ndistinct);

	return 0;
}

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Error in running DBT2

[Peter Geoghegan]

On Tue, May 13, 2014 at 2:51 PM, David G Johnston
 wrote:
> The error is more typical of someone creating a database with UTF-8 encoding
> but then tries storing Latin-1 (or some other) encoded data.

I didn't say that it was the only problem. Just that I had noticed
that other people were using standard_conforming_strings = off when
running DBT2, on the couple of occasions were I looked through a
result set. The fact that this may have accidentally worked before,
when that was the default does not imply that it was ever correct, but
it may at least be possible to get it to work at least as well as it
ever did by doing that. DBT2 doesn't seem to have anything to say
about encoding one way or the other. As I said, it has many problems.
Another thing that I noticed that I didn't like was that it uses the C
locale.

-- 
Peter Geoghegan


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Error in running DBT2

[David G Johnston]

Rohit Goyal wrote
> Hi Peter,
> 
> On Tue, May 13, 2014 at 9:44 PM, Peter Geoghegan <

> pg@

> > wrote:
> 
>>
>> On Tue, May 13, 2014 at 12:36 PM, Rohit Goyal <

> rhtgyl.87@

> > wrote:
>>
>>> This pattern the above found many times. Please guide me through!!!
>>>
>>
>> IIRC, people have been working around this by setting
>> standard_conforming_strings to "off". It really ought to be fixed in a
>> principled way, though -- the real issue here is that dbt2 has severe
>> bit-rot.
>>
>> Can you please elaborate, where I can make this changes :) ?

Peter?

The error is more typical of someone creating a database with UTF-8 encoding
but then tries storing Latin-1 (or some other) encoded data.  Since not all
byte sequences in the source data encoding are valid in UTF-8 when one of
the invalid sequences is present the import fails.  PostgreSQL makes no
attempt to perform data conversion on import.

The solution is to create the database with the correct encoding - whatever
it may be.  This "DBT2" application should provide guidance on this topic. 
Otherwise you could use "SQL_ASCII" - which is effectively punting on the
issue:

http://www.postgresql.org/docs/9.3/static/sql-createdatabase.html
http://www.postgresql.org/docs/9.3/static/multibyte.html#MULTIBYTE-CHARSET-SUPPORTED

IIRC given that the error happens in a SQL COPY
"standard_conforming_strings" has no bearing on the outcome.





--
View this message in context: 
http://postgresql.1045698.n5.nabble.com/Error-in-running-DBT2-tp5803793p5803817.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] buildfarm / handling (undefined) locales

[Tom Lane]

Andrew Dunstan  writes:
> On 05/13/2014 04:14 PM, Tom Lane wrote:
>> But independently of whether it's a fatal error or not: when there's
>> no relevant command-line argument then we print the
>> 
>> invalid locale name ""
>> 
>> message which is surely pretty unhelpful.  It'd be better if we could
>> finger the incorrect environment setting.  Unfortunately, we don't know
>> for sure which environment variable(s) setlocale was looking at --- I
>> believe it's somewhat platform specific.  We could probably print
>> something like this instead:
>> 
>> environment locale settings are invalid
>> 
>> Thoughts?

> I'd also be tempted to add the settings for LC_ALL and LANG and note 
> that they are possible sources of the problem, or maybe only do that if 
> they match the locale being rejected.

Well, all we know is that we asked setlocale for the default locale from
the environment and it failed.  We don't really know which variable(s)
it looked at.  I think printing specific variables could easily be as
misleading as it is helpful; and given the lack of prior complaints,
I'm doubtful that it's worth going to the effort of trying to print a
platform-specific collection of variables.

Attached is a draft patch that behaves like this:

$ initdb --locale=foo
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

initdb: invalid locale name "foo"

$ LANG=foo initdb
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

initdb: environment locale settings are invalid

Most of the bulk of the patch is getting rid of the no-longer-useful
boolean result of check_locale_name.

regards, tom lane

diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 2a51916..3a9dc53 100644
*** a/src/bin/initdb/initdb.c
--- b/src/bin/initdb/initdb.c
*** static void trapsig(int signum);
*** 252,258 
  static void check_ok(void);
  static char *escape_quotes(const char *src);
  static int	locale_date_order(const char *locale);
! static bool check_locale_name(int category, const char *locale,
    char **canonname);
  static bool check_locale_encoding(const char *locale, int encoding);
  static void setlocales(void);
--- 252,258 
  static void check_ok(void);
  static char *escape_quotes(const char *src);
  static int	locale_date_order(const char *locale);
! static void check_locale_name(int category, const char *locale,
    char **canonname);
  static bool check_locale_encoding(const char *locale, int encoding);
  static void setlocales(void);
*** locale_date_order(const char *locale)
*** 2529,2535 
  }
  
  /*
!  * Is the locale name valid for the locale category?
   *
   * If successful, and canonname isn't NULL, a malloc'd copy of the locale's
   * canonical name is stored there.  This is especially useful for figuring out
--- 2529,2535 
  }
  
  /*
!  * Verify that locale name is valid for the locale category.
   *
   * If successful, and canonname isn't NULL, a malloc'd copy of the locale's
   * canonical name is stored there.  This is especially useful for figuring out
*** locale_date_order(const char *locale)
*** 2540,2546 
   *
   * this should match the backend's check_locale() function
   */
! static bool
  check_locale_name(int category, const char *locale, char **canonname)
  {
  	char	   *save;
--- 2540,2546 
   *
   * this should match the backend's check_locale() function
   */
! static void
  check_locale_name(int category, const char *locale, char **canonname)
  {
  	char	   *save;
*** check_locale_name(int category, const ch
*** 2551,2557 
  
  	save = setlocale(category, NULL);
  	if (!save)
! 		return false;			/* won't happen, we hope */
  
  	/* save may be pointing at a modifiable scratch variable, so copy it. */
  	save = pg_strdup(save);
--- 2551,2561 
  
  	save = setlocale(category, NULL);
  	if (!save)
! 	{
! 		fprintf(stderr, _("%s: setlocale failed\n"),
! progname);
! 		exit(1);
! 	}
  
  	/* save may be pointing at a modifiable scratch variable, so copy it. */
  	save = pg_strdup(save);
*** check_locale_name(int category, const ch
*** 2565,2580 
  
  	/* restore old value. */
  	if (!setlocale(category, save))
  		fprintf(stderr, _("%s: failed to restore old locale \"%s\"\n"),
  progname, save);
  	free(save);
  
! 	/* should we exit here? */
  	if (res == NULL)
! 		fprintf(stderr, _("%s: invalid locale name \"%s\"\n"),
! progname, locale);
! 
! 	return (res != NULL);
  }
  
  /*
--- 2569,2602 
  
  	/* restore old value. */
  	if (!setlocale(category, save))
+ 	{
  		fprintf(stderr, _("%s: failed to restore old locale \"%s\"\n"),
  progname, save);
+ 		exit(1);
+ 	}
  	free(save);
  
! 	/* complain if locale wasn't valid */
  	if (res == NULL)
! 	{
! 		if (*locale)
! 			fprintf(stderr, _("%s: invalid locale name \"%s

[HACKERS] 9.4 beta1 crash on Debian sid/i386

[Christoph Berg]

Building 9.4 beta1 on Debian sid/i386 fails during the regression
tests. amd64 works fine, as does i386 on the released distributions.

parallel group (11 tests):  create_cast create_aggregate drop_if_exists 
typed_table create_function_3 vacuum constraints create_table_like triggers 
inherit updatable_views
 create_aggregate ... ok
 create_function_3... ok
 create_cast  ... ok
 constraints  ... ok
 triggers ... ok
 inherit  ... ok
 create_table_like... ok
 typed_table  ... ok
 vacuum   ... ok
 drop_if_exists   ... ok
 updatable_views  ... ok
test sanity_check ... ok
test errors   ... FAILED (test process exited with exit code 2)
test select   ... FAILED (test process exited with exit code 2)
[...]

LOG:  server process (PID 8403) was terminated by signal 7: Bus error
DETAIL:  Failed process was running: select infinite_recurse();
LOG:  terminating any other active server processes

*** 
/home/cbe/projects/postgresql/9.4/trunk/build/../src/test/regress/expected/errors.out
   2014-05-11 23:16:48.0 +0200
--- 
/srv/projects/postgresql/9.4/trunk/build/src/test/regress/results/errors.out
2014-05-13 22:16:05.337798163 +0200
***
*** 444,447 
  'select infinite_recurse()' language sql;
  \set VERBOSITY terse
  select infinite_recurse();
! ERROR:  stack depth limit exceeded
--- 444,447 
  'select infinite_recurse()' language sql;
  \set VERBOSITY terse
  select infinite_recurse();
! connection to server was lost

Reading symbols from 
/srv/projects/postgresql/9.4/trunk/build/src/backend/postgres...done.
[New LWP 8403]

warning: Could not load shared library symbols for linux-gate.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Core was generated by `postgres: cbe regression [local] SELECT  
 '.
Program terminated with signal 7, Bus error.
#0  0xf72abbc3 in ScanKeywordLookup (text=0xed2ec330 "select", 
keywords=0xf773ca60 , 
num_keywords=405)
at 
/home/cbe/projects/postgresql/9.4/trunk/build/../src/backend/parser/kwlookup.c:42
42  {
(gdb) bt
#0  0xf72abbc3 in ScanKeywordLookup (text=0xed2ec330 "select", 
keywords=0xf773ca60 , 
num_keywords=405)
at 
/home/cbe/projects/postgresql/9.4/trunk/build/../src/backend/parser/kwlookup.c:42
#1  0xf72aa41f in core_yylex (yylval_param=yylval_param@entry=0xffdcb104, 
yylloc_param=yylloc_param@entry=0xffdcb108, 
yyscanner=yyscanner@entry=0xed2ec2a8) at scan.l:950
#2  0xf72abe30 in base_yylex (lvalp=lvalp@entry=0xffdcb104, 
llocp=llocp@entry=0xffdcb108, 
yyscanner=yyscanner@entry=0xed2ec2a8)
at 
/home/cbe/projects/postgresql/9.4/trunk/build/../src/backend/parser/parser.c:99
#3  0xf72949c8 in base_yyparse (yyscanner=yyscanner@entry=0xed2ec2a8) at 
gram.c:21442
#4  0xf72abd02 in raw_parser (str=str@entry=0xed2ec1e8 "select 
infinite_recurse()")
at 
/home/cbe/projects/postgresql/9.4/trunk/build/../src/backend/parser/parser.c:52
#5  0xf7441ac7 in pg_parse_query (
query_string=query_string@entry=0xed2ec1e8 "select infinite_recurse()")
at 
/home/cbe/projects/postgresql/9.4/trunk/build/../src/backend/tcop/postgres.c:563
#6  0xf73c77a8 in inline_function (context=0xffdcbc1c, func_tuple=0xedad9b98, 
funcvariadic=, args=0x0, input_collid=0, result_collid=0, 
result_type=23, 
funcid=29299)
at 
/home/cbe/projects/postgresql/9.4/trunk/build/../src/backend/optimizer/util/clauses.c:4268
#7  simplify_function (funcid=29299, result_type=23, result_typmod=-1, 
result_collid=result_collid@entry=0, input_collid=input_collid@entry=0, 
args_p=args_p@entry=0xffdcba9c, funcvariadic=funcvariadic@entry=0 '\000', 
process_args=process_args@entry=1 '\001', 
allow_non_const=allow_non_const@entry=1 '\001', 
context=context@entry=0xffdcbc1c)
at 
/home/cbe/projects/postgresql/9.4/trunk/build/../src/backend/optimizer/util/clauses.c:3796
[...]
... help bt ... ah I can also ask it to print the outermost stack frames 
separately...
(gdb) bt -10
#35639 0xf733927c in ExecutorRun (queryDesc=queryDesc@entry=0xf945d188, 
direction=direction@entry=ForwardScanDirection, count=count@entry=0)
at 
/home/cbe/projects/postgresql/9.4/trunk/build/../src/backend/executor/execMain.c:256
#35640 0xf7445463 in PortalRunSelect (portal=portal@entry=0xf945b180, 
forward=forward@entry=1 '\001', 
count=0, count@entry=2147483647, dest=dest@entry=0xf94304c8)
at 
/home/cbe/projects/postgresql/9.4/trunk/build/../src/backend/tcop/pquery.c:946
#35641 0xf744694e in PortalRun (portal=portal@entry=0xf945b180, 
count=count@entry=2147483647, 
isTopLevel=isTopLevel@entry=1 '\001', dest=dest@entry=0xf94304c8, 
altdest=altdest@entry=0xf94304c8, 

Re: [HACKERS] buildfarm / handling (undefined) locales

[Andrew Dunstan]

On 05/13/2014 04:14 PM, Tom Lane wrote:

Heikki Linnakangas  writes:

On 05/13/2014 09:58 PM, Tom Lane wrote:

... If so the issue is presumably
that the environment variable(s) were set to incorrect values.  While
we *could* abort in that situation, I've never heard of any program
that did; the normal response is to silently ignore the environment
variables and use C locale.  We're not being exactly silent about it
but I think the outcome is the expected one.

Initdb isn't like most programs. The locale given to initdb is memorized
in the data directory, and if you later notice that it was wrong, you'll
have to dump and reload. There is a strong argument for initdb to be
more strict than, say, your average text editor.

Hm, well, if that's the behavior we want then it's certainly an easy
change.

But independently of whether it's a fatal error or not: when there's
no relevant command-line argument then we print the

invalid locale name ""

message which is surely pretty unhelpful.  It'd be better if we could
finger the incorrect environment setting.  Unfortunately, we don't know
for sure which environment variable(s) setlocale was looking at --- I
believe it's somewhat platform specific.  We could probably print
something like this instead:

environment locale settings are invalid

Thoughts?






I'd also be tempted to add the settings for LC_ALL and LANG and note 
that they are possible sources of the problem, or maybe only do that if 
they match the locale being rejected.


cheers

andrew


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] buildfarm / handling (undefined) locales

[Christoph Berg]

Re: Tom Lane 2014-05-13 <27525.1400012...@sss.pgh.pa.us>
> Heikki Linnakangas  writes:
> > On 05/13/2014 09:58 PM, Tom Lane wrote:
> >> ... If so the issue is presumably
> >> that the environment variable(s) were set to incorrect values.  While
> >> we *could* abort in that situation, I've never heard of any program
> >> that did; the normal response is to silently ignore the environment
> >> variables and use C locale.  We're not being exactly silent about it
> >> but I think the outcome is the expected one.
> 
> > Initdb isn't like most programs. The locale given to initdb is memorized 
> > in the data directory, and if you later notice that it was wrong, you'll 
> > have to dump and reload. There is a strong argument for initdb to be 
> > more strict than, say, your average text editor.
> 
> Hm, well, if that's the behavior we want then it's certainly an easy
> change.

It should definitely fail. If you have some LC_ variables set, you
want to store that charset in your database. If the DB ends up using
C, that's not helpful. (Or probably even worse, as SQL_ASCII will
accept binary garbage without checking anything, so you'll only notice
when it's too late.)

Bad locales are the #1 reason for initdb problems at install time for
Debian packages - while pg_createcluster catches some of these itself
before invoking initdb, making the process more deterministic would be
a good thing.

> But independently of whether it's a fatal error or not: when there's
> no relevant command-line argument then we print the
> 
>   invalid locale name ""
> 
> message which is surely pretty unhelpful.  It'd be better if we could
> finger the incorrect environment setting.  Unfortunately, we don't know
> for sure which environment variable(s) setlocale was looking at --- I
> believe it's somewhat platform specific.  We could probably print
> something like this instead:
> 
>   environment locale settings are invalid

Definitely a good plan. The current behavior is just not helpful:

$ LANG=de_DE.utf-9 /usr/lib/postgresql/9.4/bin/initdb -D /tmp/bar
The files belonging to this database system will be owned by user "cbe".
This user must also own the server process.

initdb: invalid locale name ""
initdb: invalid locale name ""
initdb: invalid locale name ""
initdb: invalid locale name ""
initdb: invalid locale name ""
initdb: invalid locale name ""
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".

Christoph
-- 
c...@df7cb.de | http://www.df7cb.de/


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Error in running DBT2

[Rohit Goyal]

Hi Peter,

On Tue, May 13, 2014 at 9:44 PM, Peter Geoghegan  wrote:

>
> On Tue, May 13, 2014 at 12:36 PM, Rohit Goyal  wrote:
>
>> This pattern the above found many times. Please guide me through!!!
>>
>
> IIRC, people have been working around this by setting
> standard_conforming_strings to "off". It really ought to be fixed in a
> principled way, though -- the real issue here is that dbt2 has severe
> bit-rot.
>
> Can you please elaborate, where I can make this changes :) ?

Regards,
Rohit Goyal

>
>

Re: [HACKERS] buildfarm / handling (undefined) locales

[Andrew Dunstan]

On 05/13/2014 03:40 PM, Tom Lane wrote:

Tomas Vondra  writes:

On 13.5.2014 21:03, Andrew Dunstan wrote:

On 05/13/2014 02:39 PM, Tomas Vondra wrote:

However in the lastrun-logs I see 81 files, and upon closer inspection
it seems that pretty much logs for all custom locales (cs_CZ.*, sk_SK.*)
are missing:

Clearly this is from a different run, since the link above shows the run
failing at the contrib-install-check-en_US stage, and so of course all
those other locales would not have been checked in that run.

Um, no, I think this (the last run in 9.3 @ magpie):

Yeah, Tomas is right: there's something wrong with the uploading of non-C
locale test logs as of the 4.12 buildfarm scripts.  Here's what the
buildfarm shows for dromedary's last run:

http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=dromedary&dt=2014-05-13%2019%3A06%3A12

You can see that it's configured to do C and en_US.UTF-8 locales.  A look
into the lastrun-logs directory shows that it did:

$ ls
SCM-checkout.logpl-install-check-en_US.UTF-8
check-pg_upgrade.logrunlogs.tgz
check.log   startdb-C-1.log
config.log  startdb-C-2.log
configure.log   startdb-C-3.log
contrib-install-check-C.log startdb-C-4.log
contrib-install-check-en_US.UTF-8   startdb-en_US.UTF-8-1
ecpg-check.log  startdb-en_US.UTF-8-2
githead.log startdb-en_US.UTF-8-3
initdb-C.logstopdb-C-1.log
initdb-en_US.UTF-8  stopdb-C-2.log
install-check-C.log stopdb-C-3.log
install-check-en_US.UTF-8   stopdb-C-4.log
install-contrib.log stopdb-en_US.UTF-8-1
isolation-check.log stopdb-en_US.UTF-8-2
make-contrib.logstopdb-en_US.UTF-8-3
make-install.logtest-decoding-check.log
make.logtypedefs.log
pl-install-check-C.log  web-txn.data

but not all those files got put into runlogs.tgz:

$ tar tvfz runlogs.tgz
-rw-r--r--  0 buildfarm staff  40 May 13 15:06 githead.log
-rw-r--r--  0 buildfarm staff 520 May 13 15:06 SCM-checkout.log
-rw-r--r--  0 buildfarm staff  319642 May 13 15:07 config.log
-rw-r--r--  0 buildfarm staff   16060 May 13 15:07 configure.log
-rw-r--r--  0 buildfarm staff  293997 May 13 15:12 make.log
-rw-r--r--  0 buildfarm staff 2757773 May 13 15:13 check.log
-rw-r--r--  0 buildfarm staff   60462 May 13 15:13 make-contrib.log
-rw-r--r--  0 buildfarm staff   39763 May 13 15:13 make-install.log
-rw-r--r--  0 buildfarm staff   29673 May 13 15:13 install-contrib.log
-rw-r--r--  0 buildfarm staff   87376 May 13 15:15 check-pg_upgrade.log
-rw-r--r--  0 buildfarm staff  181571 May 13 15:16 test-decoding-check.log
-rw-r--r--  0 buildfarm staff1455 May 13 15:16 initdb-C.log
-rw-r--r--  0 buildfarm staff 540 May 13 15:17 startdb-C-1.log
-rw-r--r--  0 buildfarm staff 2692169 May 13 15:17 install-check-C.log
-rw-r--r--  0 buildfarm staff 297 May 13 15:17 stopdb-C-1.log
-rw-r--r--  0 buildfarm staff 540 May 13 15:18 startdb-C-2.log
-rw-r--r--  0 buildfarm staff 1678033 May 13 15:19 isolation-check.log
-rw-r--r--  0 buildfarm staff 296 May 13 15:19 stopdb-C-2.log
-rw-r--r--  0 buildfarm staff 540 May 13 15:19 startdb-C-3.log
-rw-r--r--  0 buildfarm staff  204501 May 13 15:19 pl-install-check-C.log
-rw-r--r--  0 buildfarm staff 296 May 13 15:19 stopdb-C-3.log
-rw-r--r--  0 buildfarm staff 540 May 13 15:19 startdb-C-4.log
-rw-r--r--  0 buildfarm staff  886518 May 13 15:20 contrib-install-check-C.log
-rw-r--r--  0 buildfarm staff 297 May 13 15:20 stopdb-C-4.log
-rw-r--r--  0 buildfarm staff  132144 May 13 15:23 ecpg-check.log
-rw-r--r--  0 buildfarm staff   35834 May 13 15:25 typedefs.log

which doubtless explains why they're not available from the buildfarm
server.






There's a recently introduced bug, which I'm at a bit of a loss to 
explain. I have pushed a fix for it. 



I will get out a bug fix release shortly.

cheers

andrew






--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] buildfarm / handling (undefined) locales

[Tom Lane]

Heikki Linnakangas  writes:
> On 05/13/2014 09:58 PM, Tom Lane wrote:
>> ... If so the issue is presumably
>> that the environment variable(s) were set to incorrect values.  While
>> we *could* abort in that situation, I've never heard of any program
>> that did; the normal response is to silently ignore the environment
>> variables and use C locale.  We're not being exactly silent about it
>> but I think the outcome is the expected one.

> Initdb isn't like most programs. The locale given to initdb is memorized 
> in the data directory, and if you later notice that it was wrong, you'll 
> have to dump and reload. There is a strong argument for initdb to be 
> more strict than, say, your average text editor.

Hm, well, if that's the behavior we want then it's certainly an easy
change.

But independently of whether it's a fatal error or not: when there's
no relevant command-line argument then we print the

invalid locale name ""

message which is surely pretty unhelpful.  It'd be better if we could
finger the incorrect environment setting.  Unfortunately, we don't know
for sure which environment variable(s) setlocale was looking at --- I
believe it's somewhat platform specific.  We could probably print
something like this instead:

environment locale settings are invalid

Thoughts?

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] buildfarm / handling (undefined) locales

[Tomas Vondra]

On 13.5.2014 20:58, Tom Lane wrote:
> Tomas Vondra  writes:
>> a few days ago I switched magpie into an LXC container, and while
>> fixinig unrelated issue there, I noticed that although the tests seemed
>> to run, some of the results are actually rubbish because of missing locales.
> 
>> So for example when the system misses cs_CZ.WIN-1250 (i.e. czech locale
>> with windows codepage 1250), initdb actually did this
> 
>> The files belonging to this database system will be owned by user
>> "pgbuild".
>> This user must also own the server process.
> 
>> initdb: invalid locale name ""
>> initdb: invalid locale name ""
>> initdb: invalid locale name ""
>> initdb: invalid locale name ""
>> initdb: invalid locale name ""
>> initdb: invalid locale name ""
>> The database cluster will be initialized with locale "C".
>> The default database encoding has accordingly been set to
>> "SQL_ASCII".
>> The default text search configuration will be set to "english".
> 
> Hm, I'm a bit confused as to what you actually did here.  The "invalid
> locale name" bleats seem to indicate that no --locale or --lc_xxx options
> were given on the command line; correct?  If so the issue is presumably
> that the environment variable(s) were set to incorrect values.  While
> we *could* abort in that situation, I've never heard of any program
> that did; the normal response is to silently ignore the environment
> variables and use C locale.  We're not being exactly silent about it
> but I think the outcome is the expected one.
> 
> There is a comment in the code about this:
> 
> /* should we exit here? */
> if (res == NULL)
> fprintf(stderr, _("%s: invalid locale name \"%s\"\n"),
> progname, locale);
> 
> but I think what's being questioned is whether an incorrect locale
> name *given as a command line argument* should result in an abort.
> That might be a good idea, but it looks like it'd require some
> restructuring of the code to make it possible to distinguish the
> case from bad-environment.

H. Actually the logs above were generated "manually" by setting LANG
to an incorrect value on my desktop machine, i.e. something like this:

   export LANG="cs_CZ.WIN-1250"
   pg_ctl -D /tmp/test init

I did this because the logs from the buildfarm animal contain the
messages in czech, which makes them rather unsuitable for english speakers:


http://www.pgbuildfarm.org/cgi-bin/show_stage_log.pl?nm=magpie&dt=2014-04-17%2000%3A07%3A36&stg=initdb-cs_CZ.WIN-1250

I suppose that's why the locale name is empty (it simply gets empty
string from the environment), while the buildfarm client uses a
command-line option to pass the name.

Also, I now noticed that while the example I posted has to C.SQL_ASCII,
the buildfarm then uses cs_CZ.UTF-8. I guess this possible thanks to the
'--locale' option.

If it was up to me, I'd vote to fail in such cases. I find it confusing
(and a bit 'automagic') to receive invalid locale but decide 'the user
surely meant C with SQL_ASCII'. I've actually had to deal with a few
production installations that were installed like this, and it was PITA
to fix.

Anyway, that's not what I was proposing here. I merely think we should
do a simple 'did we actually get the right locale' check somewhere in
the buildfarm client.

>> Yeah, not really what we were shooting for. I've fixed this by defining
>> the missing locales, and indeed - magpie now fails in plpython tests.
> 
> I saw that earlier today (tho right now the buildfarm server seems to
> not be responding :-().  Probably we should use some more-widely-used
> character code in that specific test?

What do you mean by 'not responding'? The tests were not running for ~2
days because of LXC/SELinux issue, causing cron failures.

Anyway, using a more-widely used character is probably the best fix
possible here.

regards
Tomas


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] buildfarm / handling (undefined) locales

[Heikki Linnakangas]

On 05/13/2014 09:58 PM, Tom Lane wrote:

Tomas Vondra  writes:

a few days ago I switched magpie into an LXC container, and while
fixinig unrelated issue there, I noticed that although the tests seemed
to run, some of the results are actually rubbish because of missing locales.



So for example when the system misses cs_CZ.WIN-1250 (i.e. czech locale
with windows codepage 1250), initdb actually did this



 The files belonging to this database system will be owned by user
 "pgbuild".
 This user must also own the server process.



 initdb: invalid locale name ""
 initdb: invalid locale name ""
 initdb: invalid locale name ""
 initdb: invalid locale name ""
 initdb: invalid locale name ""
 initdb: invalid locale name ""
 The database cluster will be initialized with locale "C".
 The default database encoding has accordingly been set to
 "SQL_ASCII".
 The default text search configuration will be set to "english".


Hm, I'm a bit confused as to what you actually did here.  The "invalid
locale name" bleats seem to indicate that no --locale or --lc_xxx options
were given on the command line; correct?  If so the issue is presumably
that the environment variable(s) were set to incorrect values.  While
we *could* abort in that situation, I've never heard of any program
that did; the normal response is to silently ignore the environment
variables and use C locale.  We're not being exactly silent about it
but I think the outcome is the expected one.


Initdb isn't like most programs. The locale given to initdb is memorized 
in the data directory, and if you later notice that it was wrong, you'll 
have to dump and reload. There is a strong argument for initdb to be 
more strict than, say, your average text editor.



There is a comment in the code about this:

 /* should we exit here? */
 if (res == NULL)
 fprintf(stderr, _("%s: invalid locale name \"%s\"\n"),
 progname, locale);

but I think what's being questioned is whether an incorrect locale
name *given as a command line argument* should result in an abort.
That might be a good idea, but it looks like it'd require some
restructuring of the code to make it possible to distinguish the
case from bad-environment.


Yeah, I think we should definitely complain and exit if the locale given 
on the command-line is invalid.


- Heikki


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 9.5: UPDATE/DELETE .. ORDER BY .. LIMIT ..

[Rukh Meski]

On Sun, May 11, 2014 at 4:47 PM, Tom Lane  wrote:
> The $64 question is whether we'd accept an implementation that fails
> if the target table has children (ie, is partitioned).  That seems
> to me to not be up to the project's usual quality expectations, but
> maybe if there's enough demand for a partial solution we should do so.
>
> It strikes me that a big part of the problem here is that the current
> support for this case assumes that the children don't all have the
> same rowtype.  Which is important if you think of "child table" as
> meaning "subclass in the OO sense".  But for ordinary partitioning
> cases it's just useless complexity, and ModifyTable isn't the only
> thing that suffers from that useless complexity.
>
> If we had a notion of "partitioned table" that involved a restriction
> that all the child tables have the exact same rowtype, we could implement
> UPDATE/DELETE in a much saner fashion --- just one plan tree, not one
> per child table --- and it would be possible to support UPDATE/DELETE
> ORDER BY LIMIT with no more work than for the single-table case.
> So that might shift the calculation as to whether we're willing to
> accept a partial implementation.

None of the use cases I have in mind would ever (have to) use this on
a parent table; in the worst case it might make sense to do it on the
child tables individually.  Personally, I think that just refusing to
operate on tables with children is a reasonable start.  I have no
interest in working on improving partitioning, but I don't think
pushing this feature back in the hopes that someone else will would
help anyone.

But definitely only my two cents on this issue.


♜


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Error in running DBT2

[Peter Geoghegan]

On Tue, May 13, 2014 at 12:36 PM, Rohit Goyal  wrote:

> This pattern the above found many times. Please guide me through!!!
>

IIRC, people have been working around this by setting
standard_conforming_strings to "off". It really ought to be fixed in a
principled way, though -- the real issue here is that dbt2 has severe
bit-rot.

-- 
Peter Geoghegan

Re: [HACKERS] buildfarm / handling (undefined) locales

[Tom Lane]

Tomas Vondra  writes:
> On 13.5.2014 21:03, Andrew Dunstan wrote:
>> On 05/13/2014 02:39 PM, Tomas Vondra wrote:
>>> However in the lastrun-logs I see 81 files, and upon closer inspection
>>> it seems that pretty much logs for all custom locales (cs_CZ.*, sk_SK.*)
>>> are missing:

>> Clearly this is from a different run, since the link above shows the run
>> failing at the contrib-install-check-en_US stage, and so of course all
>> those other locales would not have been checked in that run.

> Um, no, I think this (the last run in 9.3 @ magpie):

Yeah, Tomas is right: there's something wrong with the uploading of non-C
locale test logs as of the 4.12 buildfarm scripts.  Here's what the
buildfarm shows for dromedary's last run:

http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=dromedary&dt=2014-05-13%2019%3A06%3A12

You can see that it's configured to do C and en_US.UTF-8 locales.  A look
into the lastrun-logs directory shows that it did:

$ ls
SCM-checkout.logpl-install-check-en_US.UTF-8
check-pg_upgrade.logrunlogs.tgz
check.log   startdb-C-1.log
config.log  startdb-C-2.log
configure.log   startdb-C-3.log
contrib-install-check-C.log startdb-C-4.log
contrib-install-check-en_US.UTF-8   startdb-en_US.UTF-8-1
ecpg-check.log  startdb-en_US.UTF-8-2
githead.log startdb-en_US.UTF-8-3
initdb-C.logstopdb-C-1.log
initdb-en_US.UTF-8  stopdb-C-2.log
install-check-C.log stopdb-C-3.log
install-check-en_US.UTF-8   stopdb-C-4.log
install-contrib.log stopdb-en_US.UTF-8-1
isolation-check.log stopdb-en_US.UTF-8-2
make-contrib.logstopdb-en_US.UTF-8-3
make-install.logtest-decoding-check.log
make.logtypedefs.log
pl-install-check-C.log  web-txn.data

but not all those files got put into runlogs.tgz:

$ tar tvfz runlogs.tgz
-rw-r--r--  0 buildfarm staff  40 May 13 15:06 githead.log
-rw-r--r--  0 buildfarm staff 520 May 13 15:06 SCM-checkout.log
-rw-r--r--  0 buildfarm staff  319642 May 13 15:07 config.log
-rw-r--r--  0 buildfarm staff   16060 May 13 15:07 configure.log
-rw-r--r--  0 buildfarm staff  293997 May 13 15:12 make.log
-rw-r--r--  0 buildfarm staff 2757773 May 13 15:13 check.log
-rw-r--r--  0 buildfarm staff   60462 May 13 15:13 make-contrib.log
-rw-r--r--  0 buildfarm staff   39763 May 13 15:13 make-install.log
-rw-r--r--  0 buildfarm staff   29673 May 13 15:13 install-contrib.log
-rw-r--r--  0 buildfarm staff   87376 May 13 15:15 check-pg_upgrade.log
-rw-r--r--  0 buildfarm staff  181571 May 13 15:16 test-decoding-check.log
-rw-r--r--  0 buildfarm staff1455 May 13 15:16 initdb-C.log
-rw-r--r--  0 buildfarm staff 540 May 13 15:17 startdb-C-1.log
-rw-r--r--  0 buildfarm staff 2692169 May 13 15:17 install-check-C.log
-rw-r--r--  0 buildfarm staff 297 May 13 15:17 stopdb-C-1.log
-rw-r--r--  0 buildfarm staff 540 May 13 15:18 startdb-C-2.log
-rw-r--r--  0 buildfarm staff 1678033 May 13 15:19 isolation-check.log
-rw-r--r--  0 buildfarm staff 296 May 13 15:19 stopdb-C-2.log
-rw-r--r--  0 buildfarm staff 540 May 13 15:19 startdb-C-3.log
-rw-r--r--  0 buildfarm staff  204501 May 13 15:19 pl-install-check-C.log
-rw-r--r--  0 buildfarm staff 296 May 13 15:19 stopdb-C-3.log
-rw-r--r--  0 buildfarm staff 540 May 13 15:19 startdb-C-4.log
-rw-r--r--  0 buildfarm staff  886518 May 13 15:20 contrib-install-check-C.log
-rw-r--r--  0 buildfarm staff 297 May 13 15:20 stopdb-C-4.log
-rw-r--r--  0 buildfarm staff  132144 May 13 15:23 ecpg-check.log
-rw-r--r--  0 buildfarm staff   35834 May 13 15:25 typedefs.log

which doubtless explains why they're not available from the buildfarm
server.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Error in running DBT2

[Rohit Goyal]

Hi All,

I run the the *dbt2-pgsql-build-db -w 1 *

but, after some time, I faced this error

*/home/abhi/dbt2_install/bin/dbt2-pgsql-load-stored-procs: 45: [: c:
unexpected operator*
*/home/abhi/dbt2_install/bin/dbt2-pgsql-load-stored-procs: 53: [: c:
unexpected operator*
*unknown stored function type: c*


and script ends!!

Moreover, in between the scripts running, i was looing at the ouput i found
something like

Output directory of data files: current directory

*Generating data files for 1 warehouse(s)...*
*Generating item table data...*
*BEGIN*
*ERROR:  invalid byte sequence for encoding "UTF8": 0xf9*
*CONTEXT:  COPY item, line 1*
*ROLLBACK*

This pattern the above found many times. Please guide me through!!!

Regards,
Rohit Goyal

Re: [HACKERS] wrapping in extended mode doesn't work well with default pager

[Sergey Muraviov]

Please check this patch.


2014-05-12 22:56 GMT+04:00 Sergey Muraviov :

> Hi.
>
> I'll try to fix it tomorrow.
>
>
> 2014-05-12 18:42 GMT+04:00 Tom Lane :
>
> Greg Stark  writes:
>> > On Mon, May 12, 2014 at 2:12 PM, Greg Stark  wrote:
>> >> Hm, there was an off by one error earlier in some cases, maybe we
>> >> fixed it by breaking other case. Will investigate.
>>
>> > Those spaces are coming from the ascii wrapping indicators. i.e. the
>> periods in:
>>
>> Ah.  I wonder whether anyone will complain that the format changed?
>>
>> > Apparently we used to print those with border=1 in normal mode but in
>> > expanded mode we left out the space for those on the outermost edges
>> > since there was no need for them. If we put them in for wrapped mode
>> > then we'll be inconsistent if we don't for nonwrapped mode though. And
>> > if we don't put them in for wrapped mode then there's no way to
>> > indicate wrapping versus newlines.
>>
>> Barring anyone complaining that the format changed, I'd say the issue
>> is not that you added them but that the accounting for line length
>> fails to include them.
>>
>> regards, tom lane
>>
>
>
>
> --
> Best regards,
> Sergey Muraviov
>



-- 
Best regards,
Sergey MuraviovH
diff --git a/src/bin/psql/print.c b/src/bin/psql/print.c
index 62850d8..69f4efe 100644
--- a/src/bin/psql/print.c
+++ b/src/bin/psql/print.c
@@ -1258,45 +1258,67 @@ print_aligned_vertical(const printTableContent *cont, FILE *fout)
 	if (cont->opt->format == PRINT_WRAPPED)
 	{
 		/*
-		 * Calculate the available width to wrap the columns to after
-		 * subtracting the maximum header width and separators. At a minimum
-		 * enough to print "[ RECORD N ]"
+		 * Separators width
 		 */
 		unsigned int width,
+	min_width,
 	swidth;
 
 		if (opt_border == 0)
-			swidth = 1;			/* "header data" */
+			/*
+			 * For border = 0, one space in the middle.
+			 */
+			swidth = 1;
 		else if (opt_border == 1)
-			swidth = 3;			/* "header | data" */
-		else
-			swidth = 7;			/* "| header | data |" */
-
-		/* Wrap to maximum width */
-		width = dwidth + swidth + hwidth;
-		if ((output_columns > 0) && (width > output_columns))
 		{
-			dwidth = output_columns - hwidth - swidth;
-			width = output_columns;
+			/*
+			 * For border = 1, one for the pipe (|) in the middle
+			 * between the two spaces.
+			 */
+			swidth = 3;
 		}
+		else
+			/*
+			 * For border = 2, two more for the pipes (|) at the begging and
+			 * at the end of the lines.
+			 */
+			swidth = 7;
 
-		/* Wrap to minimum width */
+		min_width = hwidth + swidth + 3;
+
+		/* 
+		 * Record header width
+		 */
 		if (!opt_tuples_only)
 		{
-			int			delta = 1 + log10(cont->nrows) - width;
-
+			/* 
+			 * Record number
+			 */
+			unsigned int rwidth = 1 + log10(cont->nrows);
 			if (opt_border == 0)
-delta += 6;		/* "* RECORD " */
+rwidth += 9;	/* "* RECORD " */
 			else if (opt_border == 1)
-delta += 10;	/* "-[ RECORD  ]" */
+rwidth += 12;	/* "-[ RECORD  ]" */
 			else
-delta += 15;	/* "+-[ RECORD  ]-+" */
+rwidth += 15;	/* "+-[ RECORD  ]-+" */
 
-			if (delta > 0)
-dwidth += delta;
+			if (rwidth > min_width)
+min_width = rwidth;
 		}
-		else if (dwidth < 3)
-			dwidth = 3;
+
+		if ((hheight > 1) && (opt_border < 2))
+			hwidth++;  /* for wrapping indicator*/
+
+		/* Wrap to minimum width */
+		width = hwidth + swidth + dwidth;
+		if ((width < min_width) || (output_columns < min_width))
+			dwidth = min_width - hwidth - swidth;
+		else if ((output_columns > 0) &&
+ (width > output_columns))
+			/*
+			 * Wrap to maximum width
+			 */
+			dwidth = output_columns - hwidth - swidth;
 	}
 
 	/* print records */
@@ -1357,12 +1379,17 @@ print_aligned_vertical(const printTableContent *cont, FILE *fout)
 int			swidth,
 			twidth = hwidth + 1;
 
-fputs(hline ? format->header_nl_left : " ", fout);
+if ((hheight > 1) || (opt_border == 2))
+	fputs(hline ? format->header_nl_left : " ", fout);
 strlen_max_width(hlineptr[hline].ptr, &twidth,
  encoding);
 fprintf(fout, "%-s", hlineptr[hline].ptr);
 
 swidth = hwidth - twidth;
+if ((hheight > 1) &&
+	(opt_border < 2) && 
+	(cont->opt->format == PRINT_WRAPPED))
+	swidth--;
 if (swidth > 0) /* spacer */
 	fprintf(fout, "%*s", swidth, " ");
 
@@ -1382,7 +1409,11 @@ print_aligned_vertical(const printTableContent *cont, FILE *fout)
 			else
 			{
 /* Header exhausted but more data for column */
-fprintf(fout, "%*s", hwidth + 2, "");
+unsigned int ewidth = hwidth + 2;
+if ((opt_border < 2) &&
+	(cont->opt->format == PRINT_WRAPPED))
+	ewidth--;
+fprintf(fout, "%*s", ewidth, "");
 			}
 
 			/* Separator */
@@ -1401,13 +1432,24 @@ print_aligned_vertical(const printTableContent *cont, FILE *fout)
 			/* Data */
 			if (!dcomplete)
 			{
-int			target_width,
+int			target_width = dwidth,
 			bytes_to_output,
 			swidth;
 
-fputs(!dcomplete && !offset ?

Re: [HACKERS] buildfarm / handling (undefined) locales

[Tomas Vondra]

On 13.5.2014 21:03, Andrew Dunstan wrote:
> 
> On 05/13/2014 02:39 PM, Tomas Vondra wrote:
>> On 13.5.2014 20:27, Tomas Vondra wrote:
>>> Hi all,
>>>
>>> a few days ago I switched magpie into an LXC container, and while
>>> fixinig unrelated issue there, I noticed that although the tests seemed
>>> to run, some of the results are actually rubbish because of missing
>>> locales.
>> I forgot to mention that I noticed another (possible) issue - ISTM the
>> buildfarm client is not packing all the log files. For for the failure
>> in REL9_3_STABLE, there are 32 log files packed:
>>
>>
>> http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=magpie&dt=2014-05-13%2014%3A59%3A46
>>
>>
>> However in the lastrun-logs I see 81 files, and upon closer inspection
>> it seems that pretty much logs for all custom locales (cs_CZ.*, sk_SK.*)
>> are missing:
>>
>>  initdb-cs_CZ.ISO-8859-2
>>  initdb-cs_CZ.UTF-8
>>  initdb-cs_CZ.WIN-1250
>>  initdb-sk_SK.ISO-8859-2
>>  initdb-sk_SK.UTF-8
>>  install-check-cs_CZ.ISO-8859-2
>>  install-check-cs_CZ.UTF-8
>>  install-check-cs_CZ.WIN-1250
>>  install-check-sk_SK.ISO-8859-2
>>  install-check-sk_SK.UTF-8
>>  pl-install-check-cs_CZ.ISO-8859-2
>>  pl-install-check-cs_CZ.UTF-8
>>  pl-install-check-cs_CZ.WIN-1250
>>  pl-install-check-sk_SK.ISO-8859-2
>>  pl-install-check-sk_SK.UTF-8
>>  startdb-cs_CZ.ISO-8859-2-1
>>  startdb-cs_CZ.ISO-8859-2-2
>>  startdb-cs_CZ.ISO-8859-2-3
>>  startdb-cs_CZ.UTF-8-1
>>  startdb-cs_CZ.UTF-8-2
>>  startdb-cs_CZ.UTF-8-3
>>  startdb-cs_CZ.WIN-1250-1
>>  startdb-cs_CZ.WIN-1250-2
>>  startdb-sk_SK.ISO-8859-2-1
>>  startdb-sk_SK.ISO-8859-2-2
>>  startdb-sk_SK.ISO-8859-2-3
>>  startdb-sk_SK.UTF-8-1
>>  startdb-sk_SK.UTF-8-2
>>  startdb-sk_SK.UTF-8-3
>>  stopdb-cs_CZ.ISO-8859-2-1
>>  stopdb-cs_CZ.ISO-8859-2-2
>>  stopdb-cs_CZ.ISO-8859-2-3
>>  stopdb-cs_CZ.UTF-8-1
>>  stopdb-cs_CZ.UTF-8-2
>>  stopdb-cs_CZ.UTF-8-3
>>  stopdb-cs_CZ.WIN-1250-1
>>  stopdb-sk_SK.ISO-8859-2-1
>>  stopdb-sk_SK.ISO-8859-2-2
>>  stopdb-sk_SK.ISO-8859-2-3
>>  stopdb-sk_SK.UTF-8-1
>>  stopdb-sk_SK.UTF-8-2
>>  stopdb-sk_SK.UTF-8-3
>>
>> A few days back this was reported properly
>> (http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=magpie&dt=2014-04-17%2000%3A07%3A36).
>>
>> I doubt this is related to the LXC container, but I was running 4.9
>> version of the buildfarm client back then, while now it's running in
>> 4.12. I'd guess this is the actual cause.
>>
> 
> 
> Clearly this is from a different run, since the link above shows the run
> failing at the contrib-install-check-en_US stage, and so of course all
> those other locales would not have been checked in that run.

Um, no, I think this (the last run in 9.3 @ magpie):


http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=magpie&dt=2014-05-13%2014%3A59%3A46

ends with this:

   Details for system "magpie" failure at stage PLCheck-cs_CZ.WIN-1250,
snapshot taken 2014-05-13 14:59:46

> So there is no error here. If you want to check for an error compare the
> contents of lastrun-logs/*.log with the contents of runlogs.tgz.

$ pwd
/var/buildfarm/magpie/build/REL9_3_STABLE/magpie.lastrun-logs

http://pastebin.com/xC33LjrE (tar -tzvf runlogs.tgz)
http://pastebin.com/HsLJmLzW (ls -l)


But maybe I'm missing something ...

Tomas


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 9.4 release notes

[Bruce Momjian]

On Fri, May  9, 2014 at 04:53:18PM +0200, Andres Freund wrote:
> Hi,
> 
> I've attached a fair number of fixes for the current state of the
> release notes. Many of them are fixing factual errors, others are more
> minors.
> I think the whole 'logical decoding' section will need an entire
> rewrite, but I'll do that separately.
> 
> I've added a couple of  for things that are clearly
> unfinished.

Slightly adjusted attached patch applied.  Thanks.  Let me know if you
have any follow-on changes.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + Everyone has their own god. +
commit 062f53518927f9bfe1820578ce79d3180b1aa2ca
Author: Bruce Momjian 
Date:   Tue May 13 15:12:54 2014 -0400

docs:  9.4 release notes adjustments

Patch by Andres Freund, slight adjustments by me

diff --git a/doc/src/sgml/release-9.4.sgml b/doc/src/sgml/release-9.4.sgml
new file mode 100644
index e537a1f..cabfbdd
*** a/doc/src/sgml/release-9.4.sgml
--- b/doc/src/sgml/release-9.4.sgml
***
*** 29,36 
  
   

!Logical change-set extraction allows database
!changes to be optionally recorded in logical format

   
  
--- 29,36 
  
   

!Logical decoding allows database
!changes to be streamed out in customizable format

   
  
***
*** 223,228 
--- 223,239 
  
  
   
+   Handle domains over arrays like plain arrays in PL/Python
+   (Rodolfo Campero)
+  
+ 
+  
+   Previously they were treated as strings.
+  
+ 
+ 
+ 
+  
Have libpq's PQconnectdbParams()
and 
  
  
+ 
+  
+   The maximum number of background workers
+   that can be registered
+   by RegisterBackgroundWorker() is now limited to
+   max_worker_processes
+  
+ 
+ 
 
  

***
*** 452,466 
  

 
! Freeze
! tuples when tables are written with CLUSTER or VACUUM FULL (Robert Haas,
  Andres Freund)
 
  
 
! This avoids the need to freeze the tuples in the future.
 

  
--- 472,486 
  

 
! Attempt to freeze
! tuples when tables are rewritten with CLUSTER or VACUUM FULL (Robert Haas,
  Andres Freund)
 
  
 
! This can avoid the need to freeze the tuples in the future.
 

  
***
*** 545,556 
  

 
! Add xid and xmin
! to system views pg_stat_activity
! and pg_stat_replication
  (Christian Kruse)
 

--- 565,573 
  

 
! Add backend_xid and backend_xmin columns to
! the system view pg_stat_activity
! and backend_xmin to pg_stat_replication
  (Christian Kruse)
 

***
*** 571,580 
 
  
 
! Such keys are faster and have improved security
! over previous options.  New variable ssl_ecdh_curve
! controls the curve that is used.
 

  
--- 588,597 
 
  
 
! Such keys are faster and have improved security over previous
! options. The new configuration
! parameter ssl_ecdh_curve
! controls which curve is used.
 

  
***
*** 617,631 
  

 
! Add SQL-level command ALTER SYSTEM command
! to edit the postgresql.conf configuration file
! (Amit Kapila)
 
  
 
! Previously postgresql.conf could only be edited at
! the file system level.
 

  
--- 634,647 
  

 
! Add SQL-level ALTER SYSTEM command
! to adjust server-wide settings (Amit Kapila)
 
  
 
! Previously such settings could only be changed by
! editing postgresql.conf at the file system level.
 

  
***
*** 680,687 
 
  
 
! Hint bits are not normally logged, except when checksums are
! enabled.  This is useful for tools like pg_rewind.
 

  
--- 696,703 
 
  
 
! Hint bits are not normally logged, except when checksums are enabled.
! This is useful for external tools like pg_rewind.
 

  
***
*** 702,710 
 
  
 
! Such libraries are auto-LOAD'ed, unlike local_preload_libraries.
 

  
--- 718,727 
 
  
 
! In contrast
! to local_preload_libraries,
! this parameter can load any shared library, not just those in
! the $libdir/plugins directory.
 

  
***
*** 775,790 
  

 
! Add recovery.conf
! parameter recovery_min_appl

Re: [HACKERS] buildfarm / handling (undefined) locales

[Andrew Dunstan]

On 05/13/2014 02:39 PM, Tomas Vondra wrote:

On 13.5.2014 20:27, Tomas Vondra wrote:

Hi all,

a few days ago I switched magpie into an LXC container, and while
fixinig unrelated issue there, I noticed that although the tests seemed
to run, some of the results are actually rubbish because of missing locales.

I forgot to mention that I noticed another (possible) issue - ISTM the
buildfarm client is not packing all the log files. For for the failure
in REL9_3_STABLE, there are 32 log files packed:


http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=magpie&dt=2014-05-13%2014%3A59%3A46

However in the lastrun-logs I see 81 files, and upon closer inspection
it seems that pretty much logs for all custom locales (cs_CZ.*, sk_SK.*)
are missing:

 initdb-cs_CZ.ISO-8859-2
 initdb-cs_CZ.UTF-8
 initdb-cs_CZ.WIN-1250
 initdb-sk_SK.ISO-8859-2
 initdb-sk_SK.UTF-8
 install-check-cs_CZ.ISO-8859-2
 install-check-cs_CZ.UTF-8
 install-check-cs_CZ.WIN-1250
 install-check-sk_SK.ISO-8859-2
 install-check-sk_SK.UTF-8
 pl-install-check-cs_CZ.ISO-8859-2
 pl-install-check-cs_CZ.UTF-8
 pl-install-check-cs_CZ.WIN-1250
 pl-install-check-sk_SK.ISO-8859-2
 pl-install-check-sk_SK.UTF-8
 startdb-cs_CZ.ISO-8859-2-1
 startdb-cs_CZ.ISO-8859-2-2
 startdb-cs_CZ.ISO-8859-2-3
 startdb-cs_CZ.UTF-8-1
 startdb-cs_CZ.UTF-8-2
 startdb-cs_CZ.UTF-8-3
 startdb-cs_CZ.WIN-1250-1
 startdb-cs_CZ.WIN-1250-2
 startdb-sk_SK.ISO-8859-2-1
 startdb-sk_SK.ISO-8859-2-2
 startdb-sk_SK.ISO-8859-2-3
 startdb-sk_SK.UTF-8-1
 startdb-sk_SK.UTF-8-2
 startdb-sk_SK.UTF-8-3
 stopdb-cs_CZ.ISO-8859-2-1
 stopdb-cs_CZ.ISO-8859-2-2
 stopdb-cs_CZ.ISO-8859-2-3
 stopdb-cs_CZ.UTF-8-1
 stopdb-cs_CZ.UTF-8-2
 stopdb-cs_CZ.UTF-8-3
 stopdb-cs_CZ.WIN-1250-1
 stopdb-sk_SK.ISO-8859-2-1
 stopdb-sk_SK.ISO-8859-2-2
 stopdb-sk_SK.ISO-8859-2-3
 stopdb-sk_SK.UTF-8-1
 stopdb-sk_SK.UTF-8-2
 stopdb-sk_SK.UTF-8-3

A few days back this was reported properly
(http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=magpie&dt=2014-04-17%2000%3A07%3A36).
I doubt this is related to the LXC container, but I was running 4.9
version of the buildfarm client back then, while now it's running in
4.12. I'd guess this is the actual cause.




Clearly this is from a different run, since the link above shows the run 
failing at the contrib-install-check-en_US stage, and so of course all 
those other locales would not have been checked in that run.


So there is no error here. If you want to check for an error compare the 
contents of lastrun-logs/*.log with the contents of runlogs.tgz.


cheers

andrew


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] buildfarm / handling (undefined) locales

[Tom Lane]

Tomas Vondra  writes:
> a few days ago I switched magpie into an LXC container, and while
> fixinig unrelated issue there, I noticed that although the tests seemed
> to run, some of the results are actually rubbish because of missing locales.

> So for example when the system misses cs_CZ.WIN-1250 (i.e. czech locale
> with windows codepage 1250), initdb actually did this

> The files belonging to this database system will be owned by user
> "pgbuild".
> This user must also own the server process.

> initdb: invalid locale name ""
> initdb: invalid locale name ""
> initdb: invalid locale name ""
> initdb: invalid locale name ""
> initdb: invalid locale name ""
> initdb: invalid locale name ""
> The database cluster will be initialized with locale "C".
> The default database encoding has accordingly been set to
> "SQL_ASCII".
> The default text search configuration will be set to "english".

Hm, I'm a bit confused as to what you actually did here.  The "invalid
locale name" bleats seem to indicate that no --locale or --lc_xxx options
were given on the command line; correct?  If so the issue is presumably
that the environment variable(s) were set to incorrect values.  While
we *could* abort in that situation, I've never heard of any program
that did; the normal response is to silently ignore the environment
variables and use C locale.  We're not being exactly silent about it
but I think the outcome is the expected one.

There is a comment in the code about this:

/* should we exit here? */
if (res == NULL)
fprintf(stderr, _("%s: invalid locale name \"%s\"\n"),
progname, locale);

but I think what's being questioned is whether an incorrect locale
name *given as a command line argument* should result in an abort.
That might be a good idea, but it looks like it'd require some
restructuring of the code to make it possible to distinguish the
case from bad-environment.

> Yeah, not really what we were shooting for. I've fixed this by defining
> the missing locales, and indeed - magpie now fails in plpython tests.

I saw that earlier today (tho right now the buildfarm server seems to
not be responding :-().  Probably we should use some more-widely-used
character code in that specific test?

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Sending out a request for more buildfarm animals?

[Tomas Vondra]

On 10.5.2014 20:21, Tomas Vondra wrote:
> On 9.5.2014 00:47, Tomas Vondra wrote:
>
>> So, if I get this right, the proposal is to have 7 animals:
>>
>>
>> 1) all branches/locales, frequent builds (every few hours)
>>   magpie  - gcc
>>   fulmar  - icc
>>   treepie - clang
>>
>> 2) single branch/locale, CLOBBER, built once a week
>>   magpie2 - gcc
>>   fulmar2 - icc
>>   treepie - clang
>>
>> 3) single branch/locale, recursive CLOBBER, built once a month
>>
>>
>> I don't particularly mind the number of animals, although I was shooting
>> for lower number.
>>
>> The only question is - should we use 3 animals for the recursive CLOBBER
>> too? I mean, one for each compiler?
> 
> OK. I've switched the three original animals (magpie, fulmar, treepie)
> back to the original configuration (no clobber, all branches, multiple
> locales).
> 
> And I've requested 6 more animals - two for each compiler. One set for
> tests with basic CLOBBER, one set for recursive CLOBBER.

Can someone please approve the animals I've requested a few days ago?
I'm already running the clobber tests with '--nosend --nostatus' and
it's already reporting some errors. Would be nice to get it to the
buildfarm.

regards
Tomas


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] buildfarm / handling (undefined) locales

[Tomas Vondra]

On 13.5.2014 20:27, Tomas Vondra wrote:
> Hi all,
> 
> a few days ago I switched magpie into an LXC container, and while
> fixinig unrelated issue there, I noticed that although the tests seemed
> to run, some of the results are actually rubbish because of missing locales.

I forgot to mention that I noticed another (possible) issue - ISTM the
buildfarm client is not packing all the log files. For for the failure
in REL9_3_STABLE, there are 32 log files packed:


http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=magpie&dt=2014-05-13%2014%3A59%3A46

However in the lastrun-logs I see 81 files, and upon closer inspection
it seems that pretty much logs for all custom locales (cs_CZ.*, sk_SK.*)
are missing:

initdb-cs_CZ.ISO-8859-2
initdb-cs_CZ.UTF-8
initdb-cs_CZ.WIN-1250
initdb-sk_SK.ISO-8859-2
initdb-sk_SK.UTF-8
install-check-cs_CZ.ISO-8859-2
install-check-cs_CZ.UTF-8
install-check-cs_CZ.WIN-1250
install-check-sk_SK.ISO-8859-2
install-check-sk_SK.UTF-8
pl-install-check-cs_CZ.ISO-8859-2
pl-install-check-cs_CZ.UTF-8
pl-install-check-cs_CZ.WIN-1250
pl-install-check-sk_SK.ISO-8859-2
pl-install-check-sk_SK.UTF-8
startdb-cs_CZ.ISO-8859-2-1
startdb-cs_CZ.ISO-8859-2-2
startdb-cs_CZ.ISO-8859-2-3
startdb-cs_CZ.UTF-8-1
startdb-cs_CZ.UTF-8-2
startdb-cs_CZ.UTF-8-3
startdb-cs_CZ.WIN-1250-1
startdb-cs_CZ.WIN-1250-2
startdb-sk_SK.ISO-8859-2-1
startdb-sk_SK.ISO-8859-2-2
startdb-sk_SK.ISO-8859-2-3
startdb-sk_SK.UTF-8-1
startdb-sk_SK.UTF-8-2
startdb-sk_SK.UTF-8-3
stopdb-cs_CZ.ISO-8859-2-1
stopdb-cs_CZ.ISO-8859-2-2
stopdb-cs_CZ.ISO-8859-2-3
stopdb-cs_CZ.UTF-8-1
stopdb-cs_CZ.UTF-8-2
stopdb-cs_CZ.UTF-8-3
stopdb-cs_CZ.WIN-1250-1
stopdb-sk_SK.ISO-8859-2-1
stopdb-sk_SK.ISO-8859-2-2
stopdb-sk_SK.ISO-8859-2-3
stopdb-sk_SK.UTF-8-1
stopdb-sk_SK.UTF-8-2
stopdb-sk_SK.UTF-8-3

A few days back this was reported properly
(http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=magpie&dt=2014-04-17%2000%3A07%3A36).
I doubt this is related to the LXC container, but I was running 4.9
version of the buildfarm client back then, while now it's running in
4.12. I'd guess this is the actual cause.

regards
Tomas


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] buildfarm / handling (undefined) locales

[Tomas Vondra]

Hi all,

a few days ago I switched magpie into an LXC container, and while
fixinig unrelated issue there, I noticed that although the tests seemed
to run, some of the results are actually rubbish because of missing locales.

So for example when the system misses cs_CZ.WIN-1250 (i.e. czech locale
with windows codepage 1250), initdb actually did this

The files belonging to this database system will be owned by user
"pgbuild".
This user must also own the server process.

initdb: invalid locale name ""
initdb: invalid locale name ""
initdb: invalid locale name ""
initdb: invalid locale name ""
initdb: invalid locale name ""
initdb: invalid locale name ""
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to
"SQL_ASCII".
The default text search configuration will be set to "english".

Yeah, not really what we were shooting for. I've fixed this by defining
the missing locales, and indeed - magpie now fails in plpython tests.

Now, the question is - is this a pilot error or should we check this
somehow? For example a simple check whether server_encoding/lc_* seems
like a good idea (although I wonder whether the fallback logic in initdb
is really a good idea).

regards
Tomas


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Wanna help PostgreSQL

[Jeff Janes]

On Tue, May 13, 2014 at 10:12 AM, Zhe-Wei Jiang wrote:

> Hi everyone,
>
> I'm new to the open source community, and wanna help in some
> implementation work.
> However, when I looked into the TODO list, I found most of the items are
> very old.
>

So that is one thing you can do to help, try to improve the todo page!
 I've thought we should create a new badge for staleness (in addition to
the Easy and Done ones) and add it to everything currently on the page,
then remove it from ones that were recently evaluated and still seem to be
relevant.

And in the process, we should clarify why they haven't been done yet.
 Often what really needs to be done is not the implementation, but rather
the evaluation of whether it is worth the trade offs involved.


> Therefore, I'd like to know if there is any entry-level implementation
> needed by PostgreSQL now?
>

Did anything on the todo list strike your fancy, especially anything marked
Easy?  Even if it is old, it still might be valid. Then discuss it here on
the hackers list, and if it not still valid we can remove it from the list.
 No point putting the sour milk back in the fridge.

How much experience do you have using PostgreSQL (or for that matter,
coding in C)?

Cheers,

Jeff

Re: [HACKERS] Wanna help PostgreSQL

[Stephen Frost]

* Zhe-Wei Jiang (jrreinha...@gmail.com) wrote:
> Therefore, I'd like to know if there is any entry-level implementation
> needed by PostgreSQL now?
> Any advise is appreciated.

You might take a look at the GSoC ideas page instead:

https://wiki.postgresql.org/wiki/GSoC_2014

Thanks,

Stephen


signature.asc
Description: Digital signature

[HACKERS] Wanna help PostgreSQL

[Zhe-Wei Jiang]

Hi everyone,

I'm new to the open source community, and wanna help in some implementation
work.
However, when I looked into the TODO list, I found most of the items are
very old.
Therefore, I'd like to know if there is any entry-level implementation
needed by PostgreSQL now?
Any advise is appreciated.
Thanks!!

Regards,
Zhe-Wei Jiang

Re: [HACKERS] [PATCH] Fix harmless access to uninitialized memory in ri_triggers.c.

[Heikki Linnakangas]

On 05/08/2014 07:33 PM, and...@2ndquadrant.com wrote:

When cache invalidations arrive while ri_LoadConstraintInfo() is busy
filling a new cache entry, InvalidateConstraintCacheCallBack()
compares the - not yet initialized - oidHashValue field with the
to-be-invalidated hash value. To fix check whether the entry is
already marked as invalid.


Thanks, applied.

- Heikki


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New timezones used in regression tests

[Christoph Berg]

Re: Alvaro Herrera 2014-05-13 <20140513135526.gr6...@eldon.alvh.no-ip.org>
> > I especially like MTC, Mars Time Coordinated. But whatever scheme gets
> > chosen, it won't be a standard 24h day, so PostgreSQL has a whole lot
> > of different problems to solve than to "fix" that little
> > Mars/Mons_Olympus gem now... :)
> 
> Maybe a new type, mars_timestamptz()?  Or perhaps the celestial body
> name should be part of the typmod for standard timestamptz ...?

Supporting other worlds would warrant bumping to PostgreSQL 10.0 :)

Christoph
-- 
c...@df7cb.de | http://www.df7cb.de/


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_recvlogical, stdout and SIGHUP

[Heikki Linnakangas]

On 05/13/2014 04:35 PM, Andres Freund wrote:

On 2014-05-13 16:31:25 +0300, Heikki Linnakangas wrote:

Another thing I noticed is that if when the output goes to a file, the file
isn't re-opened immediately on SIGHUP. Only after receiving some data from
the server. I believe that's also not intentional.


Hm. I can't really get excited about that one. Not doing that seems to
complicate matters unneccessarily. What's the problem here?


Not sure if it matters in any real-world scenario, but I found it pretty 
surprising while playing with it. It should be trivial to fix; ISTM the 
problem is that there is a "continue" in the loop when select() is 
interrupted by signal, but the re-opening is done after the select() in 
the loop. I think all you need to do is move the check for output_reopen 
to the beginning of the loop.


- Heikki


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 9.5: UPDATE/DELETE .. ORDER BY .. LIMIT ..

[Tom Lane]

Amit Kapila  writes:
> How about sorting step, are you thinking to have MergeAppend
> node for it beneath ModifyTable?

Well yeah, that's pretty much the point.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] btree_gist valgrind warnings about uninitialized memory

[Andres Freund]

On 2014-05-13 15:33:08 +0300, Heikki Linnakangas wrote:
> On 05/06/2014 10:03 PM, Andres Freund wrote:
> >Hi,
> >
> >I am not sure whether these are new for 9.4 but they're worth looking at
> >anyway:
> >Valgrind was run with:
> >  --trace-children=yes --quiet \
> >  --track-origins=yes --leak-check=no \
> >  --read-var-info=yes \
> >  --suppressions=/home/andres/src/postgresql/src/tools/valgrind.supp \
> >  
> >All seem to be due btree_bit's fault and they seem to have a common
> >origin. Didn't look at code.

> I committed a fix to gbt_bit_xfrm() to zero the padding bytes.

Thanks!

> Thanks for the Valgrinding!

That leaves the spgist thing and some infrastructure till we can setup a
valgrind animal... From what we've caught with it so far that seems
rather worthwile.
What's your plans with your spgist fix? Commit it once 9.5 is branched?

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Updating config.guess/config.sub for ppc64le

[Tom Lane]

Christoph Berg  writes:
> Re: Tom Lane 2014-05-10 <27476.1399729...@sss.pgh.pa.us>
>> Our normal procedure is
>> o update config.guess and config.sub at the start of beta
>> (from http://savannah.gnu.org/projects/config)

> Fwiw, shouldn't that also happen in back branches?

No, we are not in the habit of back-patching such changes, at least
not automatically.  I'd be willing to consider it once the new scripts
have survived a beta-testing cycle ... however, a look at our commit
logs shows we have never actually updated config.guess/config.sub in
any back branch.

> Updating config.* there gives you portability to new architectures for
> free - and there should be no risk of breaking anything.

The policy of not back-patching dates back to circa 2000, when new
config scripts *routinely* broke things due to changes in what they
printed on some machines (again, there's lots of evidence on this point
in our commit history).  Perhaps that's less of a concern nowadays.
Still, there seems to be zero field demand for doing this.

As for "new architectures for free", nope --- spinlock assembly code
is usually the gating factor for that, not the config scripts.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New timezones used in regression tests

[Alvaro Herrera]

Christoph Berg wrote:

> Of course, Wikipedia has something to say about this:
> http://en.wikipedia.org/wiki/Timekeeping_on_Mars

Nice.

> I especially like MTC, Mars Time Coordinated. But whatever scheme gets
> chosen, it won't be a standard 24h day, so PostgreSQL has a whole lot
> of different problems to solve than to "fix" that little
> Mars/Mons_Olympus gem now... :)

Maybe a new type, mars_timestamptz()?  Or perhaps the celestial body
name should be part of the typmod for standard timestamptz ...?

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New timezones used in regression tests

[Alvaro Herrera]

Tom Lane wrote:
> Christoph Berg  writes:
> > 84df54b22e8035addc7108abd9ff6995e8c49264 introduced timestamp
> > constructors. In the regression tests, various time zones are tested,
> > including America/Metlakatla. Now, if you configure using
> > --with-system-tzdata, you'll get an error if that zone isn't there.
> > Unfortunately, this is what I'm getting now when trying to build beta1
> > on Ubuntu 10.04 (lucid) with tzdata 2010i-1:
> 
> I agree, that seems an entirely gratuitous choice of zone.

Well, it wasn't entirely gratuituous -- it had a very large offset from
UTC to one direction and made a sudden jump to the opposite direction in
1900.  Obviously I didn't notice the timezone had been created in 2011.
(I bet if this hadn't been tried in a platform with such an old tzdata,
this wouldn't have been noticed.  Don't Ubuntu update tzdata on their
LTS platforms!?)

> I'm quite unimpressed by the dependency on Mars/Mons_Olympus, too ... that
> might not fail *today*, but considering it's a real location, assuming it
> is not in the IANA database seems like a recipe for future failure.
> Maybe something like Nehwon/Lankhmar?  Or maybe we should not try to be
> cute but just test Foo/Bar.

I liked Mars/Mons_Olympus, but Nehwon/Lankhmar is good too.
/me adds more to his to-read list

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_recvlogical, stdout and SIGHUP

[Andres Freund]

On 2014-05-13 16:31:25 +0300, Heikki Linnakangas wrote:
> pg_recvlogical re-opens the output file on SIGHUP. If the output goes to
> stdout, it will close stdout on SIGHUP. That's a bug, isn't it?

Yes. An annoying one at that because it'll mean a a new connection will
use that fd and we'll start writing stdout stuff to it...

> Another thing I noticed is that if when the output goes to a file, the file
> isn't re-opened immediately on SIGHUP. Only after receiving some data from
> the server. I believe that's also not intentional.

Hm. I can't really get excited about that one. Not doing that seems to
complicate matters unneccessarily. What's the problem here?

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] pg_recvlogical, stdout and SIGHUP

[Heikki Linnakangas]

pg_recvlogical re-opens the output file on SIGHUP. If the output goes to 
stdout, it will close stdout on SIGHUP. That's a bug, isn't it?


Another thing I noticed is that if when the output goes to a file, the 
file isn't re-opened immediately on SIGHUP. Only after receiving some 
data from the server. I believe that's also not intentional.


- Heikki


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] btree_gist valgrind warnings about uninitialized memory

[Heikki Linnakangas]

On 05/06/2014 10:03 PM, Andres Freund wrote:

Hi,

I am not sure whether these are new for 9.4 but they're worth looking at
anyway:
Valgrind was run with:
  --trace-children=yes --quiet \
  --track-origins=yes --leak-check=no \
  --read-var-info=yes \
  --suppressions=/home/andres/src/postgresql/src/tools/valgrind.supp \
  
All seem to be due btree_bit's fault and they seem to have a common
origin. Didn't look at code.

==27401== Conditional jump or move depends on uninitialised value(s)
==27401==at 0x4C2C812: bcmp (mc_replace_strmem.c:935)
==27401==by 0x8A8533: byteacmp (varlena.c:2735)
==27401==by 0x8D70A2: DirectFunctionCall2Coll (fmgr.c:1050)
==27401==by 0x180822C6: gbt_bitcmp (btree_bit.c:68)
==27401==by 0x18079C05: gbt_vsrt_cmp (btree_utils_var.c:430)
==27401==by 0x919F66: qsort_arg (qsort_arg.c:121)
==27401==by 0x91A531: qsort_arg (qsort_arg.c:186)
==27401==by 0x91A531: qsort_arg (qsort_arg.c:186)
==27401==by 0x18079E68: gbt_var_picksplit (btree_utils_var.c:484)
==27401==by 0x180825AC: gbt_bit_picksplit (btree_bit.c:180)
==27401==by 0x8D7AD3: FunctionCall2Coll (fmgr.c:1324)
==27401==by 0x497701: gistUserPicksplit (gistsplit.c:433)


In a btree_gist bit/varbit index, an internal index item  consists of a 
lower and upper key. The full Varbit struct is not stored as the 
lower/upper key, but only the "bits" array. The lower key is expanded to 
next INTALIGNed size, so that when the lower and upper keys are put 
together into one Datum, the length-field of the upper key is properly 
aligned.


The bug is in gbt_bit_xfrm(), which expands a VarBit struct to an 
INTALIGNed bytea, containing just the bits array. It doesn't initialize 
the padding bytes. That can confuse comparisons against lower/upper 
keys, as the comparison routine doesn't know which bytes are padding, 
and will merrily compare them as well. I was able to get wrong query 
results after modifying gbt_bit_xfrm() to knowingly initialize the 
padding bytes to random():


create table varbittest (b varbit);
insert into varbittest select '001'::varbit from generate_series(1, 100) 
union all select '01'::varbit from generate_series(1, 100) union all 
select '1'::varbit from generate_series(1, 100);

create index varbittest_idx on varbittest using gist(b);
set enable_seqscan=off;

-- should return 200
postgres=# select count(*) from varbittest where b >= '01';
 count
---
   169
(1 row)


I committed a fix to gbt_bit_xfrm() to zero the padding bytes. However, 
if you have an existing btree_gist index on bit/varbit, you will have to 
reindex as there can be non-zero padding bytes on disk already. That 
should be mentioned in the release notes.


Thanks for the Valgrinding!

- Heikki


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal for CSN based snapshots

[Heikki Linnakangas]

On 05/13/2014 08:08 AM, Rajeev rastogi wrote:

>The core of the design is to store the LSN of the commit record in
>pg_clog. Currently, we only store 2 bits per transaction there,
>indicating if the transaction committed or not, but the patch will
>expand it to 64 bits, to store the LSN. To check the visibility of an
>XID in a snapshot, the XID's commit LSN is looked up in pg_clog, and
>compared with the snapshot's LSN.

Isn't it will be bit in-efficient to look in to pg_clog to read XID's commit
LSN for every visibility check?


Maybe. If no hint bit is set on the tuple, you have to check the clog 
anyway to determine if the tuple is committed. And if for XIDs older 
than xmin or newer than xmax, you don't need to check pg_clog. But it's 
true that for tuples with hint bit set, and xmin < XID < xmax, you have 
to check the pg_clog in the new system, when currently you only need to 
do a binary search of the local array in the snapshot. My gut feeling is 
that it won't be significantly slower in practice. If it becomes a 
problem, some rearrangement pg_clog code might help, or you could build 
a cache of XID->CSN mappings that you've alread looked up in 
SnapshotData. So I don't think that's going to be a show-stopper.


- Heikki


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal for CSN based snapshots

[Heikki Linnakangas]

On 05/13/2014 09:44 AM, Amit Kapila wrote:

On Mon, May 12, 2014 at 7:26 PM, Heikki Linnakangas
 wrote:

In theory, we could use a snapshot LSN as the cutoff-point for
HeapTupleSatisfiesVisibility(). Maybe it's just because this is new, but
that makes me feel uneasy.


To accomplish this won't XID-CSN map table be required and how will
it be maintained (means when to clear and add a entry to that map table)?


Not sure I understand. The clog is a mapping from XID to CSN. What 
vacuum needs to know is whether the xmin and/or xmax is visible to 
everyone (and whether they committed or aborted). To determine that, it 
needs the oldest still active snapshot LSN. That can be found by 
scanning the proc array. It's pretty much the same as a regular MVCC 
visibility check, but using the oldest still-active snapshot.


- Heikki


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Updating config.guess/config.sub for ppc64le

[Christoph Berg]

Re: Tom Lane 2014-05-10 <27476.1399729...@sss.pgh.pa.us>
> Christoph Berg  writes:
> > to support ppc64le, config.guess needs to be updated. The attached
> > patch is what was reported to work for Ubuntu.
> 
> Our normal procedure is
> 
>   o update config.guess and config.sub at the start of beta
> (from http://savannah.gnu.org/projects/config)
> 
> (memo to self: this better happen today).  Is that patch just subbing
> in the latest upstream scripts, or is it doing something else?

Fwiw, shouldn't that also happen in back branches?

Updating config.* there gives you portability to new architectures for
free - and there should be no risk of breaking anything.

Christoph
-- 
c...@df7cb.de | http://www.df7cb.de/


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New timezones used in regression tests

[Christoph Berg]

Re: Robert Haas 2014-05-13 

> On Mon, May 12, 2014 at 7:16 PM, Tom Lane  wrote:
> > I'm quite unimpressed by the dependency on Mars/Mons_Olympus, too ... that
> > might not fail *today*, but considering it's a real location, assuming it
> > is not in the IANA database seems like a recipe for future failure.
> > Maybe something like Nehwon/Lankhmar?  Or maybe we should not try to be
> > cute but just test Foo/Bar.
> 
> Personally, I think it would be *awesome* if our regression tests
> started failing due to the establishment of Mars/Mons_Olympus as a
> real time zone.

Of course, Wikipedia has something to say about this:
http://en.wikipedia.org/wiki/Timekeeping_on_Mars

I especially like MTC, Mars Time Coordinated. But whatever scheme gets
chosen, it won't be a standard 24h day, so PostgreSQL has a whole lot
of different problems to solve than to "fix" that little
Mars/Mons_Olympus gem now... :)

Christoph
-- 
c...@df7cb.de | http://www.df7cb.de/


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers