Re: [HACKERS] initdb failure on Debian sid/mips64el in EventTriggerEndCompleteQuery

[Christoph Berg]

Re: Tom Lane 2017-08-13 <14677.1502638...@sss.pgh.pa.us>
> Christoph Berg  writes:
> > Seems to be a gcc-7 problem affecting several packages on mips64el:
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871514
> 
> Hm, unless there is a use of sigsetjmp earlier in that clamav
> routine, I would not assume that that's the same issue.  The bug
> I suspect we are looking at here is very specific to sigsetjmp
> callers: it usually amounts to the compiler unsafely trying to
> use the same temporary location for multiple purposes.

It appears to have been the same issue - non-long ints spilled on the
stack and loaded back as long int:

Changes:
 gcc-7 (7.2.0-3) unstable; urgency=high
 .
   * Update to SVN 20170901 (r251583) from the gcc-7-branch.
 - Fix PR target/81504 (PPC), PR c++/82040.
   * Apply proposed patch for PR target/81803 (James Cowgill), conditionally
 for mips* targets. Closes: #871514.

The package built successfully on mips64el now.

Christoph


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure on Debian sid/mips64el in EventTriggerEndCompleteQuery

[Christoph Berg]

Re: Tom Lane 2017-08-13 <14517.1502638...@sss.pgh.pa.us>
> I suspect you could work around this with
> 
>   boolisCompleteQuery = (context <= PROCESS_UTILITY_QUERY);
> - boolneedCleanup;
> + volatile bool   needCleanup;
>   boolcommandCollected = false;
> 
> If that fixes it, it's definitely a compiler bug.  That function does
> not change needCleanup after the sigsetjmp call, so per POSIX it
> should not have to label the variable volatile.  This is far from
> being the first such bug we've seen though.

In the meantime, gcc-7 is at version 7.2.0-1, so I gave 9.6 on
mips64el a new try. It's still failing at initdb time, and indeed
adding "volatile" makes initdb proceed, but then the rest of the
testsuite fails in various ways:

DETAIL:  Failed process was running: CREATE TABLE enumtest_child (parent 
rainbow REFERENCES enumtest_parent);
DETAIL:  Failed process was running: create table trigtest2 (i int references 
trigtest(i) on delete cascade);
DETAIL:  Failed process was running: CREATE TABLE trunc_b (a int REFERENCES 
truncate_a);
DETAIL:  Failed process was running: CREATE SCHEMA evttrig
CREATE TABLE one (col_a SERIAL PRIMARY KEY, col_b text DEFAULT 
'forty two')
CREATE INDEX one_idx ON one (col_b)
CREATE TABLE two (col_c INTEGER CHECK (col_c > 0) REFERENCES 
one DEFAULT 42);

Hopefully the compiler gets fixed soonish on mips64el...

Thanks for the analysis,
Christoph


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure on Debian sid/mips64el in EventTriggerEndCompleteQuery

[Tom Lane]

Christoph Berg  writes:
> Seems to be a gcc-7 problem affecting several packages on mips64el:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871514

Hm, unless there is a use of sigsetjmp earlier in that clamav
routine, I would not assume that that's the same issue.  The bug
I suspect we are looking at here is very specific to sigsetjmp
callers: it usually amounts to the compiler unsafely trying to
use the same temporary location for multiple purposes.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure on Debian sid/mips64el in EventTriggerEndCompleteQuery

[Tom Lane]

Christoph Berg  writes:
> 10beta3 and 9.6.4 are both failing during initdb on mips64el on
> Debian/sid (unstable):
> All other architectures have succeeded, as well as the 9.6.4 build for
> Debian/stretch (stable) on mips64el. The difference might be the
> compiler version (6.3.0 vs 7.1.0).

It's hard to explain that stack trace other than as a compiler bug.
There shouldn't be any event triggers active here, so 
EventTriggerBeginCompleteQuery should have done nothing and returned
false.  I don't put complete faith in gdb reports of local variable
values, but it says
needCleanup = 0 '\000'
which agrees with that.  Also the core dump appears to be because
currentEventTriggerState is NULL (please check that), which is
expected if EventTriggerBeginCompleteQuery did nothing.  However, then
EventTriggerEndCompleteQuery should not have gotten called at all.

I suspect you could work around this with

boolisCompleteQuery = (context <= PROCESS_UTILITY_QUERY);
-   boolneedCleanup;
+   volatile bool   needCleanup;
boolcommandCollected = false;

If that fixes it, it's definitely a compiler bug.  That function does
not change needCleanup after the sigsetjmp call, so per POSIX it
should not have to label the variable volatile.  This is far from
being the first such bug we've seen though.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure on Debian sid/mips64el in EventTriggerEndCompleteQuery

[Christoph Berg]

Re: To PostgreSQL Hackers 2017-08-13 
<20170813130127.g3tcyzzvuvlpz...@msg.df7cb.de>
> 10beta3 and 9.6.4 are both failing during initdb on mips64el on
> Debian/sid (unstable):
> 
> https://buildd.debian.org/status/fetch.php?pkg=postgresql-9.6=mips64el=9.6.4-1=1502374949=0
> https://buildd.debian.org/status/fetch.php?pkg=postgresql-10=mips64el=10%7Ebeta3-1=1502535836=0
> 
> All other architectures have succeeded, as well as the 9.6.4 build for
> Debian/stretch (stable) on mips64el. The difference might be the
> compiler version (6.3.0 vs 7.1.0).

Seems to be a gcc-7 problem affecting several packages on mips64el:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871514

Christoph


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] initdb failure on Debian sid/mips64el in EventTriggerEndCompleteQuery

[Christoph Berg]

10beta3 and 9.6.4 are both failing during initdb on mips64el on
Debian/sid (unstable):

https://buildd.debian.org/status/fetch.php?pkg=postgresql-9.6=mips64el=9.6.4-1=1502374949=0
https://buildd.debian.org/status/fetch.php?pkg=postgresql-10=mips64el=10%7Ebeta3-1=1502535836=0

All other architectures have succeeded, as well as the 9.6.4 build for
Debian/stretch (stable) on mips64el. The difference might be the
compiler version (6.3.0 vs 7.1.0).

Command was: "initdb" -D 
"/home/myon/postgresql-9.6/postgresql-9.6-9.6.3/build/src/t
est/regress/./tmp_check/data" --noclean --nosync > 
"/home/myon/postgresql-9.6/postgr
esql-9.6-9.6.3/build/src/test/regress/log/initdb.log" 2>&1

 build/src/test/regress/log/initdb.log 
Running in noclean mode.  Mistakes will not be cleaned up.
The files belonging to this database system will be owned by user "myon".
This user must also own the server process.

The database cluster will be initialized with locales
  COLLATE:  de_DE.utf8
  CTYPE:de_DE.utf8
  MESSAGES: C
  MONETARY: de_DE.utf8
  NUMERIC:  de_DE.utf8
  TIME: de_DE.utf8
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "german".

Data page checksums are disabled.

creating directory 
/home/myon/postgresql-9.6/postgresql-9.6-9.6.3/build/src/test/regress/./tmp_check/data
 ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... Segmentation fault (core dumped)
child process exited with exit code 139


$ gdb build/tmp_install/usr/lib/postgresql/9.6/bin/postgres 
build/src/test/regress/tmp_check/data/core 
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "mips64el-linux-gnuabi64".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from 
build/tmp_install/usr/lib/postgresql/9.6/bin/postgres...done.
[New LWP 24217]
[Thread debugging using libthread_db enabled]
Using host libthread_db library 
"/lib/mips64el-linux-gnuabi64/libthread_db.so.1".
Core was generated by 
`/home/myon/postgresql-9.6/postgresql-9.6-9.6.3/build/tmp_install/usr/lib/postgr'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00aababa6634 in EventTriggerEndCompleteQuery ()
at ./build/../src/backend/commands/event_trigger.c:1263
1263MemoryContextDelete(currentEventTriggerState->cxt);
(gdb) bt full
#0  0x00aababa6634 in EventTriggerEndCompleteQuery ()
at ./build/../src/backend/commands/event_trigger.c:1263
prevstate = 
#1  0x00aabad6d508 in ProcessUtilitySlow (
parsetree=parsetree@entry=0xaac3688428, 
queryString=queryString@entry=0xaac3687888 "REVOKE ALL on pg_authid FROM 
public;\n", context=context@entry=PROCESS_UTILITY_TOPLEVEL, 
params=params@entry=0x0, 
completionTag=completionTag@entry=0x985218 "", 
dest=0xaabb0a0378 ) at 
./build/../src/backend/tcop/utility.c:1582
isTopLevel = 1 '\001'
isCompleteQuery = 1 '\001'
needCleanup = 0 '\000'
commandCollected = 
address = {classId = 2, objectId = 0, objectSubId = 0}
secondaryObject = {classId = 0, objectId = 0, objectSubId = 0}
#2  0x00aabad6c6cc in standard_ProcessUtility (parsetree=0xaac3688428, 
queryString=0xaac3687888 "REVOKE ALL on pg_authid FROM public;\n", 
context=, params=0x0, dest=0xaabb0a0378 , 
completionTag=0x985218 "") at ./build/../src/backend/tcop/utility.c:907
isTopLevel = 1 '\001'
__func__ = "standard_ProcessUtility"
#3  0x00aabad6d33c in ProcessUtility (parsetree=, 
queryString=, context=, params=, 
dest=, completionTag=)
at ./build/../src/backend/tcop/utility.c:336
No locals.
#4  0x00aabad68e80 in PortalRunUtility (portal=portal@entry=0xaac368a8a8, 
utilityStmt=utilityStmt@entry=0xaac3688428, 
isTopLevel=isTopLevel@entry=1 '\001', 
setHoldSnapshot=setHoldSnapshot@entry=0 '\000', 
dest=0xaabb0a0378 , completionTag=0x985218 "")
at ./build/../src/backend/tcop/pquery.c:1193
snapshot = 0xaac368c8e8
__func__ = "PortalRunUtility"
#5  0x00aabad69d70 in PortalRunMulti (portal=portal@entry=0xaac368a8a8, 

[HACKERS] initdb failure on RH 5.10

[BRUSSER Michael]

Hi,

One of our sites ran into a problem with database installation:

initdb failed
FATAL: unexpected data beyond EOF in block 19 of relation base/1/2609
HINT: This has been seen to occur with buggy kernels; consider updating your 
system.
STATEMENT: COMMENT ON FUNCTION euc_jis_2004_to_shift_jis_2004
  (INTEGER, INTEGER, CSTRING, INTERNAL, INTEGER)
   IS 'internal conversion function for EUC_JIS_2004 to SHIFT_JIS_2004';


initdb is called like this:
initdb -D data-dir -L input-dir -E UTF8 --locale=C

This is Postgres 8.4.4, the installation piece has been stable and always 
worked, but this time they have a new Red Hat 5.10 server

# uname -a
Linux hostname 2.6.18-371.el5 #1 SMP Thu Sep 5 21:21:44 EDT 2013 x86_64 
x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.10 (Tikanga)

I am not sure if this is helpful, but just in case:
# ldd euc_jis_2004_and_shift_jis_2004.so
   linux-vdso.so.1 = (0x7fff259fd000)
   libc.so.6 = /lib64/libc.so.6 (0x2b3d5a756000)
   /lib64/ld-linux-x86-64.so.2 (0x00384840)

And these libs are softlinks:
/lib64/libc.so.6 - libc-2.5.so
/lib64/ld-linux-x86-64.so.2 - ld-2.5.so

The only thought I have at this point is to run it with strace, but maybe this 
is a known issue and someone has a better idea?

Thank you,
Michael.







This email and any attachments are intended solely for the use of the 
individual or entity to whom it is addressed and may be confidential and/or 
privileged.

If you are not one of the named recipients or have received this email in error,

(i) you should not read, disclose, or copy it,

(ii) please notify sender of your receipt by reply email and delete this email 
and all attachments,

(iii) Dassault Systemes does not accept or assume any liability or 
responsibility for any use of or reliance on this email.

For other languages, go to http://www.3ds.com/terms/email-disclaimer

Re: [HACKERS] initdb failure on RH 5.10

[Tom Lane]

BRUSSER Michael michael.brus...@3ds.com writes:
 initdb failed
 FATAL: unexpected data beyond EOF in block 19 of relation base/1/2609
 HINT: This has been seen to occur with buggy kernels; consider updating your 
 system.
 STATEMENT: COMMENT ON FUNCTION euc_jis_2004_to_shift_jis_2004
   (INTEGER, INTEGER, CSTRING, INTERNAL, INTEGER)
IS 'internal conversion function for EUC_JIS_2004 to 
 SHIFT_JIS_2004';

That's ... surprising.  We've only ever seen that error message in cases
with heavy concurrent updates and wonky underlying storage; initdb is not
where anyone would expect it.

 initdb is called like this:
 initdb -D data-dir -L input-dir -E UTF8 --locale=C

It's not exactly customary to use -L in initdb calls.  Is it possible
you're pointing it to an incompatible library directory?  Not that I see
how that would lead to this behavior, but you're definitely dealing with
something pretty weird.

 This is Postgres 8.4.4, the installation piece has been stable and always 
 worked, but this time they have a new Red Hat 5.10 server

What in the world are they doing using 8.4.4?  The entire 8.4.x release
series is out of support anyway, but there is little if any excuse not
to be using the last minor release, 8.4.22.

I'd call your attention also to the fact that RHEL 5.10 is obsolete.
5.11 came out last month, and Red Hat are not known for updating back-rev
release series with inessential bug fixes.

If you can still reproduce this with 5.11 and 8.4.22, people might be
interested in looking more closely.  Otherwise, well, you're dealing
with five-year-old software with a very long list of known bugs.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[BRUSSER Michael]

On Fri, Dec 10, 2010 at 10:54 AM, Andrew Dunstan and...@dunslane.net wrote:
 Here's my understanding.

 It's not initdb that's really complaining. The timezone files are not inputs
 to initdb. It's the postgres that initdb invokes that's complaining.

 Postges will look for the share file in two places: the configured install
 directory or a share directory whose path is calculated relative to its own 
 location

.initdb's -L flag doesn't override that, it only overrides where
 initdb itself looks for files (such as the BKI file). The bottom line I
 think is that if you intend to use a non-standard layout you need to specify
 the paths for everything and then not move them after installation. If you
 want the installation to be movable, just specify --prefix, but then if you
 move it you have to move the whole thing together. You can't just relocate
 one directory and expect it to work. It won't.

I'm not sure if our current approach would work with v8.4. This is what we do 
in the nutshell:
- build Postgres
- do not run install
- collect all generated libraries, executables and input files and pack them 
along with other app
- distribute the tar-ball to the customer
- untar and install the app
the installation script at some point calls initdb, create database, 
createlang, create user, 
it creates config files... done.

I guess this eliminates option 1 - using the configured install directory

I tried to figure out the second option share directory whose path is 
calculated relative to its own location
and ran initdb with strace. Chances are I overlooked something, but I only 
saw Postgres trying to open the 
timezone files in the original build path.
   
Do you see some way to support remote deployment with the approach I described?
Thanks,
Michael.




This email and any attachments are intended solely for the use of the 
individual or entity to whom it is addressed and may be confidential and/or 
privileged.  If you are not one of the named recipients or have received this 
email in error, (i) you should not read, disclose, or copy it, (ii) please 
notify sender of your receipt by reply email and delete this email and all 
attachments, (iii) Dassault Systemes does not accept or assume any liability or 
responsibility for any use of or reliance on this email.For other languages, go 
to http://www.3ds.com/terms/email-disclaimer.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[Alvaro Herrera]

Excerpts from BRUSSER Michael's message of lun dic 13 12:34:49 -0300 2010:

 I'm not sure if our current approach would work with v8.4. This is what we do 
 in the nutshell:
 - build Postgres
 - do not run install
 - collect all generated libraries, executables and input files and pack them 
 along with other app
 - distribute the tar-ball to the customer
 - untar and install the app
 the installation script at some point calls initdb, create database, 
 createlang, create user, 
 it creates config files... done.

I think this procedure needs to be fixed.  It should be:
- build Postgres
- install
- collect the files, pack
- distribute
- unpack to the destination directory

Since our packaging is (supposed to be) relocatable, you can install in
the customer's machine at a different destination directory, and it
should work.  If it doesn't, report it as a bug.

-- 
Álvaro Herrera alvhe...@commandprompt.com
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[Tom Lane]

BRUSSER Michael michael.brus...@3ds.com writes:
 I'm not sure if our current approach would work with v8.4. This is what we do 
 in the nutshell:
 - build Postgres
 - do not run install
 - collect all generated libraries, executables and input files and pack them 
 along with other app
 - distribute the tar-ball to the customer
 - untar and install the app
 the installation script at some point calls initdb, create database, 
 createlang, create user, 
 it creates config files... done.

If that ever worked, with any version of Postgres, I'd be astonished.

Here is what you should do:

1. Build Postgres with --prefix set to some empty directory.
2. Run install.
3. Bundle up the resulting install tree as part of your tarball.  Do not
editorialize upon the relative locations of its contents.
4. Drop the install tree wherever you want on the target machine.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[BRUSSER Michael]

Here is what you should do:

1. Build Postgres with --prefix set to some empty directory.
2. Run install.
3. Bundle up the resulting install tree as part of your tarball.  Do not
editorialize upon the relative locations of its contents.
4. Drop the install tree wherever you want on the target machine.

-

Would it be a completely crazy idea if I try to modify Postgres to look at some 
env. var 
(similar to PGDATA)  and if exists and path is valid look there for the 
timezone files?

Michael



This email and any attachments are intended solely for the use of the 
individual or entity to whom it is addressed and may be confidential and/or 
privileged.  If you are not one of the named recipients or have received this 
email in error, (i) you should not read, disclose, or copy it, (ii) please 
notify sender of your receipt by reply email and delete this email and all 
attachments, (iii) Dassault Systemes does not accept or assume any liability or 
responsibility for any use of or reliance on this email.For other languages, go 
to http://www.3ds.com/terms/email-disclaimer.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[Andrew Dunstan]

On 12/13/2010 12:16 PM, BRUSSER Michael wrote:

Here is what you should do:

1. Build Postgres with --prefix set to some empty directory.
2. Run install.
3. Bundle up the resulting install tree as part of your tarball.  Do not
editorialize upon the relative locations of its contents.
4. Drop the install tree wherever you want on the target machine.

-

Would it be a completely crazy idea if I try to modify Postgres to look at some 
env. var
(similar to PGDATA)  and if exists and path is valid look there for the 
timezone files?



Yes, that's only the first problem you have encountered. I predict there 
will be others if you go down this path. You are making life much harder 
for yourself than you need to. Just keep everything together and you'll 
be fine. If that's really impossible, use symlinks (or junction points 
on Windows) to achieve the same effect.


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[Tom Lane]

Andrew Dunstan and...@dunslane.net writes:
 On 12/13/2010 12:16 PM, BRUSSER Michael wrote:
 Would it be a completely crazy idea if I try to modify Postgres to look at 
 some env. var
 (similar to PGDATA)  and if exists and path is valid look there for the 
 timezone files?

 Yes, that's only the first problem you have encountered. I predict there 
 will be others if you go down this path.

As far as the timezone data files go, you might have use for the
--with-system-tzdata configure option.  Note that that is not an
environment variable; it requires a hard coded path to the system
copy of the Olsen timezone database.  The expectation is that any
system-maintained copy will have a very well-defined location for
those files on any given platform, so there is no need for runtime
tweaking.  You would use this only on platforms where you expect
that normal system maintenance procedures will keep the TZ files more
up-to-date than the ones included with Postgres.

I agree with Andrew's point in general.  Trying to customize the
relative layout of the various Postgres-installed files is an
exercise in pointless pain.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[BRUSSER Michael]

On 12/09/2010 03:36 PM, BRUSSER Michael wrote:
 No, we do not use --with-system-tzdata  option.
 I looked at the makefile and at the output of pg_config. We may need to do 
 some cleanup there, but I did not pick any clues.
 The problem occurs on all our UNIX platforms.
 Is there anything I could do to shed more light on it?
 I can post the output of pg_config if it helps, or try to run initdb with 
 truss or strace.


Yes, please show us the pg_config (and please don't top-post).
cheers
andrew

---

I tried to build Postgres with minimal options outside of our normal build 
framework to isolate possible interference, but got the same result.

configure \
--prefix=$prefix  \
--without-readline \
--without-zlib  \
--without-tcl  \
--without-perl  \
--disable-rpath  \
21 | tee _configure.log

- gmake, gmake check, gmake install  -- all clean

- initdb:
initdb  -D  /home/local_path/pgdata   -L   /home/local_path/share -E UTF8 
--locale=C  \
 -d  2 log2   log1

log1 ends with line
  creating template1 database in /home/local_path/pgdata/base/1 ...

In log2 I got the same errors as I mentioned earlier:
  LOG:  could not open directory /home/build-path/.../share/timezone: No 
such file or directory 
... ...

Directories timezone and timezonesets  exist in /home/local_path/share,  
pointed to by the -L option
The local share dir has the same contents as the build share dir.

% pg_config 
 BINDIR = /home/build-path/build/Linux/qe_x86_64/bin
 DOCDIR = /home/build-path/build/Linux/qe_x86_64/share/doc
 HTMLDIR = /home/build-path/build/Linux/qe_x86_64/share/doc
 INCLUDEDIR = /home/build-path/build/Linux/qe_x86_64/include
 PKGINCLUDEDIR = /home/build-path/build/Linux/qe_x86_64/include
 INCLUDEDIR-SERVER = /home/build-path/build/Linux/qe_x86_64/include/server
 LIBDIR = /home/build-path/build/Linux/qe_x86_64/lib
 PKGLIBDIR = /home/build-path/build/Linux/qe_x86_64/lib
 LOCALEDIR = /home/build-path/build/Linux/qe_x86_64/share/locale
 MANDIR = /home/build-path/build/Linux/qe_x86_64/share/man
 SHAREDIR = /home/build-path/build/Linux/qe_x86_64/share
 SYSCONFDIR = /home/build-path/build/Linux/qe_x86_64/etc
 PGXS = /home/build-path/build/Linux/qe_x86_64/lib/pgxs/src/makefiles/pgxs.mk
 CONFIGURE = '--prefix=/home/build-path/build/Linux/qe_x86_64' 
'--without-readline' '--without-zlib'
 CC = gcc
 CPPFLAGS = -D_GNU_SOURCE
 CFLAGS = -O2 -Wall -Wmissing-prototypes -Wpointer-arith 
-Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -fwrapv
 CFLAGS_SL = -fpic
 LDFLAGS = -Wl,--as-needed 
-Wl,-rpath,'/home/build-path/build/Linux/qe_x86_64/lib'
 LDFLAGS_SL = 
 LIBS = -lpgport -lcrypt -ldl -lm 
 VERSION = PostgreSQL 8.4.4


This is probably not helpful, but just in case
% strings initdb | grep build-path
 /home/build-path/build/Linux/qe_x86_64/bin
 /home/build-path/build/Linux/qe_x86_64/share/man
 /home/build-path/build/Linux/qe_x86_64/share/doc/
 /home/build-path/build/Linux/qe_x86_64/share/locale
 /home/build-path/build/Linux/qe_x86_64/lib
 /home/build-path/build/Linux/qe_x86_64/include/server
 /home/build-path/build/Linux/qe_x86_64/include
 /home/build-path/build/Linux/qe_x86_64/etc
 /home/build-path/build/Linux/qe_x86_64/share

Thank you,
Michael.




This email and any attachments are intended solely for the use of the 
individual or entity to whom it is addressed and may be confidential and/or 
privileged.  If you are not one of the named recipients or have received this 
email in error, (i) you should not read, disclose, or copy it, (ii) please 
notify sender of your receipt by reply email and delete this email and all 
attachments, (iii) Dassault Systemes does not accept or assume any liability or 
responsibility for any use of or reliance on this email.For other languages, go 
to http://www.3ds.com/terms/email-disclaimer.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[Robert Haas]

On Fri, Dec 10, 2010 at 4:08 AM, BRUSSER Michael
michael.brus...@3ds.com wrote:
 - initdb:
 initdb  -D  /home/local_path/pgdata   -L   /home/local_path/share -E UTF8 
 --locale=C  \
  -d  2 log2   log1

 log1 ends with line
  creating template1 database in /home/local_path/pgdata/base/1 ...

 In log2 I got the same errors as I mentioned earlier:
  LOG:  could not open directory /home/build-path/.../share/timezone: No 
 such file or directory
 ... ...

Does it really have a pathname with three dots here?

I'd have more confidence we knew what we were looking at if you sent
the actual log, rather than an edited version of the lines you think
are most important.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[Andrew Dunstan]

On 12/10/2010 07:32 AM, Robert Haas wrote:

On Fri, Dec 10, 2010 at 4:08 AM, BRUSSER Michael
michael.brus...@3ds.com  wrote:

- initdb:
initdb  -D  /home/local_path/pgdata   -L   /home/local_path/share -E UTF8 
--locale=C  \
  -d  2  log2  log1

log1 ends with line
  creating template1 database in /home/local_path/pgdata/base/1 ...

In log2 I got the same errors as I mentioned earlier:
  LOG:  could not open directory /home/build-path/.../share/timezone: No 
such file or directory
... ...

Does it really have a pathname with three dots here?

I'd have more confidence we knew what we were looking at if you sent
the actual log, rather than an edited version of the lines you think
are most important.



And why the heck use initdb's -L? initdb knows quite well where to get 
its files from, if you haven't mucked up the installation. I don't think 
I've ever seen a case in production where it's been necessary.


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[BRUSSER Michael]

 In log2 I got the same errors as I mentioned earlier:
   LOG:  could not open directory /home/build-path/.../share/timezone: No 
 such file or directory
 ... ...
 Does it really have a pathname with three dots here?

 I'd have more confidence we knew what we were looking at if you sent
 the actual log, rather than an edited version of the lines you think
 are most important.


And why the heck use initdb's -L? initdb knows quite well where to get 
its files from, if you haven't mucked up the installation. I don't think 
I've ever seen a case in production where it's been necessary.

-
1) The pathnames referenced in the log do not have any dots, I just truncated 
them - 
 they are absolute pathnames to where the timezone files are locates in the 
source or build tree
 I'll post the entire log if it is still needed after this ...

2) We have been using 'initdb -L' since Postgres 7.2
The way it works here we build Postgres and it gets bundled as a part of a 
larger application, which can be installed anywhere.
 Perhaps this makes it difficult for initdb to know quite well where to 
get its files from.

 If I remove the -L switch initdb breaks very early before it gets to the 
timezones:  share_path points back to the build location, 
 which is not available, so it cannot locate postgres.bki and other input 
files.

 The timezones piece seems to be the only problem - if I compile with  
--with-system-tzdata=/usr/share/zoneinfo 
 and then use initdb -L  installation succeeds when the source/build tree 
is not accessible.
 
 Not claiming any knowledge in this area - would it be reasonable to expect 
that if -L option works for other input files it should
 also work for timezones?

Thanks for all your help




This email and any attachments are intended solely for the use of the 
individual or entity to whom it is addressed and may be confidential and/or 
privileged.  If you are not one of the named recipients or have received this 
email in error, (i) you should not read, disclose, or copy it, (ii) please 
notify sender of your receipt by reply email and delete this email and all 
attachments, (iii) Dassault Systemes does not accept or assume any liability or 
responsibility for any use of or reliance on this email.For other languages, go 
to http://www.3ds.com/terms/email-disclaimer.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[Robert Haas]

On Fri, Dec 10, 2010 at 9:00 AM, BRUSSER Michael
michael.brus...@3ds.com wrote:
 1) The pathnames referenced in the log do not have any dots, I just truncated 
 them -
     they are absolute pathnames to where the timezone files are locates in 
 the source or build tree
     I'll post the entire log if it is still needed after this ...

Well, why are you configuring the prefix to be inside your build tree?
 This is an odd thing to do.  The prefix should be set to where you
expect to install the files, not where you're building them.

CONFIGURE = '--prefix=/home/build-path/build/Linux/qe_x86_64'
'--without-readline' '--without-zlib'

Still:

     Not claiming any knowledge in this area - would it be reasonable to 
 expect that if -L option works for other input files it should
     also work for timezones?

...this seems reasonable.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[Andrew Dunstan]

On 12/10/2010 09:55 AM, Robert Haas wrote:

On Fri, Dec 10, 2010 at 9:00 AM, BRUSSER Michael
michael.brus...@3ds.com  wrote:

1) The pathnames referenced in the log do not have any dots, I just truncated 
them -
 they are absolute pathnames to where the timezone files are locates in the 
source or build tree
 I'll post the entire log if it is still needed after this ...

Well, why are you configuring the prefix to be inside your build tree?
  This is an odd thing to do.  The prefix should be set to where you
expect to install the files, not where you're building them.

CONFIGURE = '--prefix=/home/build-path/build/Linux/qe_x86_64'
'--without-readline' '--without-zlib'

Still:


 Not claiming any knowledge in this area - would it be reasonable to expect 
that if -L option works for other input files it should
 also work for timezones?

...this seems reasonable.



OK, this has nothing at all to do with the absence of the build path. It 
has to do with using a non-standard sharedir.I have reproduced it thus:


   [and...@aurelia blurflinst]$ mv share/ foo
   [and...@aurelia blurflinst]$ bin/initdb -L `pwd`/foo/postgresql data3
   The files belonging to this database system will be owned by user
   andrew.
   This user must also own the server process.

   The database cluster will be initialized with locale en_US.utf8.
   The default database encoding has accordingly been set to UTF8.
   The default text search configuration will be set to english.

   creating directory data3 ... ok
   creating subdirectories ... ok
   selecting default max_connections ... 10
   selecting default shared_buffers ... 400kB
   creating configuration files ... ok
   creating template1 database in data3/base/1 ... WARNING:  could not
   open directory
   /home/andrew/pgl/blurflinst/share/postgresql/timezonesets: No such
   file or directory
   HINT:  This may indicate an incomplete PostgreSQL installation, or
   that the file /home/andrew/pgl/blurflinst/bin/postgres has been
   moved away from its proper location.
   FATAL:  invalid value for parameter timezone_abbreviations: Default
   child process exited with exit code 1
   initdb: removing data directory data3
   [and...@aurelia blurflinst]$

I will dig a bit further.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[BRUSSER Michael]

Well, why are you configuring the prefix to be inside your build tree?
 This is an odd thing to do.  The prefix should be set to where you
expect to install the files, not where you're building them.

CONFIGURE = '--prefix=/home/build-path/build/Linux/qe_x86_64'
'--without-readline' '--without-zlib'

Still:

 Not claiming any knowledge in this area - would it be reasonable to 
 expect that if -L option works for other input files it should
 also work for timezones?

...this seems reasonable.

-

It is odd, but this only reflects some quick hack I did to have a test case 
outside of our standard build framework.
I wanted to make sure the odd problem with timezones is not coming from our 
overloaded build environment.
Production makefile does it much better.

If I don't find a better option - do you see any disadvantage in  using 
--with-system-tzdata  and OS timezone info on Solaris and Linux?
  


This email and any attachments are intended solely for the use of the 
individual or entity to whom it is addressed and may be confidential and/or 
privileged.  If you are not one of the named recipients or have received this 
email in error, (i) you should not read, disclose, or copy it, (ii) please 
notify sender of your receipt by reply email and delete this email and all 
attachments, (iii) Dassault Systemes does not accept or assume any liability or 
responsibility for any use of or reliance on this email.For other languages, go 
to http://www.3ds.com/terms/email-disclaimer.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[Andrew Dunstan]

On 12/10/2010 10:25 AM, Andrew Dunstan wrote:





 Not claiming any knowledge in this area - would it be 
reasonable to expect that if -L option works for other input files 
it should

 also work for timezones?

...this seems reasonable.



OK, this has nothing at all to do with the absence of the build path. 
It has to do with using a non-standard sharedir.I have reproduced it 
thus:





[snip]


I will dig a bit further.




Here's my understanding.

It's not initdb that's really complaining. The timezone files are not 
inputs to initdb. It's the postgres that initdb invokes that's complaining.


Postges will look for the share file in two places: the configured 
install directory or a share directory whose path is calculated relative 
to its own location. initdb's -L flag doesn't override that, it only 
overrides where initdb itself looks for files (such as the BKI file). 
The bottom line I think is that if you intend to use a non-standard 
layout you need to specify the paths for everything and then not move 
them after installation. If you want the installation to be movable, 
just specify --prefix, but then if you move it you have to move the 
whole thing together. You can't just relocate one directory and expect 
it to work. It won't.


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[Robert Haas]

On Fri, Dec 10, 2010 at 10:54 AM, Andrew Dunstan and...@dunslane.net wrote:
 Here's my understanding.

 It's not initdb that's really complaining. The timezone files are not inputs
 to initdb. It's the postgres that initdb invokes that's complaining.

That was my impression, too, from the log that was sent.

 Postges will look for the share file in two places: the configured install
 directory or a share directory whose path is calculated relative to its own
 location. initdb's -L flag doesn't override that, it only overrides where
 initdb itself looks for files (such as the BKI file). The bottom line I
 think is that if you intend to use a non-standard layout you need to specify
 the paths for everything and then not move them after installation. If you
 want the installation to be movable, just specify --prefix, but then if you
 move it you have to move the whole thing together. You can't just relocate
 one directory and expect it to work. It won't.

So in theory we could have a GUC under file locations to override
this, similarly to data_directory or hba_file or ident_file.  But
since it's been like this for a really long time (I think), I wouldn't
be inclined to go monkeying with it unless more than one person
complains.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[Tom Lane]

Robert Haas robertmh...@gmail.com writes:
 So in theory we could have a GUC under file locations to override
 this, similarly to data_directory or hba_file or ident_file.  But
 since it's been like this for a really long time (I think), I wouldn't
 be inclined to go monkeying with it unless more than one person
 complains.

Even if we did see more than one request, I'd be inclined not to change
it.  The installation layout stuff is too complicated and fragile
already; and AFAICS it does cover every *reasonable* use case.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[Andrew Dunstan]

On 12/10/2010 11:19 AM, Tom Lane wrote:

Robert Haasrobertmh...@gmail.com  writes:

So in theory we could have a GUC under file locations to override
this, similarly to data_directory or hba_file or ident_file.  But
since it's been like this for a really long time (I think), I wouldn't
be inclined to go monkeying with it unless more than one person
complains.

Even if we did see more than one request, I'd be inclined not to change
it.  The installation layout stuff is too complicated and fragile
already; and AFAICS it does cover every *reasonable* use case.




I agree, although I think Robert's really long time is in fact since 
8.0, before which we didnt provide for relocation at all.


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[BRUSSER Michael]

On 12/10/2010 11:19 AM, Tom Lane wrote:
 Robert Haasrobertmh...@gmail.com  writes:
 So in theory we could have a GUC under file locations to override
 this, similarly to data_directory or hba_file or ident_file.  But
 since it's been like this for a really long time (I think), I wouldn't
 be inclined to go monkeying with it unless more than one person
 complains.
 Even if we did see more than one request, I'd be inclined not to change
 it.  The installation layout stuff is too complicated and fragile
 already; and AFAICS it does cover every *reasonable* use case.

   

I agree, although I think Robert's really long time is in fact since 
8.0, before which we didnt provide for relocation at all.

cheers
andrew
--

Thanks, everyone!!!
I'm still not sure I completely understand how to package all components to 
allow independent installation,
but I guess this is what this weekend is good for  :(

Michael




This email and any attachments are intended solely for the use of the 
individual or entity to whom it is addressed and may be confidential and/or 
privileged.  If you are not one of the named recipients or have received this 
email in error, (i) you should not read, disclose, or copy it, (ii) please 
notify sender of your receipt by reply email and delete this email and all 
attachments, (iii) Dassault Systemes does not accept or assume any liability or 
responsibility for any use of or reliance on this email.For other languages, go 
to http://www.3ds.com/terms/email-disclaimer.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] initdb failure with Postgres 8.4.4

[BRUSSER Michael]

Initdb fails for me when host machine has no access to the Postgres build 
location.

LOG:  could not open directory .../install/share/timezone: No such file or 
directory
LOG:  could not open directory .../install/share/timezone: No such file or 
directory
WARNING:  could not open directory .../install/share/timezonesets: No such 
file or directory
HINT:  This may indicate an incomplete PostgreSQL installation,
   or that the file .../bin.Linux/postgres has been moved away from its 
proper location.
FATAL:  invalid value for parameter timezone_abbreviations: Default
DEBUG:  shmem_exit(1): 0 callbacks to make
DEBUG:  proc_exit(1): 0 callbacks to make
DEBUG:  exit(1)

The postgres executable is in the proper place.
The truncated paths .../install/share/timezone, etc point to the original build 
location.

The timezone and other files  are available in the local directory:
share % ls -1
conversion_create.sql
information_schema.sql
pg_hba.conf.sample
pg_ident.conf.sample
pg_service.conf.sample
postgres.bki
postgres.description
postgresql.conf.sample
postgres.shdescription
psqlrc.sample
recovery.conf.sample
snowball_create.sql
sql_features.txt
system_views.sql
timezone
timezonesets
tsearch_data

I call initdb with the -L option pointing to that directory, from initdb log I 
can see that it is indeed being used for certain things -

POSTGRES_BKI=.../share/postgres.bki
POSTGRES_DESCR=.../share/postgres.description
POSTGRES_SHDESCR=.../share/postgres.shdescription
POSTGRESQL_CONF_SAMPLE=.../share/postgresql.conf.sample
PG_HBA_SAMPLE=.../share/pg_hba.conf.sample
PG_IDENT_SAMPLE=.../share/pg_ident.conf.sample

I am wondering why the timezone files cannot be found there as well?
This is quite urgent and any help will be greatly appreciated.
Michael.
.




This email and any attachments are intended solely for the use of the 
individual or entity to whom it is addressed and may be confidential and/or 
privileged.  If you are not one of the named recipients or have received this 
email in error, (i) you should not read, disclose, or copy it, (ii) please 
notify sender of your receipt by reply email and delete this email and all 
attachments, (iii) Dassault Systemes does not accept or assume any liability or 
responsibility for any use of or reliance on this email.For other languages, go 
to http://www.3ds.com/terms/email-disclaimer.

Re: [HACKERS] initdb failure with Postgres 8.4.4

[Tom Lane]

BRUSSER Michael michael.brus...@3ds.com writes:
 Initdb fails for me when host machine has no access to the Postgres build 
 location.

 LOG:  could not open directory .../install/share/timezone: No such file or 
 directory

Moving the install tree works for me.  Did you do something odd with the
--with-system-tzdata configuration option?

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[BRUSSER Michael]

No, we do not use --with-system-tzdata  option.
I looked at the makefile and at the output of pg_config. We may need to do some 
cleanup there, but I did not pick any clues. 
The problem occurs on all our UNIX platforms.
Is there anything I could do to shed more light on it? 
I can post the output of pg_config if it helps, or try to run initdb with truss 
or strace.

Thanks,
Michael.


-Original Message-
From: Tom Lane [mailto:t...@sss.pgh.pa.us] 
Sent: Thursday, December 09, 2010 2:56 PM
To: BRUSSER Michael
Cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] initdb failure with Postgres 8.4.4 

BRUSSER Michael michael.brus...@3ds.com writes:
 Initdb fails for me when host machine has no access to the Postgres build 
 location.

 LOG:  could not open directory .../install/share/timezone: No such file or 
 directory

Moving the install tree works for me.  Did you do something odd with the
--with-system-tzdata configuration option?

regards, tom lane


This email and any attachments are intended solely for the use of the 
individual or entity to whom it is addressed and may be confidential and/or 
privileged.  If you are not one of the named recipients or have received this 
email in error, (i) you should not read, disclose, or copy it, (ii) please 
notify sender of your receipt by reply email and delete this email and all 
attachments, (iii) Dassault Systemes does not accept or assume any liability or 
responsibility for any use of or reliance on this email.For other languages, go 
to http://www.3ds.com/terms/email-disclaimer.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure with Postgres 8.4.4

[Andrew Dunstan]

On 12/09/2010 03:36 PM, BRUSSER Michael wrote:

No, we do not use --with-system-tzdata  option.
I looked at the makefile and at the output of pg_config. We may need to do some 
cleanup there, but I did not pick any clues.
The problem occurs on all our UNIX platforms.
Is there anything I could do to shed more light on it?
I can post the output of pg_config if it helps, or try to run initdb with truss 
or strace.



Yes, please show us the pg_config (and please don't top-post).

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] initdb failure on win32

[Magnus Hagander]

  There is a fixed buffer of 1024 bytes. I have a feeling 
 ythat one has 
  to be changed to dynamic.
  
 You're quite right. The 'InfoBufferSize' denotes 1122 bytes 
 after this call.
 
 if(!GetTokenInformation(AccessToken,TokenGroups,InfoBuffer,102
 4,InfoBufferSize)) 

Ah. 
Your stop-gap fix is to double the size of the buffer.

I'll try to get a patch in for dynamic buffer handling in the next
couple of days as the real fix.

Just to be sure: Bruce, open item please for Use dynamic buffer for
token buffer in win32 admin check


//Magnus

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] initdb failure on win32

[Bruce Momjian]

Magnus Hagander wrote:
   There is a fixed buffer of 1024 bytes. I have a feeling 
  ythat one has 
   to be changed to dynamic.
   
  You're quite right. The 'InfoBufferSize' denotes 1122 bytes 
  after this call.
  
  if(!GetTokenInformation(AccessToken,TokenGroups,InfoBuffer,102
  4,InfoBufferSize)) 
 
 Ah. 
 Your stop-gap fix is to double the size of the buffer.
 
 I'll try to get a patch in for dynamic buffer handling in the next
 couple of days as the real fix.
 
 Just to be sure: Bruce, open item please for Use dynamic buffer for
 token buffer in win32 admin check

OK, text updated.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

[HACKERS] initdb failure on win32

[Thomas Hallgren]

I'm trying to run initdb on win32, freshly compiled from CVS HEAD, but I 
get the following error:

creating template1 database in data/base/1 ... failed to get token 
information: 122

Any ideas?
Regards,
Thomas Hallgren
---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] initdb failure on win32

[Magnus Hagander]

Yes, one :-)

Is your account by any chance a member of a large number of groups? If
so, how many?

There is a fixed buffer of 1024 bytes. I have a feeling ythat one has to
be changed to dynamic.

//Magnus

-Original Message-
From: Thomas Hallgren [mailto:[EMAIL PROTECTED] 
Sent: den 17 augusti 2004 20:27
To: [EMAIL PROTECTED]
Subject: [HACKERS] initdb failure on win32


I'm trying to run initdb on win32, freshly compiled from CVS 
HEAD, but I 
get the following error:

creating template1 database in data/base/1 ... failed to get token 
information: 122

Any ideas?

Regards,

Thomas Hallgren


---(end of 
broadcast)---
TIP 4: Don't 'kill -9' the postmaster


---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] initdb failure on win32

[Thomas Hallgren]

Magnus,
There is a fixed buffer of 1024 bytes. I have a feeling ythat one has to
be changed to dynamic.
You're quite right. The 'InfoBufferSize' denotes 1122 bytes after this call.
if(!GetTokenInformation(AccessToken,TokenGroups,InfoBuffer,1024,InfoBufferSize)) 

Regards,
Thomas Hallgren
---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
 joining column's datatypes do not match

Re: [HACKERS] initdb failure in CVS

[Andrew Dunstan]

Tom Lane said:
 Andrew Dunstan [EMAIL PROTECTED] writes:
 Did we actually find a current system where it broke with a straight
 putenv(LC_ALL)?

 Well, the Single Unix Spec does not say that that works, and neither
 does the HPUX man page for putenv, so you're going to have a hard time
 convincing me that it's a portable solution.

Yeah, you're right. In fact SUS seems to be quite brain-dead on this - I
can't see that it says anything about how to unset an enviroment value.
How strange.


 I think the real story is
 simply that no one has tested the C version of initdb hard enough to
 notice whether that line actually accomplished anything or not.



I tested on the platforms that were available to me.

OTOH, I wonder if maybe what we previously did w.r.t. TZ didn't work, but
the timezone libs were smart enough to disregard an empty TZ value.

Anyway, you seem to have found a portable solution, so all's well that
ends well.

cheers

andrew



---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] initdb failure in CVS

[Tom Lane]

Andrew Dunstan [EMAIL PROTECTED] writes:
 OTOH, I wonder if maybe what we previously did w.r.t. TZ didn't work, but
 the timezone libs were smart enough to disregard an empty TZ value.

The code in variable.c clearly does *not* work on machines where
putenv() copies what you hand it :-(.  I suspect that no one noticed
because the clear_tz path isn't really used in normal situations.

I left it alone for the moment because I expect that we'll be ripping
it all out as soon as the dust settles with src/timezone.  We can define
a less brain-dead API for testing validity of a timezone name ...

regards, tom lane

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

[HACKERS] initdb failure in CVS

[Bruce Momjian]

I am seeing the following failure of initdb in CVS:

The files belonging to this database system will be owned by user postgres.
This user must also own the server process.

The database cluster will be initialized with locale C.

creating directory /u/pg/data ... ok
creating directory /u/pg/data/global ... ok
creating directory /u/pg/data/pg_xlog ... ok
creating directory /u/pg/data/pg_clog ... ok
creating directory /u/pg/data/base ... ok
creating directory /u/pg/data/base/1 ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 1000
creating configuration files ... ok

creating template1 database in /u/pg/data/base/1 ... 

   FATAL:  invalid value for parameter client_encoding: 

initdb: child process exited with exit code 1
initdb: failed
initdb: removing data directory /u/pg/data

The problem seems to be related to a commit made to initdb a few days
ago.

revision 1.24
date: 2004/05/05 16:09:31;  author: tgl;  state: Exp;  lines: +23 -2
Use a more portable technique for unsetting environment variables,
and unset PGCLIENTENCODING to prevent backend from dying if it's set
to something incompatible with the -E option.

I don't have any encoding set in my system.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] initdb failure in CVS

[Tom Lane]

Bruce Momjian [EMAIL PROTECTED] writes:
 I am seeing the following failure of initdb in CVS:
  FATAL:  invalid value for parameter client_encoding: 

Hmm.  Apparently the pg_unsetenv routine I put into initdb.c doesn't
work on your platform.  Which is odd, because we've used exactly the
same technique to unset TZ in variable.c for years and years, and not
had any reports of trouble.

We might have to put configure to work to figure out how to do unsetenv
properly.  Any thoughts?

regards, tom lane

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] initdb failure in CVS

[Rod Taylor]

On Wed, 2004-05-05 at 13:48, Bruce Momjian wrote:
 I am seeing the following failure of initdb in CVS:
 
  FATAL:  invalid value for parameter client_encoding: 

I get the same thing.


---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] initdb failure in CVS

[Andrew Dunstan]

Bruce Momjian wrote:
I am seeing the following failure of initdb in CVS:

The files belonging to this database system will be owned by user postgres.
This user must also own the server process.

The database cluster will be initialized with locale C.

creating directory /u/pg/data ... ok
creating directory /u/pg/data/global ... ok
creating directory /u/pg/data/pg_xlog ... ok
creating directory /u/pg/data/pg_clog ... ok
creating directory /u/pg/data/base ... ok
creating directory /u/pg/data/base/1 ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 1000
creating configuration files ... ok
	creating template1 database in /u/pg/data/base/1 ... 

   FATAL:  invalid value for parameter client_encoding: 
initdb: child process exited with exit code 1
initdb: failed
initdb: removing data directory /u/pg/data
The problem seems to be related to a commit made to initdb a few days
ago.
revision 1.24
date: 2004/05/05 16:09:31;  author: tgl;  state: Exp;  lines: +23 -2
Use a more portable technique for unsetting environment variables,
and unset PGCLIENTENCODING to prevent backend from dying if it's set
to something incompatible with the -E option.
I don't have any encoding set in my system.
 

The change is based on this code from here (backend/commands/variable.c):
/*
* unsetenv() works fine, but is BSD, not POSIX, and is not available
* under Solaris, among others. Apparently putenv() called as below
* clears the process-specific environment variables. Other
* reasonable arguments to putenv() (e.g. TZ=, TZ, ) result in a
* core dump (under Linux anyway). - thomas 1998-01-26
*/
if (tzbuf[0] == 'T')
{
strcpy(tzbuf, =);
if (putenv(tzbuf) != 0)
elog(LOG, could not clear TZ environment variable);
tzset();
}
The Linux man page for putenv says this:
Description for libc4, libc5, glibc: If the argument string is of the
form name, and does not contain an = character, then the variable
name is removed from the environment. If putenv() has to allocate a
new array environ, and the previous array was also allocated by
putenv(), then it will be freed. In no case will the old storage asso-
ciated to the environment variable itself be freed.
The libc4 and libc5 and glibc 2.1.2 versions conform to SUSv2: the
pointer string given to putenv() is used. In particular, this string
becomes part of the environment; changing it later will change the
environment. (Thus, it is an error is to call putenv() with an auto-
matic variable as the argument, then return from the calling function
while string is still part of the environment.) However, glibc
2.0-2.1.1 differs: a copy of the string is used. On the one hand this
causes a memory leak, and on the other hand it violates SUSv2. This has
been fixed in glibc2.1.2.
The BSD4.4 version, like glibc 2.0, uses a copy.
I suspect you have seen this latter effect, i.e. it in effect did
putenv(PGCLIENTENCODING=);
putenv(=);
leaving you with an empty string as the env value rather than unsetting it.
Did we actually find a current system where it broke with a straight 
putenv(LC_ALL)?

cheers
andrew


---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
  http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] initdb failure in CVS

[Tom Lane]

Bruce Momjian [EMAIL PROTECTED] writes:
 I am seeing the following failure of initdb in CVS:

Okay, I did some more work on handling unsetenv cleanly ... give it
another try.

regards, tom lane

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] initdb failure in CVS

[Bruce Momjian]

Thanks, works.

---

Tom Lane wrote:
 Bruce Momjian [EMAIL PROTECTED] writes:
  I am seeing the following failure of initdb in CVS:
 
 Okay, I did some more work on handling unsetenv cleanly ... give it
 another try.
 
   regards, tom lane
 
 ---(end of broadcast)---
 TIP 7: don't forget to increase your free space map settings
 

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] initdb failure in CVS

[Tom Lane]

Andrew Dunstan [EMAIL PROTECTED] writes:
 Did we actually find a current system where it broke with a straight 
 putenv(LC_ALL)?

Well, the Single Unix Spec does not say that that works, and neither
does the HPUX man page for putenv, so you're going to have a hard time
convincing me that it's a portable solution.  I think the real story is
simply that no one has tested the C version of initdb hard enough to
notice whether that line actually accomplished anything or not.

regards, tom lane

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] initdb failure

[Christopher Browne]

[EMAIL PROTECTED] (Bruce Momjian) writes:
 Peter Eisentraut wrote:
 Bruce Momjian writes:
 
  The argument that you want a warning because you might have mixed
  newlines in the file seems less likely than this case where they are
  using a literal carriage return as a data value at the end of the line.
 
 I don't agree with that assessment.  Who actually has CRs in their data?

 Uh, people storing Win32 text documents.

Or people storing XML data that may contain all sorts of whitespace
that may include various inter-line metadata.
-- 
let name=cbbrowne and tld=libertyrms.info in String.concat @ [name;tld];;
http://dev6.int.libertyrms.com/
Christopher Browne
(416) 646 3304 x124 (land)

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Peter Eisentraut]

Jon Jensen writes:

 I hate to mention it, but would it be useful/non-overkill to make either
 of those things (context message maximum length and funny character
 escaping) configurable somehow?

I'm thinking the same.  We have conflicting needs of different groups:
Users on real Unix platforms converting old data need the checks, users
on Windows platforms with new data don't want them.  (I wonder what people
on Mac OS that mix Unix-style and native applications would want.)

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

Bruce Momjian wrote:
 Peter Eisentraut wrote:
  Tom Lane writes:
  
   so it appears that cygwin's echo generates a different newline style
   than what got put into sql_features.txt.  A possible way to fix this is
   to put the \. line into sql_features.txt, but maybe there's a cleaner
   answer.  Peter, any thoughts?
  
  There's no clean answer to this on Cygwin.  This specific case is just a
  little problem that we could solve locally, but in general you'll just end
  up annoying people if you require them to use consistent line endings on
  Cygwin.
 
 At this point, we have one mixed-EOL case (initdb, fixable), and one \r
 in data case (from 7.2).  Seems the safest solution is to fix initdb and
 see what other failure reports we get.
 
 If we downgrade it to a warning, we will not know about the failures;
 someone could miss a warning in a dump, but probably will not mess an
 error and an empty table.  If we get more reports like initdb, we can
 reevaluate, but it seems the safest course is to keep our existing code,
 which only removes carriage returns when it quite sure.

OK, I have applied a patch to make initdb create a COPY file with
consistent end-of-line termination, so it passes the COPY EOL checking.

If we find more cases where load files have intended inconsistent EOL
termination, we can revisit the COPY error for inconsistent EOL
termination.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

Peter Eisentraut wrote:
 Oliver Elphick writes:
 
  +para
  + With a large dump, it may be difficult to identify where any errors are
  + occurring.  You may use the -e option to psql to print the SQL commands
  + as they are run, so that it is easy to see precisely which commands are
  + causing errors.
   /para
 
 That is just not true.  If you use -f, it will tell you the line number of
 the command causing the error.  Add the line number of the COPY error
 message, there you have it.

You are assuming it is easy to find what is on a specific line of the
dump file.  I am not sure that is always easy for people with limited
Unix skills, or MSWin folks.  I am not sure I would have thought to add
the file offset to find the problem COPY line.  I guess I would have
eventually, but it wouldn't have been my first idea, and I might _not_
have used -f on the load, and if the load took an hour, I would have to
run it again to get that line number.

I can see the point that the table name is only really valuable when you
are loading a dump, and not valuabvle when you are just doing a copy. 
However, copy is used enought in dumps that the exta word (the table
name) doesn't see to hurt.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Peter Eisentraut]

Bruce Momjian writes:

 You are assuming it is easy to find what is on a specific line of the
 dump file.  I am not sure that is always easy for people with limited
 Unix skills, or MSWin folks.  I am not sure I would have thought to add
 the file offset to find the problem COPY line.  I guess I would have
 eventually, but it wouldn't have been my first idea, and I might _not_
 have used -f on the load, and if the load took an hour, I would have to
 run it again to get that line number.

That is all besides the point.  If adding -f to the command line is for
some reason prohibitive, then the same applies to -e.  That is all.

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Peter Eisentraut]

Bruce Momjian writes:

 OK, I have applied a patch to make initdb create a COPY file with
 consistent end-of-line termination, so it passes the COPY EOL checking.

Maybe instead we can change the COPY command to read the file directly and
not via stdin?  Then we don't need the \. marker.

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

Peter Eisentraut wrote:
 Bruce Momjian writes:
 
  You are assuming it is easy to find what is on a specific line of the
  dump file.  I am not sure that is always easy for people with limited
  Unix skills, or MSWin folks.  I am not sure I would have thought to add
  the file offset to find the problem COPY line.  I guess I would have
  eventually, but it wouldn't have been my first idea, and I might _not_
  have used -f on the load, and if the load took an hour, I would have to
  run it again to get that line number.
 
 That is all besides the point.  If adding -f to the command line is for
 some reason prohibitive, then the same applies to -e.  That is all.

I see, both -e give query before error, -f gives line number before
error.  I suppose the -e is clearer because you don't have to find the
line in the file, but the -e output makes it more likely they would miss
an error line in the output.

Seems we should recommend -f rather than  for restores anyway, right?

Reporting the table with the error is clearer, but this brings up
another case --- what happens with pg_dumpall?  Do we print the database
name or will they know the database name from the table name?

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

Peter Eisentraut wrote:
 Bruce Momjian writes:
 
  OK, I have applied a patch to make initdb create a COPY file with
  consistent end-of-line termination, so it passes the COPY EOL checking.
 
 Maybe instead we can change the COPY command to read the file directly and
 not via stdin?  Then we don't need the \. marker.

Yes, we need a temp table then.  The single-line fix seemed easier.

Also, the file is a mix of things from a file and echo's, it isn't just
the \.:


  echo UPDATE information_schema.sql_implementation_info SET character_value 
= '$combined_version' WHERE implementation_info_name = 'DBMS VERSION';

  echo COPY information_schema.sql_features (feature_id, feature_name, 
sub_feature_id, sub_feature_name, is_supported, comments) FROM STDIN;
  cat $datadir/sql_features.txt
  echo \.
) |
tr -d '\r' | # make newlines consistent for Win32

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Alvaro Herrera]

On Sat, Sep 27, 2003 at 01:08:37PM -0400, Bruce Momjian wrote:
 Peter Eisentraut wrote:

  That is all besides the point.  If adding -f to the command line is for
  some reason prohibitive, then the same applies to -e.  That is all.
 
 Seems we should recommend -f rather than  for restores anyway, right?

Please keep in mind that it is not always possible to use -f.  In my
case I had a file larger than 2GB and psql was compiled without large
file support.  Detecting an error in that situation would have required
recompiling psql.

Also I don't see the point in not extending the context message of the
error.  It's not like it's going to take too much processing power, nor
screen estate, so what is it?

-- 
Alvaro Herrera (alvherre[a]dcc.uchile.cl)
Right now the sectors on the hard disk run clockwise, but I heard a rumor that
you can squeeze 0.2% more throughput by running them counterclockwise.
It's worth the effort. Recommended.  (Gerry Pourwelle)

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Tom Lane]

Bruce Momjian [EMAIL PROTECTED] writes:
 Peter Eisentraut wrote:
 Maybe instead we can change the COPY command to read the file directly and
 not via stdin?  Then we don't need the \. marker.

I like that idea too.

 Yes, we need a temp table then.  The single-line fix seemed easier.

Why a temp table?  The COPY command is only sourcing the contents of
$datadir/sql_features.txt.

regards, tom lane

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Tom Lane]

Peter Eisentraut [EMAIL PROTECTED] writes:
 Jon Jensen writes:
 I hate to mention it, but would it be useful/non-overkill to make either
 of those things (context message maximum length and funny character
 escaping) configurable somehow?

 I'm thinking the same.  We have conflicting needs of different groups:
 Users on real Unix platforms converting old data need the checks, users
 on Windows platforms with new data don't want them.  (I wonder what people
 on Mac OS that mix Unix-style and native applications would want.)

I wouldn't object to adding a GUC variable that turns off the
mixed-newlines error (as long as it defaults to enabled ;-)).

Our intention to remove the check in the future might then reduce
to simply flipping the factory default for this variable.

regards, tom lane

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

Tom Lane wrote:
 Bruce Momjian [EMAIL PROTECTED] writes:
  Peter Eisentraut wrote:
  Maybe instead we can change the COPY command to read the file directly and
  not via stdin?  Then we don't need the \. marker.
 
 I like that idea too.
 
  Yes, we need a temp table then.  The single-line fix seemed easier.
 
 Why a temp table?  The COPY command is only sourcing the contents of
 $datadir/sql_features.txt.

Sorry, I meant temp file.  I have SQL on the brain.  :-)

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

[HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Tom Lane]

Tim McAuley [EMAIL PROTECTED] writes:
 Another question now. I am unable to compile Postgresql 7.4 beta 3 under 
 cygwin (Windows 2K, using cgyipc 2).

 I am getting the error:
 
 creating information schema... ERROR:  end-of-copy marker does not match 
 previous newline style
 CONTEXT:  COPY FROM, line 361
 

That's interesting.  COPY is complaining because the \. terminator in
the file it's been fed has a different kind of newline after it than
the newlines earlier in the file (LF vs CR/LF, no doubt).  The part
of the initdb script that must be causing this is

  echo COPY information_schema.sql_features (feature_id, feature_name, 
sub_feature_id, sub_feature_name, is_supported, comments) FROM STDIN;
  cat $datadir/sql_features.txt
  echo \.
) \
| $PGPATH/postgres $PGSQL_OPT template1  /dev/null || exit_nicely
echo ok

so it appears that cygwin's echo generates a different newline style
than what got put into sql_features.txt.  A possible way to fix this is
to put the \. line into sql_features.txt, but maybe there's a cleaner
answer.  Peter, any thoughts?

regards, tom lane

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Matthew T. O'Connor]

On Fri, 2003-09-26 at 11:01, Tom Lane wrote:
 so it appears that cygwin's echo generates a different newline style
 than what got put into sql_features.txt.  A possible way to fix this is
 to put the \. line into sql_features.txt, but maybe there's a cleaner
 answer.  Peter, any thoughts?

Does cygwin still have the install time option of what type of line feed
to use?  I know at one point (a long time ago) when I installed cygwin,
and chose windows line feeds (CRLF) that it caused problems with several
applications.  So the problem might be that with CYGWIN you could have
either type of line feed depending on what the user selected during
install.


---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Michael Meskes]

On Fri, Sep 26, 2003 at 11:01:34AM -0400, Tom Lane wrote:
  I am getting the error:
  
  creating information schema... ERROR:  end-of-copy marker does not match 
  previous newline style
  CONTEXT:  COPY FROM, line 361
  
 
 That's interesting.  COPY is complaining because the \. terminator in
 the file it's been fed has a different kind of newline after it than
 the newlines earlier in the file (LF vs CR/LF, no doubt).  The part

Well actually I'm not so sure. I tried installing 7.4 from Oliver's
Debian packages and experienced exactly the same. We have yet to find
out why, but I doubt newlines changed between his Debian system and mine.

 of the initdb script that must be causing this is
 
   echo COPY information_schema.sql_features (feature_id, feature_name, 
 sub_feature_id, sub_feature_name, is_supported, comments) FROM STDIN;
   cat $datadir/sql_features.txt
   echo \.
 ) \
   | $PGPATH/postgres $PGSQL_OPT template1  /dev/null || exit_nicely
 echo ok
 
 so it appears that cygwin's echo generates a different newline style
 than what got put into sql_features.txt.  A possible way to fix this is
 to put the \. line into sql_features.txt, but maybe there's a cleaner
 answer.  Peter, any thoughts?

Putting the \. line into sql_features.txt did help me, too. 

Michael
-- 
Michael Meskes
Email: Michael at Fam-Meskes dot De
ICQ: 179140304, AIM/Yahoo: michaelmeskes, Jabber: [EMAIL PROTECTED]
Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL!

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Peter Eisentraut]

Tom Lane writes:

 so it appears that cygwin's echo generates a different newline style
 than what got put into sql_features.txt.  A possible way to fix this is
 to put the \. line into sql_features.txt, but maybe there's a cleaner
 answer.  Peter, any thoughts?

There's no clean answer to this on Cygwin.  This specific case is just a
little problem that we could solve locally, but in general you'll just end
up annoying people if you require them to use consistent line endings on
Cygwin.

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

Peter Eisentraut wrote:
 Tom Lane writes:
 
  so it appears that cygwin's echo generates a different newline style
  than what got put into sql_features.txt.  A possible way to fix this is
  to put the \. line into sql_features.txt, but maybe there's a cleaner
  answer.  Peter, any thoughts?
 
 There's no clean answer to this on Cygwin.  This specific case is just a
 little problem that we could solve locally, but in general you'll just end
 up annoying people if you require them to use consistent line endings on
 Cygwin.

This error is coming from the new 7.4 COPY code that allows \r\n as a
line terminator.  Requiring the end-of-line to be consistent seemed to
be the only way to be sure we were not eating a literal carriage return
in the data stream.  Let's put the \. into sql_features.txt and see if
that fixes it, or can we use echo -c \.\n in initdb?

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Tom Lane]

Peter Eisentraut [EMAIL PROTECTED] writes:
 Tom Lane writes:
 so it appears that cygwin's echo generates a different newline style
 than what got put into sql_features.txt.  A possible way to fix this is
 to put the \. line into sql_features.txt, but maybe there's a cleaner
 answer.  Peter, any thoughts?

 There's no clean answer to this on Cygwin.  This specific case is just a
 little problem that we could solve locally, but in general you'll just end
 up annoying people if you require them to use consistent line endings on
 Cygwin.

Yeah, I was wondering whether you wouldn't propose dropping the newline
consistency check.  I'm not very comfortable with that, but maybe we
should.  Bruce?

regards, tom lane

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

Tom Lane wrote:
 Peter Eisentraut [EMAIL PROTECTED] writes:
  Tom Lane writes:
  so it appears that cygwin's echo generates a different newline style
  than what got put into sql_features.txt.  A possible way to fix this is
  to put the \. line into sql_features.txt, but maybe there's a cleaner
  answer.  Peter, any thoughts?
 
  There's no clean answer to this on Cygwin.  This specific case is just a
  little problem that we could solve locally, but in general you'll just end
  up annoying people if you require them to use consistent line endings on
  Cygwin.
 
 Yeah, I was wondering whether you wouldn't propose dropping the newline
 consistency check.  I'm not very comfortable with that, but maybe we
 should.  Bruce?

I posted on that a few minutes ago.  Yea, we can drop it, but we risk
eating carraige returns as data values.  I am not sure how consistently
we output literal carriage returns in old dumps, nor how many apps
produce on literal carriage returns in COPY. If we conditionally eat
them, we run the risk of discarding some of their data without warning. 
Perhaps we can throw a warning rather than an error, and adjust initdb
to be consistent.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Andrew Dunstan]

Bruce Momjian wrote:

I posted on that a few minutes ago.  Yea, we can drop it, but we risk
eating carraige returns as data values.  I am not sure how consistently
we output literal carriage returns in old dumps, nor how many apps
produce on literal carriage returns in COPY. If we conditionally eat
them, we run the risk of discarding some of their data without warning. 
Perhaps we can throw a warning rather than an error, and adjust initdb
to be consistent.

 

Would the best longterm solution be to require escaping CR in a data 
value? (Yes I know this might cause backwards compatibility problems, at 
least for a while).

cheers

andrew

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Tom Lane]

Bruce Momjian [EMAIL PROTECTED] writes:
 Perhaps we can throw a warning rather than an error, and adjust initdb
 to be consistent.

I like the idea of reducing the newline consistency check to a warning.
There is one thing we'd have to watch for though (it's already an issue
but would become a bigger one): client-side COPY code had better be
prepared to absorb backend Notice messages while processing COPY IN.
Currently libpq doesn't read input data at all during a COPY IN loop,
which means that if the COPY generates more than a few K of warning
messages, the backend gets blocked on a full pipe and the whole
operation locks up.  I have been meaning to fix that in libpq anyway,
but what other client libraries might have the same issue?  Anyone know
whether JDBC would need a similar fix?

regards, tom lane

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

Andrew Dunstan wrote:
 Bruce Momjian wrote:
 
 I posted on that a few minutes ago.  Yea, we can drop it, but we risk
 eating carraige returns as data values.  I am not sure how consistently
 we output literal carriage returns in old dumps, nor how many apps
 produce on literal carriage returns in COPY. If we conditionally eat
 them, we run the risk of discarding some of their data without warning. 
 Perhaps we can throw a warning rather than an error, and adjust initdb
 to be consistent.
 
   
 
 
 Would the best longterm solution be to require escaping CR in a data 
 value? (Yes I know this might cause backwards compatibility problems, at 
 least for a while).

Yes, we do have that documented, but we supported it in earlier
releases, so we can't be sure who is using it, and it probably exists in
older dumps.

The particular problem is not literal carriage returns, which we don't
support any more, but carriage returns that happen to be at the end of
the line, right up against \n.  If we relax this, we will conditionally
strip off the \r.

In fact, we can do that now:

create table xx(text);

Right now, if all the column value end with \r, we will silently eat it.
But if some have \r and some do not, we will throw an error.  (Above,
when I say \r, I mean literal carriage return, not the \r string,
which we have always handled cleanly.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

Tom Lane wrote:
 Bruce Momjian [EMAIL PROTECTED] writes:
  Perhaps we can throw a warning rather than an error, and adjust initdb
  to be consistent.
 
 I like the idea of reducing the newline consistency check to a warning.
 There is one thing we'd have to watch for though (it's already an issue
 but would become a bigger one): client-side COPY code had better be
 prepared to absorb backend Notice messages while processing COPY IN.
 Currently libpq doesn't read input data at all during a COPY IN loop,
 which means that if the COPY generates more than a few K of warning
 messages, the backend gets blocked on a full pipe and the whole
 operation locks up.  I have been meaning to fix that in libpq anyway,
 but what other client libraries might have the same issue?  Anyone know
 whether JDBC would need a similar fix?

Wow, that sounds big.  The ERROR will only happen once, while the
WARNING could happen a lot --- we could add code to throw the WARNING
only once per COPY command --- that would probably make sense.  I don't
see how we could get all clients to handle this for 7.4, particularly
clients from previous releases.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

Peter Eisentraut wrote:
 Tom Lane writes:
 
  so it appears that cygwin's echo generates a different newline style
  than what got put into sql_features.txt.  A possible way to fix this is
  to put the \. line into sql_features.txt, but maybe there's a cleaner
  answer.  Peter, any thoughts?
 
 There's no clean answer to this on Cygwin.  This specific case is just a
 little problem that we could solve locally, but in general you'll just end
 up annoying people if you require them to use consistent line endings on
 Cygwin.

Here is a little diff to make initdb behave if we decide to keep the
COPY check.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073
Index: src/bin/initdb/initdb.sh
===
RCS file: /cvsroot/pgsql-server/src/bin/initdb/initdb.sh,v
retrieving revision 1.201
diff -c -c -r1.201 initdb.sh
*** src/bin/initdb/initdb.sh7 Sep 2003 03:36:03 -   1.201
--- src/bin/initdb/initdb.sh26 Sep 2003 18:23:16 -
***
*** 1087,1094 
echo COPY information_schema.sql_features (feature_id, feature_name, 
sub_feature_id, sub_feature_name, is_supported, comments) FROM STDIN;
cat $datadir/sql_features.txt
echo \.
! ) \
!   | $PGPATH/postgres $PGSQL_OPT template1  /dev/null || exit_nicely
  echo ok
  
  $ECHO_N vacuuming database template1... $ECHO_C
--- 1087,1095 
echo COPY information_schema.sql_features (feature_id, feature_name, 
sub_feature_id, sub_feature_name, is_supported, comments) FROM STDIN;
cat $datadir/sql_features.txt
echo \.
! ) |
! tr -d '\r' | # make newlines consistent for Win32
! $PGPATH/postgres $PGSQL_OPT template1  /dev/null || exit_nicely
  echo ok
  
  $ECHO_N vacuuming database template1... $ECHO_C

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Tom Lane]

Bruce Momjian [EMAIL PROTECTED] writes:
 ... we could add code to throw the WARNING
 only once per COPY command --- that would probably make sense.

Seems like a bit of a kluge, but perhaps the best compromise.  It would
be quite likely that you'd get the same warning on many lines of a COPY,
and that probably isn't really going to help people.

regards, tom lane

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Peter Eisentraut]

Tom Lane writes:

 Yeah, I was wondering whether you wouldn't propose dropping the newline
 consistency check.  I'm not very comfortable with that, but maybe we
 should.  Bruce?

I don't mind if we keep it on pure-POSIX platforms.  But one of the nicer
developments on Windows in recent(?) times is that you can actually use
any kind of line separator and most programs still work correctly (with
the notable exception of Notepad).  The drawback of this is that you never
really know what kind of line separator a program is actually going to
write, especially when you start mixing Cygwin, MinGW, and native.  So
this newline consistency check is not only going to be pretty annoying on
Windows, it's going to cancel a feature of the operating system
environment.

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

Tom Lane wrote:
 Bruce Momjian [EMAIL PROTECTED] writes:
  ... we could add code to throw the WARNING
  only once per COPY command --- that would probably make sense.
 
 Seems like a bit of a kluge, but perhaps the best compromise.  It would
 be quite likely that you'd get the same warning on many lines of a COPY,
 and that probably isn't really going to help people.

I can't think of any cases where we throw _tons_ of warnings at people,
like we could with COPY.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[scott.marlowe]

On Fri, 26 Sep 2003, Bruce Momjian wrote:

 Tom Lane wrote:
  Peter Eisentraut [EMAIL PROTECTED] writes:
   Tom Lane writes:
   so it appears that cygwin's echo generates a different newline style
   than what got put into sql_features.txt.  A possible way to fix this is
   to put the \. line into sql_features.txt, but maybe there's a cleaner
   answer.  Peter, any thoughts?
  
   There's no clean answer to this on Cygwin.  This specific case is just a
   little problem that we could solve locally, but in general you'll just end
   up annoying people if you require them to use consistent line endings on
   Cygwin.
  
  Yeah, I was wondering whether you wouldn't propose dropping the newline
  consistency check.  I'm not very comfortable with that, but maybe we
  should.  Bruce?
 
 I posted on that a few minutes ago.  Yea, we can drop it, but we risk
 eating carraige returns as data values.  I am not sure how consistently
 we output literal carriage returns in old dumps, nor how many apps
 produce on literal carriage returns in COPY. If we conditionally eat
 them, we run the risk of discarding some of their data without warning. 
 Perhaps we can throw a warning rather than an error, and adjust initdb
 to be consistent.

I'm running into issues where 7.4's pg_dump/pg_dumpall from a 7.2 database 
to a 7.4beta3 database is producing some errors like this:

ERROR:  literal newline found in data
HINT:  Use \n to represent newline.
CONTEXT:  COPY FROM, line 59

ERROR:  literal carriage return found in data
HINT:  Use \r to represent carriage return.
CONTEXT:  COPY FROM, line 41


These show up with little or no context, only the line number of the 
dump file.  Since I'm wrapping these up in pg_dumpall, I don't have the 
dump file so I don't know where the error is really occuring.  It would be 
nice to have such occurances echo the table / row they are getting the 
error on, or maybe just the first 20 or so characters, so they'd be easier 
to identify.

Is this related to this issue?


---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Tom Lane]

scott.marlowe [EMAIL PROTECTED] writes:
 I'm running into issues where 7.4's pg_dump/pg_dumpall from a 7.2 database 
 to a 7.4beta3 database is producing some errors like this:

 ERROR:  literal newline found in data
 HINT:  Use \n to represent newline.
 CONTEXT:  COPY FROM, line 59

 ERROR:  literal carriage return found in data
 HINT:  Use \r to represent carriage return.
 CONTEXT:  COPY FROM, line 41

Really?  7.2 should dump data \r or \n as the backslash versions ...
and does in my tests.  Can you make a reproducible test case?



 It would be nice to have such occurances echo the table / row they are
 getting the error on, or maybe just the first 20 or so characters, so
 they'd be easier to identify.

That's not a bad idea.  I think it would be fairly easy now for the
CONTEXT line of the error message to include the input data line:

CONTEXT:  COPY FROM, line 41: data here 

at least up through the field where the error gets thrown, and with some
limit on the length of the data that will get echoed.  If people like
that idea I'll see about making it happen.

regards, tom lane

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Tom Lane]

Peter Eisentraut [EMAIL PROTECTED] writes:
 Tom Lane writes:
 Yeah, I was wondering whether you wouldn't propose dropping the newline
 consistency check.  I'm not very comfortable with that, but maybe we
 should.  Bruce?

 I don't mind if we keep it on pure-POSIX platforms.  But one of the nicer
 developments on Windows in recent(?) times is that you can actually use
 any kind of line separator and most programs still work correctly (with
 the notable exception of Notepad).

Not sure if we should make the behavior Windows-specific though.  And
didn't Michael report seeing the same initdb failure on Debian?  That
confuses me a bit --- why would there be a newline discrepancy on Debian?

regards, tom lane

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

Tom Lane wrote:
  ERROR:  literal carriage return found in data
  HINT:  Use \r to represent carriage return.
  CONTEXT:  COPY FROM, line 41
 
 Really?  7.2 should dump data \r or \n as the backslash versions ...
 and does in my tests.  Can you make a reproducible test case?
 
 
 
  It would be nice to have such occurances echo the table / row they are
  getting the error on, or maybe just the first 20 or so characters, so
  they'd be easier to identify.
 
 That's not a bad idea.  I think it would be fairly easy now for the
 CONTEXT line of the error message to include the input data line:
 
   CONTEXT:  COPY FROM, line 41: data here 
 
 at least up through the field where the error gets thrown, and with some
 limit on the length of the data that will get echoed.  If people like
 that idea I'll see about making it happen.

Also, he wanted table name too.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] initdb failure

[Andrew Dunstan]

Tom Lane wrote:

I don't mind if we keep it on pure-POSIX platforms.  But one of the nicer
developments on Windows in recent(?) times is that you can actually use
any kind of line separator and most programs still work correctly (with
the notable exception of Notepad).
   

Not sure if we should make the behavior Windows-specific though.  And
didn't Michael report seeing the same initdb failure on Debian?  That
confuses me a bit --- why would there be a newline discrepancy on Debian?
 

Is this affected by the fact that the platform where a dump is made 
might have a different line-end discipline from the one where the dump 
is restored?

cheers

andrew



---(end of broadcast)---
TIP 6: Have you searched our list archives?
  http://archives.postgresql.org

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[scott.marlowe]

On Fri, 26 Sep 2003, Tom Lane wrote:

 scott.marlowe [EMAIL PROTECTED] writes:
  I'm running into issues where 7.4's pg_dump/pg_dumpall from a 7.2 database 
  to a 7.4beta3 database is producing some errors like this:
 
  ERROR:  literal newline found in data
  HINT:  Use \n to represent newline.
  CONTEXT:  COPY FROM, line 59
 
  ERROR:  literal carriage return found in data
  HINT:  Use \r to represent carriage return.
  CONTEXT:  COPY FROM, line 41
 
 Really?  7.2 should dump data \r or \n as the backslash versions ...
 and does in my tests.  Can you make a reproducible test case?

The attached file produces this problem.  Note it's a blank trailing field 
that looks to be causing it.  The error for this .sql file is:

ERROR:  literal carriage return found in data
HINT:  Use \r to represent carriage return.
CONTEXT:  COPY FROM, line 2

Note that loading this into pico and saving it back out fixes the problem.

If I remove the preceding row that doesn't end in a blank field, I get a 
different error, this one:

ERROR:  end-of-copy marker does not match previous newline style
CONTEXT:  COPY FROM, line 2

  It would be nice to have such occurances echo the table / row they are
  getting the error on, or maybe just the first 20 or so characters, so
  they'd be easier to identify.
 
 That's not a bad idea.  I think it would be fairly easy now for the
 CONTEXT line of the error message to include the input data line:
 
   CONTEXT:  COPY FROM, line 41: data here 
 
 at least up through the field where the error gets thrown, and with some
 limit on the length of the data that will get echoed.  If people like
 that idea I'll see about making it happen.

table name too, like Bruce said.  The bothersome bit is that in pg_dump, 
it says the line, relative to just this part of the copy command, so you 
don't even know which table is giving the error.
--
-- PostgreSQL database dump
--

\connect - marl8412

--
-- TOC entry 2 (OID 283043147)
-- Name: people2; Type: TABLE; Schema: ; Owner: marl8412
--

CREATE TABLE people2 (
id integer,
persons text
);


--
-- Data for TOC entry 3 (OID 283043147)
-- Name: people2; Type: TABLE DATA; Schema: ; Owner: marl8412
--

COPY people2 (id, persons) FROM stdin;
59  Chance Terry--S
60  

\.



---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

scott.marlowe wrote:
 The attached file produces this problem.  Note it's a blank trailing field 
 that looks to be causing it.  The error for this .sql file is:
 
 ERROR:  literal carriage return found in data
 HINT:  Use \r to represent carriage return.
 CONTEXT:  COPY FROM, line 2
 
 Note that loading this into pico and saving it back out fixes the problem.
 
 If I remove the preceding row that doesn't end in a blank field, I get a 
 different error, this one:
 
 ERROR:  end-of-copy marker does not match previous newline style
 CONTEXT:  COPY FROM, line 2

OK, 'vi' shows it as:

COPY people2 (id, persons) FROM stdin;
59  Chance Terry--S
60  ^M
\.

which is _exactly the case the error was supposed to catch.  Now, the
big question is where did this dump come from?  Pg version?  OS platform?

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

Bruce Momjian wrote:
 scott.marlowe wrote:
  The attached file produces this problem.  Note it's a blank trailing field 
  that looks to be causing it.  The error for this .sql file is:
  
  ERROR:  literal carriage return found in data
  HINT:  Use \r to represent carriage return.
  CONTEXT:  COPY FROM, line 2
  
  Note that loading this into pico and saving it back out fixes the problem.
  
  If I remove the preceding row that doesn't end in a blank field, I get a 
  different error, this one:
  
  ERROR:  end-of-copy marker does not match previous newline style
  CONTEXT:  COPY FROM, line 2
 
 OK, 'vi' shows it as:
   
   COPY people2 (id, persons) FROM stdin;
   59  Chance Terry--S
   60  ^M
   \.

The significant issue here is that we don't want to throw a warning in
this case;  it should be an error --- this is clearly a data value we
don't want to discard as part of the end-of-line.

The argument that you want a warning because you might have mixed
newlines in the file seems less likely than this case where they are
using a literal carriage return as a data value at the end of the line.

Seems we should just fix initdb as my patch does and see if we get any
more mixed-newline cases or literal carriage-return data cases.  My
guess is that we will get more of the latter.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[scott.marlowe]

On Fri, 26 Sep 2003, Bruce Momjian wrote:

 scott.marlowe wrote:
  The attached file produces this problem.  Note it's a blank trailing field 
  that looks to be causing it.  The error for this .sql file is:
  
  ERROR:  literal carriage return found in data
  HINT:  Use \r to represent carriage return.
  CONTEXT:  COPY FROM, line 2
  
  Note that loading this into pico and saving it back out fixes the problem.
  
  If I remove the preceding row that doesn't end in a blank field, I get a 
  different error, this one:
  
  ERROR:  end-of-copy marker does not match previous newline style
  CONTEXT:  COPY FROM, line 2
 
 OK, 'vi' shows it as:
   
   COPY people2 (id, persons) FROM stdin;
   59  Chance Terry--S
   60  ^M
   \.
 
 which is _exactly the case the error was supposed to catch.  Now, the
 big question is where did this dump come from?  Pg version?  OS platform?

The originating system is a RedHat 7.2 box with postgresql 7.2.x running 
on it.

The destination system is a RedHat 7.2 box with postgresql 7.4 beta3 
running on it.

The data likely came out of a (gasp, horrors) windows box.


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

scott.marlowe wrote:
  OK, 'vi' shows it as:
  
  COPY people2 (id, persons) FROM stdin;
  59  Chance Terry--S
  60  ^M
  \.
  
  which is _exactly the case the error was supposed to catch.  Now, the
  big question is where did this dump come from?  Pg version?  OS platform?
 
 The originating system is a RedHat 7.2 box with postgresql 7.2.x running 
 on it.
 
 The destination system is a RedHat 7.2 box with postgresql 7.4 beta3 
 running on it.
 
 The data likely came out of a (gasp, horrors) windows box.

OK, try this on your 7.2:

test= create table test(x text);
CREATE TABLE
test= insert into test values ('\r');
INSERT 17158 1
test= copy test to '/tmp/out';
COPY

Then 'vi' /tmp/out. It should show \r, not ^M.  Please report back.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Michael Meskes]

On Fri, Sep 26, 2003 at 03:05:58PM -0400, Tom Lane wrote:
 Not sure if we should make the behavior Windows-specific though.  And
 didn't Michael report seeing the same initdb failure on Debian?  That
 confuses me a bit --- why would there be a newline discrepancy on Debian?

I take it there are no special characters in that COPY, right? The only
difference between the two machines we found was that my was running
under UTF-8. 

Michael
-- 
Michael Meskes
Email: Michael at Fam-Meskes dot De
ICQ: 179140304, AIM/Yahoo: michaelmeskes, Jabber: [EMAIL PROTECTED]
Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL!

---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[scott.marlowe]

On Fri, 26 Sep 2003, Bruce Momjian wrote:

 scott.marlowe wrote:
   OK, 'vi' shows it as:
 
 COPY people2 (id, persons) FROM stdin;
 59  Chance Terry--S
 60  ^M
 \.
   
   which is _exactly the case the error was supposed to catch.  Now, the
   big question is where did this dump come from?  Pg version?  OS platform?
  
  The originating system is a RedHat 7.2 box with postgresql 7.2.x running 
  on it.
  
  The destination system is a RedHat 7.2 box with postgresql 7.4 beta3 
  running on it.
  
  The data likely came out of a (gasp, horrors) windows box.
 
 OK, try this on your 7.2:
 
   test= create table test(x text);
   CREATE TABLE
   test= insert into test values ('\r');
   INSERT 17158 1
   test= copy test to '/tmp/out';
   COPY
 
 Then 'vi' /tmp/out. It should show \r, not ^M.  Please report back.

I'm not much of a vi guy, so the out file thing didn't tell me much, but 
if I try and dump it from the 7.4 beta3 box into a new table, I get:

ERROR:  end-of-copy marker does not match previous newline style
CONTEXT:  COPY FROM, line 2



---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[scott.marlowe]

On Fri, 26 Sep 2003, Bruce Momjian wrote:

 scott.marlowe wrote:
   OK, 'vi' shows it as:
 
 COPY people2 (id, persons) FROM stdin;
 59  Chance Terry--S
 60  ^M
 \.
   
   which is _exactly the case the error was supposed to catch.  Now, the
   big question is where did this dump come from?  Pg version?  OS platform?
  
  The originating system is a RedHat 7.2 box with postgresql 7.2.x running 
  on it.
  
  The destination system is a RedHat 7.2 box with postgresql 7.4 beta3 
  running on it.
  
  The data likely came out of a (gasp, horrors) windows box.
 
 OK, try this on your 7.2:
 
   test= create table test(x text);
   CREATE TABLE
   test= insert into test values ('\r');
   INSERT 17158 1
   test= copy test to '/tmp/out';
   COPY
 
 Then 'vi' /tmp/out. It should show \r, not ^M.  Please report back.

Figured out iv (-b switch, man pages rock) and it's a ^M



---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Peter Eisentraut]

scott.marlowe writes:

 table name too, like Bruce said.  The bothersome bit is that in pg_dump,
 it says the line, relative to just this part of the copy command, so you
 don't even know which table is giving the error.

I don't see the problem.  Can't you identify the failing command by the
line number that psql gives you?

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Peter Eisentraut]

Bruce Momjian writes:

 The argument that you want a warning because you might have mixed
 newlines in the file seems less likely than this case where they are
 using a literal carriage return as a data value at the end of the line.

I don't agree with that assessment.  Who actually has CRs in their data?

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[scott.marlowe]

On Fri, 26 Sep 2003, Peter Eisentraut wrote:

 Bruce Momjian writes:
 
  The argument that you want a warning because you might have mixed
  newlines in the file seems less likely than this case where they are
  using a literal carriage return as a data value at the end of the line.
 
 I don't agree with that assessment.  Who actually has CRs in their data?

I do.  And so do lots of other people.


---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Peter Eisentraut]

scott.marlowe writes:

 but I get basically the same thing if I dump it to a .sql file and do:

 psql dbname dbname.sql

Use psql -f dbname.sql instead.

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Larry Rosenman]

--On Friday, September 26, 2003 14:36:08 -0600 scott.marlowe 
[EMAIL PROTECTED] wrote:

On Fri, 26 Sep 2003, Peter Eisentraut wrote:

Bruce Momjian writes:

 The argument that you want a warning because you might have mixed
 newlines in the file seems less likely than this case where they are
 using a literal carriage return as a data value at the end of the line.
I don't agree with that assessment.  Who actually has CRs in their data?
I do.  And so do lots of other people.
I store E-Mail type comments in some of my DB's and they may contain cr's.

Don't ASSUME that we don't do stuff like this with a database.



--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


pgp0.pgp
Description: PGP signature

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Tom Lane]

Peter Eisentraut [EMAIL PROTECTED] writes:
 scott.marlowe writes:
 but I get basically the same thing if I dump it to a .sql file and do:
 psql dbname dbname.sql

 Use psql -f dbname.sql instead.

This doesn't seem like a good argument not to add more information to
the CONTEXT line for COPY errors.  Sure, in theory the existing info
should be sufficient, but what if the information is not coming in
through psql?  (For instance, maybe the COPY data is being generated
on-the-fly by some other program.)  Or what if the dump file is so large
you can't easily edit it to determine which line number is in question?
There are plenty of scenarios where it's not all that convenient to
triangulate on a problem from outside information.  Minimalism isn't
really a virtue in error reports anyway.

I'm thinking maybe:

CONTEXT:  COPY tablename, line 41: data ...

would serve the purpose nicely.

regards, tom lane

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[scott.marlowe]

On Fri, 26 Sep 2003, Peter Eisentraut wrote:

 scott.marlowe writes:
 
  but I get basically the same thing if I dump it to a .sql file and do:
 
  psql dbname dbname.sql
 
 Use psql -f dbname.sql instead.

and the output is:

psql:webport.sql:803: ERROR:  function odbc_user already exists with 
same argument types
REVOKE
REVOKE
GRANT
You are now connected as new user ayousuff.
psql:webport.sql:869: ERROR:  literal newline found in data
HINT:  Use \n to represent newline.
CONTEXT:  COPY FROM, line 59
You are now connected as new user smarlowe.
You are now connected as new user ayousuff.
CREATE INDEX
CREATE INDEX
CREATE INDEX


the same.  It doesn't tell me which table in my dump caused the problem, 
and it certainly isn't line 59 of the dump file, but of the table 
producing the error.


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[scott.marlowe]

On Fri, 26 Sep 2003, Tom Lane wrote:

 Peter Eisentraut [EMAIL PROTECTED] writes:
  scott.marlowe writes:
  but I get basically the same thing if I dump it to a .sql file and do:
  psql dbname dbname.sql
 
  Use psql -f dbname.sql instead.
 
 This doesn't seem like a good argument not to add more information to
 the CONTEXT line for COPY errors.  Sure, in theory the existing info
 should be sufficient, but what if the information is not coming in
 through psql?  (For instance, maybe the COPY data is being generated
 on-the-fly by some other program.)  Or what if the dump file is so large
 you can't easily edit it to determine which line number is in question?
 There are plenty of scenarios where it's not all that convenient to
 triangulate on a problem from outside information.  Minimalism isn't
 really a virtue in error reports anyway.
 
 I'm thinking maybe:
 
   CONTEXT:  COPY tablename, line 41: data ...
 
 would serve the purpose nicely.

Yeah, just having the table name and line number would be plenty for me.  
It's the lack of a table name that makes it so frustrating.  I had to 
basically dump / restore the tables one at a time to figure out which one 
was causing the error.  On a database with hundreds of tables, that could 
be painful.


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[scott.marlowe]

On Fri, 26 Sep 2003, Peter Eisentraut wrote:

 scott.marlowe writes:
 
  table name too, like Bruce said.  The bothersome bit is that in pg_dump,
  it says the line, relative to just this part of the copy command, so you
  don't even know which table is giving the error.
 
 I don't see the problem.  Can't you identify the failing command by the
 line number that psql gives you?

OK, here's the output from pg_dump while it's running

ERROR:  function odbc_user already exists with same argument types
REVOKE
REVOKE
GRANT
You are now connected as new user ayousuff.
ERROR:  literal newline found in data
HINT:  Use \n to represent newline.
CONTEXT:  COPY FROM, line 59
You are now connected as new user smarlowe.
You are now connected as new user ayousuff.
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
You are now connected as new user smarlowe.
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX

So how am I supposed to figure out which table and which row broke?  Keep 
in mind, I'm doing this:

pg_dump -h otherserver dbname|psql -h desthost dbname

but I get basically the same thing if I dump it to a .sql file and do:

psql dbname dbname.sql




---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Tom Lane]

Peter Eisentraut [EMAIL PROTECTED] writes:
 Bruce Momjian writes:
 The argument that you want a warning because you might have mixed
 newlines in the file seems less likely than this case where they are
 using a literal carriage return as a data value at the end of the line.

 I don't agree with that assessment.  Who actually has CRs in their data?

The question isn't so much who has CRs in their data as who is trying
to import data files in which CRs aren't correctly represented as \r ?
Not anyone upgrading from a recent PG release ... though 7.1 or before
would have an issue.

If we eliminate the mixed-newline check then we will silently discard
CRs that appear at the tail end of the last column of a data line.  This
is somewhat worrisome because the failure will not be obvious.  But in
most cases (CR in the middle of a data field, or in a field that isn't
last), we will detect and report an error due to field count mismatch.
I think that that might be good enough.

If we were designing COPY from scratch today, we'd certainly not think
that a mixed-newline check was appropriate; it's only because of our
historical handling of literal CRs that the question comes up at all.

regards, tom lane

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Peter Eisentraut]

scott.marlowe writes:

 psql:webport.sql:803: ERROR:  function odbc_user already exists with
 same argument types
 REVOKE
 REVOKE
 GRANT
 You are now connected as new user ayousuff.
 psql:webport.sql:869: ERROR:  literal newline found in data
 HINT:  Use \n to represent newline.
 CONTEXT:  COPY FROM, line 59
 You are now connected as new user smarlowe.
 You are now connected as new user ayousuff.
 CREATE INDEX
 CREATE INDEX
 CREATE INDEX


 the same.  It doesn't tell me which table in my dump caused the problem,
 and it certainly isn't line 59 of the dump file, but of the table
 producing the error.

The table is the one the COPY command on line 869 in file webport.sql
mentions and the problem data is on line 928 of that file.

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Tom Lane]

I said:
 The question isn't so much who has CRs in their data as who is trying
 to import data files in which CRs aren't correctly represented as \r ?
 Not anyone upgrading from a recent PG release ... though 7.1 or before
 would have an issue.

Actually, checking the CVS logs shows that 7.2.1 was the first release
that would emit a data CR or LF as \r or \n rather than literally.  If
Scott is dumping from an original 7.2 installation rather than a dot-
release, that would explain his report.

I talked to Bruce about this on the phone, and we now both feel that
that change was too recent to assume that people won't be trying to load
dumps containing bare CRs into 7.4.  Accordingly, it seems that we do
need to throw an error or at least a warning for mixed newlines.
Eventually we can get rid of that behavior and just treat all newline
sequences alike, but I think we can't do it until 7.2.0 is in the
ancient history category.

regards, tom lane

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Peter Eisentraut]

Tom Lane writes:

 This doesn't seem like a good argument not to add more information to
 the CONTEXT line for COPY errors.  Sure, in theory the existing info
 should be sufficient, but what if the information is not coming in
 through psql?  (For instance, maybe the COPY data is being generated
 on-the-fly by some other program.)

Then you look into the code of that other program.  Or you look into the
server log, where you should log the statements you generate if you are
testing your code.

 Or what if the dump file is so large you can't easily edit it to
 determine which line number is in question?

The line number is already contained in the available information about
the error.  If the file is too large to load into an editor, you could use

perl -npi -e '$. == 123456 and s/\r/\\r/;'

or something along those lines.

 There are plenty of scenarios where it's not all that convenient to
 triangulate on a problem from outside information.

Maybe, but the ones I've seen mentioned so far are not among them.  I'm
not opposed to making errors more easy to identify, but considering that
someone else in this thread didn't even know about psql's option to print
line numbers of errors, I think some people haven't done their homework
yet.

 Minimalism isn't really a virtue in error reports anyway.

 I'm thinking maybe:

   CONTEXT:  COPY tablename, line 41: data ...

 would serve the purpose nicely.

The only thing that would really help in the general case is the number of
the character where the error occurs.  If you print the actual data you
lose if the data is repeated within the, er, data and/or if the section of
the data that you print contains crazy characters that mess up the
display.

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

Peter Eisentraut wrote:
 Bruce Momjian writes:
 
  The argument that you want a warning because you might have mixed
  newlines in the file seems less likely than this case where they are
  using a literal carriage return as a data value at the end of the line.
 
 I don't agree with that assessment.  Who actually has CRs in their data?

Uh, people storing Win32 text documents.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Bruce Momjian]

Peter Eisentraut wrote:
 scott.marlowe writes:
 
  table name too, like Bruce said.  The bothersome bit is that in pg_dump,
  it says the line, relative to just this part of the copy command, so you
  don't even know which table is giving the error.
 
 I don't see the problem.  Can't you identify the failing command by the
 line number that psql gives you?

If you are loading from pg_dump, you have lots of copy commands, so how
do you know which COPY command caused the failure.  You just have the
line number of _a_ copy.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] initdb failure (was Re: [GENERAL] sequence's plpgsql)

[Tom Lane]

Peter Eisentraut [EMAIL PROTECTED] writes:
 Minimalism isn't really a virtue in error reports anyway.
 I'm thinking maybe:
 CONTEXT:  COPY tablename, line 41: data ...
 would serve the purpose nicely.

 The only thing that would really help in the general case is the number of
 the character where the error occurs.  If you print the actual data you
 lose if the data is repeated within the, er, data and/or if the section of
 the data that you print contains crazy characters that mess up the
 display.

A messed-up display would be useful information in itself; and it's not
like we have no precedent for repeating the erroneous data in error
messages.  Consider pg_atoi:

regression=# create table foo (f1 int);
CREATE TABLE
regression=# copy foo from stdin;
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself.
 z44
 \.
ERROR:  invalid input syntax for integer: z44
CONTEXT:  COPY FROM, line 1
regression=#

I was chewing this over with Bruce on the phone just now, and we refined
the idea a little.  Some errors (primarily those detected inside the
datatype input procedures) can be clearly traced to a specific column,
whereas others (such as too many fields on an input line) really can't
be nailed down more tightly than a line.  So we were thinking about two
different flavors of context info:

CONTEXT:  COPY tablename, line n, field colname: column data

versus

CONTEXT:  COPY tablename, line n: line data

where in each case the data display would be truncated at 100 characters
or so (and would have to be omitted in the COPY BINARY case anyway).

If you're really concerned about funny characters messing up the report,
we could imagine replacing them by backslash sequences or some such, but
I suspect that would create more confusion than it would solve.

regards, tom lane

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org