Re: [HACKERS] pg_dump and large files - is this a problem?

2002-11-04 Thread Bruce Momjian
Does this resolve our AIX compile problem? --- Tom Lane wrote: > "Zeugswetter Andreas SB SD" <[EMAIL PROTECTED]> writes: > > The issue is, that you need to remove the #include "bootstrap_tokens.h" > > line from the lex file.

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-11-04 Thread Tom Lane
"Zeugswetter Andreas SB SD" <[EMAIL PROTECTED]> writes: > The issue is, that you need to remove the #include "bootstrap_tokens.h" > line from the lex file. Good point; I'm surprised gcc doesn't spit up on that. I've made that mod and also added the inclusion-order-correction in pqsignal.c.

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-11-04 Thread Zeugswetter Andreas SB SD
Tom Lane writes: > > I think the problem is more accurately described thus: Flex generated > > files include before "postgres.h" due to the way it lays out the > > code in the output. stdio.h does something which prevents switching to > > the large file model later on in postgres.h. (This mani

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-11-01 Thread Tom Lane
Peter Eisentraut <[EMAIL PROTECTED]> writes: > I think the problem is more accurately described thus: Flex generated > files include before "postgres.h" due to the way it lays out the > code in the output. stdio.h does something which prevents switching to > the large file model later on in post

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-31 Thread Tom Lane
Peter Eisentraut <[EMAIL PROTECTED]> writes: > The proposed fix was to include the flex output in some other file (such > as the corresponding grammar file) rather than to compile it separately. Seems like a reasonable solution. Can you make that happen in the next day or two? If not, I'll take

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-29 Thread Peter Eisentraut
Zeugswetter Andreas SB SD writes: > > AIX is too stupid to wrap unistd.h in an "#ifndef" to protect against > > double inclusion? I suppose we could do that for them... > > I guess that is exactly not wanted, since that would hide the actual > problem, namely that _LARGE_FILE_API gets defined (of

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-29 Thread Zeugswetter Andreas SB SD
> >> Yeah. AFAICS the only way around this is to avoid doing any I/O > >> operations in the flex-generated files. Fortunately, > that's not much > >> of a restriction. > > > Unfortunately I do not think that is sufficient, since the problem is already > > at the #include level. The compiler ba

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-29 Thread Tom Lane
"Zeugswetter Andreas SB SD" <[EMAIL PROTECTED]> writes: >> Yeah. AFAICS the only way around this is to avoid doing any I/O >> operations in the flex-generated files. Fortunately, that's not much >> of a restriction. > Unfortunately I do not think that is sufficient, since the problem is already

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-29 Thread Zeugswetter Andreas SB SD
> > The problem with flex is, that the generated c file does #include > > before we #include "postgres.h". > > In this situation _LARGE_FILES is not defined for unistd.h and unistd.h > > chooses to define _LARGE_FILE_API, those two are not compatible. > > Yeah. AFAICS the only way around this i

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-28 Thread Tom Lane
"Zeugswetter Andreas SB SD" <[EMAIL PROTECTED]> writes: > The problem with flex is, that the generated c file does #include > before we #include "postgres.h". > In this situation _LARGE_FILES is not defined for unistd.h and unistd.h > chooses to define _LARGE_FILE_API, those two are not compatible

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-28 Thread Zeugswetter Andreas SB SD
> > > The question is *which* seek APIs we need to support. Are there any > > > besides fseeko() and fgetpos()? > > > > On AIX we have > > int fseeko64 (FILE* Stream, off64_t Offset, int Whence); > > which is intended for large file access for programs that do NOT > > #define _LARGE_FILES > >

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-24 Thread Philip Warner
At 11:51 PM 24/10/2002 -0400, Bruce Momjian wrote: Your idea of using SEEK_SET is good, except I was concerned that the checkSeek call will move the file pointer. Is that OK? It doesn't seem appropriate. The call is made just after the file is opened (or it should be!), so SEEK_SET, 0 will no

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-24 Thread Philip Warner
At 12:07 AM 25/10/2002 -0400, Bruce Momjian wrote: I don't think we can assume that off_t can be passed to fset/getpos unless we know the platform supports it, unless people think fpos_t being integral and the same size as fpos_t is enough. We don't need to. We would #define FILE_OFFSET as fpos_

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-24 Thread Bruce Momjian
Philip Warner wrote: > Rather than having a different patch file for each platform and refusing to > code fseek/tell because we can't do SEEK_CUR, why not check for FSEEKO64 > and revert to a simple solution: > > #ifdef HAVE_FSEEKO64 > #define FSEEK fseeko64 > #define FTELL ftello64 > #define FI

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-24 Thread Bruce Momjian
Philip Warner wrote: > > I just reread the patch; is it valid to assume fseek and fseeko have the > same failure modes? Or does the call to 'fseek' actually call fseeko? The fseek was a typo. It should have been fseeko as you suggested. CVS updated. Your idea of using SEEK_SET is good, except

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-24 Thread Bruce Momjian
OK, finally figured it out. I had used fseek instead of fseeko. --- Philip Warner wrote: > At 09:56 PM 24/10/2002 -0400, Bruce Momjian wrote: > > > > > You are quite correct. It should read: > > > > > > > > > > #ifd

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-24 Thread Philip Warner
I just reread the patch; is it valid to assume fseek and fseeko have the same failure modes? Or does the call to 'fseek' actually call fseeko? Philip Warner| __---_ Albatross Consulting Pty. Ltd. |--

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-24 Thread Philip Warner
At 09:38 PM 24/10/2002 -0400, Bruce Momjian wrote: OK, I am focusing on AIX now. I don't think we can go down the road of saying where large file support is needed or not needed. I think for each platform either we support large files or we don't. Rather than having a different patch file for

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-24 Thread Philip Warner
At 09:56 PM 24/10/2002 -0400, Bruce Momjian wrote: > > > You are quite correct. It should read: > > > > > > #ifdef HAVE_FSEEKO > > > ctx->hasSeek = fseeko(...,SEEK_SET); ^^ > > > #else > > >

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-24 Thread Bruce Momjian
You are going to have to be more specific than that. --- Philip Warner wrote: > > The patch will not work. Please reread my quoted email. > > At 09:32 PM 24/10/2002 -0400, Bruce Momjian wrote: > >Philip Warner wrote: > > >

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-24 Thread Philip Warner
The patch will not work. Please reread my quoted email. At 09:32 PM 24/10/2002 -0400, Bruce Momjian wrote: Philip Warner wrote: > > You are quite correct. It should read: > > #ifdef HAVE_FSEEKO > ctx->hasSeek = fseeko(...,SEEK_SET); > #else > ctx->ha

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-24 Thread Bruce Momjian
Zeugswetter Andreas SB SD wrote: > > > The question is *which* seek APIs we need to support. Are there any > > besides fseeko() and fgetpos()? > > On AIX we have > int fseeko64 (FILE* Stream, off64_t Offset, int Whence); > which is intended for large file access for programs that do NOT > #defi

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-24 Thread Bruce Momjian
Philip Warner wrote: > At 10:08 PM 23/10/2002 -0400, Bruce Momjian wrote: > >Well, that certainly changes the functionality of the code. I thought > >that fseeko test was done so that things that couldn't be seeked on were > >detected. > > You are quite correct. It should read: > > #ifd

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-24 Thread Bruce Momjian
Peter Eisentraut wrote: > Bruce Momjian writes: > > > OK, NetBSD added. > > > > Any other OS's need this? Is it safe for me to code something that > > assumes fpos_t and off_t are identical? I can't think of a good way to > > test if two data types are identical. I don't think sizeof is enough.

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-24 Thread Peter Eisentraut
Bruce Momjian writes: > OK, NetBSD added. > > Any other OS's need this? Is it safe for me to code something that > assumes fpos_t and off_t are identical? I can't think of a good way to > test if two data types are identical. I don't think sizeof is enough. No, you can't assume that fpos_t and

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-24 Thread Zeugswetter Andreas SB SD
> The question is *which* seek APIs we need to support. Are there any > besides fseeko() and fgetpos()? On AIX we have int fseeko64 (FILE* Stream, off64_t Offset, int Whence); which is intended for large file access for programs that do NOT #define _LARGE_FILES It is functionality that is avai

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Bruce Momjian
OK, NetBSD added. Any other OS's need this? Is it safe for me to code something that assumes fpos_t and off_t are identical? I can't think of a good way to test if two data types are identical. I don't think sizeof is enough. -

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Bruce Momjian
Looks like I have some more work to do. Thanks. --- Giles Lean wrote: > > > OK, does pre-1.6 NetBSD have fgetpos/fsetpos that is off_t/quad? > > Yes: > > int > fgetpos(FILE *stream, fpos_t *pos); > > int >

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Giles Lean
> OK, does pre-1.6 NetBSD have fgetpos/fsetpos that is off_t/quad? Yes: int fgetpos(FILE *stream, fpos_t *pos); int fsetpos(FILE *stream, const fpos_t *pos); Per comments in fpos_t is the same format as off_t, and off_t and fpos_t have been 64 bit since 1994. http://cvswe

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Bruce Momjian
Philip Warner wrote: > At 10:03 PM 23/10/2002 -0400, Bruce Momjian wrote: > >It is much cleaner to just duplicate the entire API so you don't have > >any limitations or failure cases. > > We may still end up using macros in pg_dump to cope with cases where off_t > & fseeko are not defined - if th

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Philip Warner
At 10:03 PM 23/10/2002 -0400, Bruce Momjian wrote: It is much cleaner to just duplicate the entire API so you don't have any limitations or failure cases. We may still end up using macros in pg_dump to cope with cases where off_t & fseeko are not defined - if there are any. I presume we would t

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Philip Warner
At 10:08 PM 23/10/2002 -0400, Bruce Momjian wrote: Well, that certainly changes the functionality of the code. I thought that fseeko test was done so that things that couldn't be seeked on were detected. You are quite correct. It should read: #ifdef HAVE_FSEEKO ctx->hasSee

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Bruce Momjian
Well, that certainly changes the functionality of the code. I thought that fseeko test was done so that things that couldn't be seeked on were detected. Not sure what isn't seek-able, maybe named pipes. I thought it was testing that so I didn't touch that variable. This was my original thought

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Bruce Momjian
Philip Warner wrote: > At 09:45 PM 23/10/2002 -0400, Bruce Momjian wrote: > >We have to write another function because fsetpos doesn't do SEEK_CUR so > >you have to implement it with more complex code. It isn't a drop in > >place thing. > > The only code that uses SEEK_CUR is the code to check if

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Philip Warner
At 11:55 AM 24/10/2002 +1000, Philip Warner wrote: The only code that uses SEEK_CUR is the code to check if seek is available - I am ver happy to change that to SEEK_SET - I can't even recall why I used SEEK_CUR. The code that does the real seeks uses SEEK_SET. Come to think of it: ctx->h

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Philip Warner
At 09:45 PM 23/10/2002 -0400, Bruce Momjian wrote: We have to write another function because fsetpos doesn't do SEEK_CUR so you have to implement it with more complex code. It isn't a drop in place thing. The only code that uses SEEK_CUR is the code to check if seek is available - I am ver hap

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Bruce Momjian
Philip Warner wrote: > At 09:41 PM 23/10/2002 -0400, Bruce Momjian wrote: > >If we get this, everything is fine. I have done that for BSD/OS today. > >I may need to do the same for NetBSD/OpenBSD too. > > What did you do to achieve this? See src/port/fseeko.c in current CVS, with some configure.

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Philip Warner
At 09:41 PM 23/10/2002 -0400, Bruce Momjian wrote: If we get this, everything is fine. I have done that for BSD/OS today. I may need to do the same for NetBSD/OpenBSD too. What did you do to achieve this? Philip Warner

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Philip Warner
At 09:36 PM 23/10/2002 -0400, Bruce Momjian wrote: We are going to need to either get fseeko workarounds for those, or disable those features in a meaningful way. ? if we have not got a 64 bit seek function of any kind, then use a 32 bit seek - the features don't need to be disabled. AFAICT

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Bruce Momjian
Philip Warner wrote: > At 11:50 PM 23/10/2002 +0200, Peter Eisentraut wrote: > > >1. Disable access to large files. > > > >2. Seek in some other way. > > This gets my vote, but I would like to see a clean implementation (not huge > quantities if ifdefs every time we call fseek); either we write

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Philip Warner
At 11:50 PM 23/10/2002 +0200, Peter Eisentraut wrote: 1. Disable access to large files. 2. Seek in some other way. This gets my vote, but I would like to see a clean implementation (not huge quantities if ifdefs every time we call fseek); either we write our own fseek as Bruce seems to be su

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Bruce Momjian
Philip Warner wrote: > At 10:42 AM 23/10/2002 -0400, Bruce Momjian wrote: > >What I am concerned about are cases that fail at runtime, specifically > >during a restore of a >2gig file. > > Please give an example that would still apply assuming we get a working > seek/tell pair that works with wha

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Philip Warner
At 10:42 AM 23/10/2002 -0400, Bruce Momjian wrote: What I am concerned about are cases that fail at runtime, specifically during a restore of a >2gig file. Please give an example that would still apply assuming we get a working seek/tell pair that works with whatever we use as an offset? If yo

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Bruce Momjian
Giles Lean wrote: > > Bruce Momjian <[EMAIL PROTECTED]> writes: > > > OK, well BSD/OS now works, but I wonder if there are any other quad > > off_t OS's out there without fseeko. > > NetBSD prior to 1.6, released September 14, 2002. (Source: CVS logs.) OK, does pre-1.6 NetBSD have fgetpos/fsetp

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Bruce Momjian
Philip Warner wrote: > At 05:50 PM 23/10/2002 -0400, Bruce Momjian wrote: > >Looking at the pg_dump code, it seems the fseeks are optional in there > >anyway because it already has code to read the file sequentially rather > > But there are features that are not available if it can't seek: eg. it

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Philip Warner
At 05:50 PM 23/10/2002 -0400, Bruce Momjian wrote: Looking at the pg_dump code, it seems the fseeks are optional in there anyway because it already has code to read the file sequentially rather But there are features that are not available if it can't seek: eg. it will not restore in a differen

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Giles Lean
Bruce Momjian <[EMAIL PROTECTED]> writes: > OK, well BSD/OS now works, but I wonder if there are any other quad > off_t OS's out there without fseeko. NetBSD prior to 1.6, released September 14, 2002. (Source: CVS logs.) OpenBSD prior to 2.7, released June 15, 2000. (Source: release notes.) F

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Bruce Momjian
Tom Lane wrote: > Peter Eisentraut <[EMAIL PROTECTED]> writes: > > First we need to decide what we want to happen and after that think about > > how to implement it. Given sizeof(off_t) > sizeof(long) and no fseeko(), > > we have the following options: > > It seems obvious to me that there are no

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Tom Lane
Bruce Momjian <[EMAIL PROTECTED]> writes: > How would we disable access to large files? I think configure should fail if it can't find a way to seek. Workaround for anyone in that situation is configure --disable-largefile. regards, tom lane ---(en

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Tom Lane
Peter Eisentraut <[EMAIL PROTECTED]> writes: > First we need to decide what we want to happen and after that think about > how to implement it. Given sizeof(off_t) > sizeof(long) and no fseeko(), > we have the following options: It seems obvious to me that there are no platforms that offer sizeof

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Bruce Momjian
Peter Eisentraut wrote: > Bruce Momjian writes: > > > I think you are right that we have to not use off_t and use long if we > > can't find a proper 64-bit seek function, but what are the failure modes > > of doing this? Exactly what happens for larger files? > > First we need to decide what we

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Peter Eisentraut
Bruce Momjian writes: > I think you are right that we have to not use off_t and use long if we > can't find a proper 64-bit seek function, but what are the failure modes > of doing this? Exactly what happens for larger files? First we need to decide what we want to happen and after that think ab

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Bruce Momjian
Philip Warner wrote: > At 01:02 AM 23/10/2002 -0400, Bruce Momjian wrote: > > >OK, you are saying if we don't have fseeko(), there is no reason to use > >off_t, and we may as well use long. What limitations does that impose, > >and are the limitations clear to the user. > > What I'm saying is th

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-23 Thread Bruce Momjian
Philip Warner wrote: > At 01:02 AM 23/10/2002 -0400, Bruce Momjian wrote: > > >OK, you are saying if we don't have fseeko(), there is no reason to use > >off_t, and we may as well use long. What limitations does that impose, > >and are the limitations clear to the user. > > What I'm saying is th

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Philip Warner
At 01:02 AM 23/10/2002 -0400, Bruce Momjian wrote: OK, you are saying if we don't have fseeko(), there is no reason to use off_t, and we may as well use long. What limitations does that impose, and are the limitations clear to the user. What I'm saying is that if we have not got fseeko then we

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Bruce Momjian
OK, you are saying if we don't have fseeko(), there is no reason to use off_t, and we may as well use long. What limitations does that impose, and are the limitations clear to the user. What has me confused is that I only see two places that use a non-zero fseeko, and in those cases, there is a

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Philip Warner
At 12:29 AM 23/10/2002 -0400, Bruce Momjian wrote: This fseeko/ftello/off_t is just too fluid, and the failure modes too serious. I agree. Can you think of a better solution than the one I suggested??? Philip Warner

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Philip Warner
At 12:32 AM 23/10/2002 -0400, Tom Lane wrote: I am wondering why pg_dump has to depend on either fseek or ftell. It doesn't - it just works better and has more features if they are available, much like zlib etc. Philip Warner

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Tom Lane
Bruce Momjian <[EMAIL PROTECTED]> writes: > I wonder if any other platforms have this limitation. I think we need > to add some type of test for no-fseeko()/ftello() and sizeof(off_t) > > sizeof(long). This fseeko/ftello/off_t is just too fluid, and the > failure modes too serious. I am wonderin

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Bruce Momjian
Sounds messy. Let me see if I can code up an fseeko/ftello for BSD/OS and add that to /port. No reason to hold up beta for that, though. I wonder if any other platforms have this limitation. I think we need to add some type of test for no-fseeko()/ftello() and sizeof(off_t) > sizeof(long). Th

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Philip Warner
At 10:46 PM 22/10/2002 -0400, Bruce Momjian wrote: Uh, not exactly. I have off_t as a quad, and I don't have fseeko, so the above conditional doesn't work. I want to use off_t, but can't use fseek(). Then when you create dumps, they will be invalid since I assume that ftello is also broken in

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Bruce Momjian
Philip Warner wrote: > At 05:37 PM 22/10/2002 -0400, Bruce Momjian wrote: > >! if (ctx->hasSeek > >! #if !defined(HAVE_FSEEKO) > >! && sizeof(off_t) <= sizeof(long) > >! #endif > >! ) > > Just to clarify my understanding: > > - HAVE_FSEEKO

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Philip Warner
At 05:37 PM 22/10/2002 -0400, Bruce Momjian wrote: ! if (ctx->hasSeek ! #if !defined(HAVE_FSEEKO) ! && sizeof(off_t) <= sizeof(long) ! #endif ! ) Just to clarify my understanding: - HAVE_FSEEKO is tested & defined in configure - If it is

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Bruce Momjian
Peter Eisentraut wrote: > Bruce Momjian writes: > > > I am concerned about one more thing. On BSD/OS, we have off_t of quad > > (8 byte), but we don't have fseeko, so this call looks questionable: > > > > if (fseeko(AH->FH, tctx->dataPos, SEEK_SET) != 0) > > Maybe you want to ask your OS pro

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Peter Eisentraut
Bruce Momjian writes: > I am concerned about one more thing. On BSD/OS, we have off_t of quad > (8 byte), but we don't have fseeko, so this call looks questionable: > > if (fseeko(AH->FH, tctx->dataPos, SEEK_SET) != 0) Maybe you want to ask your OS provider how the heck this is supposed to

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Bruce Momjian
Bruce Momjian wrote: > > So I think we're wasting our time to debate whether we need to support > > non-integral off_t ... let's just apply Bruce's version and wait to > > see if anyone has a problem before doing more work. > > I am concerned about one more thing. On BSD/OS, we have off_t of quad

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Bruce Momjian
Patch applied with shift <> changes by me. Thanks. --- Philip Warner wrote: > > I have put the latest patch at: > > http://downloads.rhyme.com.au/postgresql/pg_dump/ > > along with two dump files of the regression

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Tom Lane
Bruce Momjian <[EMAIL PROTECTED]> writes: > Your version will break more often because we are assuming we can > determine the endian-ness of the OS, _and_ for quad off_t types, > assuming we know that is stored the same too. While we have ending for > int's, I have no idea if quads are always stor

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Bruce Momjian
Tom Lane wrote: > Bruce Momjian <[EMAIL PROTECTED]> writes: > > However, since we don't know if we support any non-integral off_t > > platforms, and because a configure test would require us to have two > > code paths for with/without integral off_t, I suggest we apply my > > version of Philip's pa

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Bruce Momjian
Philip Warner wrote: > At 12:00 PM 22/10/2002 -0400, Bruce Momjian wrote: > >It does have the advantage of being more portable on systems > >that do have integral off_t > > I suspect it is no more portable than determining storage order by using > 'int i = 256', then writing in storage order, and

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Tom Lane
Bruce Momjian <[EMAIL PROTECTED]> writes: > However, since we don't know if we support any non-integral off_t > platforms, and because a configure test would require us to have two > code paths for with/without integral off_t, I suggest we apply my > version of Philip's patch and let's see if every

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Philip Warner
At 12:00 PM 22/10/2002 -0400, Bruce Momjian wrote: It does have the advantage of being more portable on systems that do have integral off_t I suspect it is no more portable than determining storage order by using 'int i = 256', then writing in storage order, and has the disadvantage that it ma

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Bruce Momjian
Tom Lane wrote: > Philip Warner <[EMAIL PROTECTED]> writes: > > None, but it will be compatible with itself (the most we can hope for), and > > will work even if shifting is not supported for off_t (how likely is > > that?). I agree shift is definitely the way to go if it works on arbitrary > >

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Tom Lane
Philip Warner <[EMAIL PROTECTED]> writes: > None, but it will be compatible with itself (the most we can hope for), and > will work even if shifting is not supported for off_t (how likely is > that?). I agree shift is definitely the way to go if it works on arbitrary > data - ie. it does not rel

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Philip Warner
At 10:16 AM 21/10/2002 -0400, Tom Lane wrote: What are the odds that dumping the bytes in it, in either order, will produce something that's compatible with any other platform? None, but it will be compatible with itself (the most we can hope for), and will work even if shifting is not supporte

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-22 Thread Philip Warner
At 09:52 PM 21/10/2002 -0400, Bruce Momjian wrote: 4) pg_restore -Fc pg_restore /tmp/x is enough; it will determine the file type, and by avoiding the pipe, you allow it to do seeks which are not much use here, but are usefull when you only restore one table in a very large backup.

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-21 Thread Larry Rosenman
On Mon, 2002-10-21 at 20:52, Bruce Momjian wrote: > Larry Rosenman wrote: > > On Mon, 2002-10-21 at 20:47, Bruce Momjian wrote: > > > > > > Here is a modified version of Philip's patch that has the changes Tom > > > suggested; treating off_t as an integral type. I did light testing on > > > my B

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-21 Thread Bruce Momjian
Larry Rosenman wrote: > On Mon, 2002-10-21 at 20:47, Bruce Momjian wrote: > > > > Here is a modified version of Philip's patch that has the changes Tom > > suggested; treating off_t as an integral type. I did light testing on > > my BSD/OS machine that has 8-byte off_t but I don't have 4 gigs of

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-21 Thread Larry Rosenman
On Mon, 2002-10-21 at 20:47, Bruce Momjian wrote: > > Here is a modified version of Philip's patch that has the changes Tom > suggested; treating off_t as an integral type. I did light testing on > my BSD/OS machine that has 8-byte off_t but I don't have 4 gigs of free > space to test larger fil

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-21 Thread Bruce Momjian
Here is a modified version of Philip's patch that has the changes Tom suggested; treating off_t as an integral type. I did light testing on my BSD/OS machine that has 8-byte off_t but I don't have 4 gigs of free space to test larger files. ftp://candle.pha.pa.us/pub/postgresql/mypatch

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-20 Thread Bruce Momjian
Philip Warner wrote: > At 09:18 PM 20/10/2002 -0400, Bruce Momjian wrote: > >I will try to apply it within the next 48 hours. > > I'm happy to apply it when necessary; but I wouldn't do it until we've from > some someone with a big-endian machine... Well, I think Tom was going to try it on his H

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-20 Thread Philip Warner
At 09:18 PM 20/10/2002 -0400, Bruce Momjian wrote: I will try to apply it within the next 48 hours. I'm happy to apply it when necessary; but I wouldn't do it until we've from some someone with a big-endian machine... Philip Wa

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-20 Thread Bruce Momjian
Your patch has been added to the PostgreSQL unapplied patches list at: http://momjian.postgresql.org/cgi-bin/pgpatches I will try to apply it within the next 48 hours. --- Philip Warner wrote: > > I have put the

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-18 Thread Philip Warner
I have put the latest patch at: http://downloads.rhyme.com.au/postgresql/pg_dump/ along with two dump files of the regression DB, one with 4 byte and the other with 8 byte offsets. I can read/restore each from the other, so it looks pretty good. Once the endianness is tested, we should be OK

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-18 Thread Philip Warner
At 12:07 AM 19/10/2002 +0200, Peter Eisentraut wrote: Any old machine has a 4-byte off_t if you configure with --disable-largefile. Thanks - done. I just dumped to a custom backup file, then dumped it do SQL, and compared each version (V7.2.1, 8 byte & 4 byte offsets), and they all looked OK.

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-18 Thread Peter Eisentraut
Philip Warner writes: > > I have made the changes to pg_dump and verified that (a) it reads old > files, (b) it handles 8 byte offsets, and (c) it dumps & seems to restore > (at least to /dev/null). > > I don't have a lot of options for testing it - should I just apply the > changes and wait for t

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-18 Thread Tom Lane
Philip Warner <[EMAIL PROTECTED]> writes: > I don't have a lot of options for testing it - should I just apply the > changes and wait for the problems, or can someone offer a bigendian machine > and/or a 4 byte off_t machine? My HP is big-endian; send in the patch and I'll check it here...

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-17 Thread Philip Warner
I have made the changes to pg_dump and verified that (a) it reads old files, (b) it handles 8 byte offsets, and (c) it dumps & seems to restore (at least to /dev/null). I don't have a lot of options for testing it - should I just apply the changes and wait for the problems, or can someone offe

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-03 Thread Philip Warner
At 11:07 PM 3/10/2002 -0400, Tom Lane wrote: >A non-integral representation >of off_t is theoretically possible but I don't believe it exists in >practice. Excellent. So I can just read/write the bytes in an appropriate order and expect whatever size it is to be a single intXX. Fine with me, un

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-03 Thread Bruce Momjian
Tom Lane wrote: > Giles Lean <[EMAIL PROTECTED]> writes: > > When talking of near-current systems with 64 bit off_t you are not > > going to find one without support for 64 bit integral types. > > I tend to agree with Giles on this point. A non-integral representation > of off_t is theoretically

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-03 Thread Tom Lane
Giles Lean <[EMAIL PROTECTED]> writes: > When talking of near-current systems with 64 bit off_t you are not > going to find one without support for 64 bit integral types. I tend to agree with Giles on this point. A non-integral representation of off_t is theoretically possible but I don't believ

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-03 Thread Giles Lean
Philip Warner writes: > Yes, but there is no guarantee that off_t is implemented as such, nor would > we be wise to assume so (most docs say explicitly not to do so). I suspect you're reading old documents, which is why I asked what you were referring to. In the '80s what you are saying would

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-03 Thread Philip Warner
At 07:15 AM 4/10/2002 +1000, Giles Lean wrote: > > My limited reading of off_t stuff now suggests that it would be brave to > > assume it is even a simple 64 bit number (or even 3 32 bit numbers). > >What are you reading?? If you find a platform with 64 bit file >offsets that doesn't support 64

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-03 Thread Giles Lean
Philip Warner writes: > My limited reading of off_t stuff now suggests that it would be brave to > assume it is even a simple 64 bit number (or even 3 32 bit numbers). What are you reading?? If you find a platform with 64 bit file offsets that doesn't support 64 bit integral types I will not

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-03 Thread Mario Weilguni
>My limited reading of off_t stuff now suggests that it would be brave to >assume it is even a simple 64 bit number (or even 3 32 bit numbers). One >alternative, which I am not terribly fond of, is to have pg_dump write >multiple files - when we get to 1 or 2GB, we just open another file, and

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-03 Thread Philip Warner
At 11:06 AM 2/10/2002 -0400, Tom Lane wrote: >It needs to get done; AFAIK no one has stepped up to do it. Do you want >to? My limited reading of off_t stuff now suggests that it would be brave to assume it is even a simple 64 bit number (or even 3 32 bit numbers). One alternative, which I am n

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-02 Thread Philip Warner
At 11:06 AM 2/10/2002 -0400, Tom Lane wrote: >It needs to get done; AFAIK no one has stepped up to do it. Do you want >to? I'll have a look; my main concern at the moment is that off_t and size_t are totally non-committal as to structure; in particular I can probably safely assume that they ar

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-02 Thread Bruce Momjian
Philip Warner wrote: > At 09:42 AM 2/10/2002 +1000, Philip Warner wrote: > >Yes, and do the peripheral stuff to support old archives etc. > > Does silence mean people agree? Does it also mean someone is doing this > (eg. whoever did the off_t support)? Or does it mean somebody else needs to > d

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-02 Thread Tom Lane
Philip Warner <[EMAIL PROTECTED]> writes: > At 09:42 AM 2/10/2002 +1000, Philip Warner wrote: >> Yes, and do the peripheral stuff to support old archives etc. > Does silence mean people agree? Does it also mean someone is doing this > (eg. whoever did the off_t support)? Or does it mean somebody

Re: [HACKERS] pg_dump and large files - is this a problem?

2002-10-02 Thread Philip Warner
At 09:42 AM 2/10/2002 +1000, Philip Warner wrote: >Yes, and do the peripheral stuff to support old archives etc. Does silence mean people agree? Does it also mean someone is doing this (eg. whoever did the off_t support)? Or does it mean somebody else needs to do it? ---

  1   2   >