Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-19 Thread Peter Eisentraut
Zdenek Kotala wrote: But how it was mentioned in this thread maybe somethink like this CREATE TABLESPACE name LOCATION '/my/location' SEGMENTS 10GB should good solution. If segments is not mentioned then default value is used. I think you would need a tool to resegmentize a table or

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-19 Thread Martijn van Oosterhout
On Wed, Mar 19, 2008 at 09:38:12AM +0100, Peter Eisentraut wrote: Another factor I just thought of is that tar, commonly used as part of a backup procedure, can on some systems only handle files up to 8 GB in size. There are supposed to be newer formats that can avoid that restriction, but

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-19 Thread Kenneth Marshall
On Wed, Mar 19, 2008 at 10:51:12AM +0100, Martijn van Oosterhout wrote: On Wed, Mar 19, 2008 at 09:38:12AM +0100, Peter Eisentraut wrote: Another factor I just thought of is that tar, commonly used as part of a backup procedure, can on some systems only handle files up to 8 GB in size.

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Peter Eisentraut
Tom Lane wrote: I think this needs to be treated as experimental until it's got a few more than zero miles under its belt. OK, then maybe we should document that. I wouldn't be too surprised to find that we have to implement it as a run-time switch instead of compile-time, in order to not

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Zeugswetter Andreas OSB SD
Why is this not the default when supported? I am wondering both from the point of view of the user, and in terms of development direction. Also it would get more buildfarm coverage if it were default. If it breaks something we'll notice earlier. No we don't, because the buildfarm does

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Larry Rosenman
On Mon, 10 Mar 2008, Tom Lane wrote: Peter Eisentraut [EMAIL PROTECTED] writes: Tom Lane wrote: Applied with minor corrections. Why is this not the default when supported? Fear. Maybe eventually, but right now I think it's too risky. One point that I already found out the hard way is

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Tom Lane
Peter Eisentraut [EMAIL PROTECTED] writes: Tom Lane wrote: I think this needs to be treated as experimental until it's got a few more than zero miles under its belt. OK, then maybe we should document that. Agreed, but at this point we don't even know what hazards we need to document.

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Tom Lane
Zeugswetter Andreas OSB SD [EMAIL PROTECTED] writes: Exactly, e.g. AIX is one of those. jfs (not the newer jfs2) has an option to enable large files, which is not the default and cannot be changed post crfs. And even if it is enabled, jfs has a 64 Gb filesize limit ! Anybody know others that

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Zdenek Kotala
Tom Lane napsal(a): Zeugswetter Andreas OSB SD [EMAIL PROTECTED] writes: Exactly, e.g. AIX is one of those. jfs (not the newer jfs2) has an option to enable large files, which is not the default and cannot be changed post crfs. And even if it is enabled, jfs has a 64 Gb filesize limit ! Anybody

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Tom Lane
Zdenek Kotala [EMAIL PROTECTED] writes: Tom Lane napsal(a): These examples suggest that maybe what we want is not so much a no segments ever mode as a segment size larger than 1GB. PS: ZFS is happy with 2^64bit size and UFS has 1TB file size limit (depends on solaris version) So even on

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Bruce Momjian
Tom Lane wrote: Zdenek Kotala [EMAIL PROTECTED] writes: Tom Lane napsal(a): These examples suggest that maybe what we want is not so much a no segments ever mode as a segment size larger than 1GB. PS: ZFS is happy with 2^64bit size and UFS has 1TB file size limit (depends on solaris

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Zdenek Kotala
Tom Lane napsal(a): Zdenek Kotala [EMAIL PROTECTED] writes: Tom Lane napsal(a): These examples suggest that maybe what we want is not so much a no segments ever mode as a segment size larger than 1GB. PS: ZFS is happy with 2^64bit size and UFS has 1TB file size limit (depends on solaris

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Tom Lane
Zdenek Kotala [EMAIL PROTECTED] writes: I think current patch could stay in CVS and I will rip out non segment code path in a new patch. Sure, I feel no need to revert what's applied. Have at it. regards, tom lane -- Sent via pgsql-patches mailing list

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Peter Eisentraut
Zdenek Kotala wrote: Yes, agree. It seems only ZFS is OK at this moment and if somebody sets 32TB he gets nonsegment mode anyway. Surely if you set the segment size to INT64_MAX, you will get nonsegmented behavior anyway, so two code paths might not be necessary at all. I looked into posix

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-11 Thread Tom Lane
Peter Eisentraut [EMAIL PROTECTED] writes: Zdenek Kotala wrote: Maybe we can put additional test into configure and collect appropriate data from buildfarm. It might be good to just check first if it returns realistic values for the example cases that have been mentioned. Yeah, please just

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-10 Thread Tom Lane
Zdenek Kotala [EMAIL PROTECTED] writes: There is latest version of nonsegment support patch. I changed LET_OS_MANAGE_FILESIZE to USE_SEGMENTED_FILES and I added -disable-segmented-files switch to configure. I kept tuplestore behavior and it still split file in both mode. Applied with minor

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-10 Thread Peter Eisentraut
Tom Lane wrote: Zdenek Kotala [EMAIL PROTECTED] writes: There is latest version of nonsegment support patch. I changed LET_OS_MANAGE_FILESIZE to USE_SEGMENTED_FILES and I added -disable-segmented-files switch to configure. I kept tuplestore behavior and it still split file in both mode.

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-10 Thread Tom Lane
Peter Eisentraut [EMAIL PROTECTED] writes: Tom Lane wrote: Applied with minor corrections. Why is this not the default when supported? Fear. Maybe eventually, but right now I think it's too risky. One point that I already found out the hard way is that sizeof(off_t) = 8 does not guarantee

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-10 Thread Alvaro Herrera
Peter Eisentraut wrote: Tom Lane wrote: Zdenek Kotala [EMAIL PROTECTED] writes: There is latest version of nonsegment support patch. I changed LET_OS_MANAGE_FILESIZE to USE_SEGMENTED_FILES and I added -disable-segmented-files switch to configure. I kept tuplestore behavior and it

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-10 Thread Tom Lane
Alvaro Herrera [EMAIL PROTECTED] writes: Also it would get more buildfarm coverage if it were default. If it breaks something we'll notice earlier. Since nothing the regression tests do even approach 1GB, the odds that the buildfarm will notice problems are approximately zero.