Re: [pgsql-patches] Remove of .SECONDARY from SGML Makefile
Bruce Momjian wrote: To get the new html dependency to work properly, I removed the .SECONDARY tag from the Makefile. SECONDARY prevents missing files from being built. That is not what it does. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [pgsql-patches] [HACKERS] [PATCHES] Last infomask bit
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: Has anyone bothered to measure the overhead added by having to mask to fetch or store the natts value? This is not a zero-cost improvement. Tom, how should this be tested? I assume some loop of the same query over and over again. I'd be satisfied by a demonstration of no meaningful difference in pgbench numbers. I ran pgbench on CVS checkout taken before the patch was applied, and I couldn't measure a difference. I got 1329-1340 TPM from three runs both with and without the patch. The tests were run on my laptop, with scaling factor 10, using pgbench postgres -t 10 -v, with fsync and full_page_writes disabled to make it CPU bound, while observing top to confirm that CPU usage was 100% during the test. I think that's enough performance testing for this patch. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [pgsql-patches] [HACKERS] [PATCHES] wal_checksum = on (default) | off
On Wed, 2007-01-10 at 23:32 -0500, Bruce Momjian wrote: Simon Riggs wrote: On Fri, 2007-01-05 at 22:57 -0500, Tom Lane wrote: Jim Nasby [EMAIL PROTECTED] writes: On Jan 5, 2007, at 6:30 AM, Zeugswetter Andreas ADI SD wrote: Ok, so when you need CRC's on a replicate (but not on the master) you Which sounds to me like a good reason to allow the option in recovery.conf as well... Actually, I'm not seeing the use-case for a slave having a different setting from the master at all? My backup server is less reliable than the primary. My backup server is more reliable than the primary. Somehow, neither of these statements seem likely to be uttered by a sane DBA ... If I take a backup of a server and bring it up on a new system, the blocks in the backup will not have been CRC checked before they go to disk. If I take the same server and now stream log records across to it, why *must* that data be CRC checked, when the original data has not been? I'm proposing choice, with a safe default. That's all. I am assuming this item is dead because no performance results have been reported. It's been on my hold queue, as a result of its lack of clear acceptance. Results from earlier tests show the routines which are dominated by CRC checking overhead are prominent in a number of important workloads. Those workloads all have a substantial disk component, so test results will vary between no saving at all on a disk-bound system to some savings on a CPU bound system. Restore RecordIsValid() #1 on oprofile results at 50-70% CPU COPYXLogInsert() #1 on oprofile results at 17% CPU (full_page_writes = on) OLTPno test with full_page_writes = on (less relevant) OLTPXLogInsert() #5 on oprofile results at 1.2% CPU (full_page_writes = off) Removing the CRC checks on WAL would likely be the easiest to remove 1% CPU on the system as it stands. Other changes require algorithmic or architectural changes to improve matters, though gains can be much larger. 1% doesn't sound much, but PostgreSQL is a very sleek beast these days. As we improve things in other areas the importance of this patch as a tuning switch will grow. Clearly the current patch is not accepted, but can we imagine a patch that saved substantial CPU time in these areas that would be acceptable? *Always* as a non-default option, IMHO, with careful documentation as to its possible use. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [pgsql-patches] [HACKERS] [PATCHES] wal_checksum = on (default) | off
Simon Riggs [EMAIL PROTECTED] writes: COPY XLogInsert() #1 on oprofile results at 17% CPU (full_page_writes = on) But what portion of that is actually CRC-related? XLogInsert does quite a lot. Anyway, I can't see degrading the reliability of the system for a gain in the range of a few percent, which is the most that we'd be likely to get here ... for a factor of two or more, maybe people would be willing to take a risk. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [pgsql-patches] [HACKERS] [PATCHES] wal_checksum = on (default) | off
On Thu, 2007-01-11 at 09:01 -0500, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: COPYXLogInsert() #1 on oprofile results at 17% CPU (full_page_writes = on) But what portion of that is actually CRC-related? XLogInsert does quite a lot. Anyway, I can't see degrading the reliability of the system for a gain in the range of a few percent, which is the most that we'd be likely to get here ... for a factor of two or more, maybe people would be willing to take a risk. All I would add is that the loss of reliability was not certain in all cases, otherwise I myself would have dropped the idea long ago. With the spectre of doubt surrounding this, I'm happy to drop the idea until we have proof/greater certainty either way. Patch revoked. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [pgsql-patches] [HACKERS] [PATCHES] wal_checksum = on (default) | off
Gregory Stark [EMAIL PROTECTED] writes: What did you think about protecting against torn writes using id numbers every 512 bytes. Pretty much not happening; or are you volunteering to fix every part of the system to tolerate injections of inserted data anywhere in a stored datum? regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [pgsql-patches] [HACKERS] [PATCHES] wal_checksum = on (default) | off
Tom Lane [EMAIL PROTECTED] writes: Gregory Stark [EMAIL PROTECTED] writes: What did you think about protecting against torn writes using id numbers every 512 bytes. Pretty much not happening; or are you volunteering to fix every part of the system to tolerate injections of inserted data anywhere in a stored datum? I was thinking to do it at a low level as the xlog records are prepared to be written to the filesystem and as the data is being read from disk. I haven't read that code yet to see where to inject it but I understand there's already a copy happening and it could be done there. Even if we optimize out all the copies we could do it in the actual i/o call using readv/writev. I wasn't thinking of doing it on actual disk buffers since it doesn't help us avoid full page writes, but it could be done there too using readv/writev in the smgr. That might be useful for a full_page_writes=off system so even if it can't guarantee no corruption at least it can guarantee no silent corruption. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [pgsql-patches] [HACKERS] [PATCHES] wal_checksum = on (default) | off
Gregory Stark [EMAIL PROTECTED] writes: Tom Lane [EMAIL PROTECTED] writes: Pretty much not happening; or are you volunteering to fix every part of the system to tolerate injections of inserted data anywhere in a stored datum? I was thinking to do it at a low level as the xlog records are prepared to be written to the filesystem and as the data is being read from disk. I haven't read that code yet to see where to inject it but I understand there's already a copy happening and it could be done there. You understand wrong ... a tuple sitting on disk is normally read directly from the shared buffer, and I don't think we want to pay for copying it. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [pgsql-patches] [HACKERS] [PATCHES] wal_checksum = on (default) | off
Tom Lane [EMAIL PROTECTED] writes: Gregory Stark [EMAIL PROTECTED] writes: Tom Lane [EMAIL PROTECTED] writes: Pretty much not happening; or are you volunteering to fix every part of the system to tolerate injections of inserted data anywhere in a stored datum? I was thinking to do it at a low level as the xlog records are prepared to be written to the filesystem and as the data is being read from disk. I haven't read that code yet to see where to inject it but I understand there's already a copy happening and it could be done there. You understand wrong ... a tuple sitting on disk is normally read directly from the shared buffer, and I don't think we want to pay for copying it. xlog records -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [pgsql-patches] [HACKERS] [PATCHES] wal_checksum = on (default) | off
Gregory Stark [EMAIL PROTECTED] writes: Tom Lane [EMAIL PROTECTED] writes: You understand wrong ... a tuple sitting on disk is normally read directly from the shared buffer, and I don't think we want to pay for copying it. xlog records Oh, sorry, had the wrong context in mind. I'm still not very impressed with the idea --- a CRC check will catch many kinds of problems, whereas this approach catches exactly one kind of problem. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [pgsql-patches] [HACKERS] [PATCHES] wal_checksum = on (default) | off
Tom Lane [EMAIL PROTECTED] writes: Oh, sorry, had the wrong context in mind. I'm still not very impressed with the idea --- a CRC check will catch many kinds of problems, whereas this approach catches exactly one kind of problem. Well in fairness I tossed in a throwaway comment at the end of that email about heap pages. I'll do the same here since I can't resist. But the main thread here is about xlog really. It just seems to me like it's better to target each problem with a solution that addresses it directly than have one feature that we hope hope addresses them all more or less. Having a CRC in WAL but not in the heap seems kind of pointless. If your hardware is unreliable the corruption could anywhere. Depending on it to solve multiple problems means we can't offer the option to disable it because it would affect other things as well. What I would like to see is a CRC option that would put CRC checks in every disk page whether heap, index, WAL, control file, etc. I think we would default that to off to match our current setup most closely. Separately we would have a feature in WAL to detect torn pages so that we can reliably detect the end of valid WAL. That would have to always be on. But having it as a separate feature means the CRC could be optional. Also, incidentally like I mentioned in my previous email, we could do the torn page detection in heap pages too by handling it in the smgr using readv/writev. No copies, no corrupted datums. Essentially the tags would be inserted on the fly as the data was copied into kernel space. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [pgsql-patches] [HACKERS] [PATCHES] Patch to log usage of temporary files
On Tue, 2007-01-09 at 17:16 -0500, Bruce Momjian wrote:
Tom Lane wrote:
/* reset flag so that die() interrupt won't cause
problems */
vfdP-fdstate = ~FD_TEMPORARY;
+ PG_TRACE1(temp__file__cleanup, vfdP-fileName);
+ if (log_temp_files = 0)
+ {
+ if (stat(vfdP-fileName, filestats) == 0)
The TRACE is in the wrong place no? I thought it was going to be after
the stat() operation so it could pass the file size.
We had that discussion already. If you only pass it after the stat()
then you cannot use DTrace, except when you already get a message in the
log and therefore don't need DTrace. DTrace can get the filesize if it
likes, but thats up to the script author.
Also, I dunno much about DTrace, but I had the idea that you can't
simply throw a PG_TRACE macro into the source and think you are done
--- isn't there a file of probe declarations to add to? Not to mention
the documentation of what probes exist.
I didn't like the macro in that area anyway. It seems too adhock to
just throw it in when we have so few places monitored now. Removed.
err... why are we removing it? The patch should have included an
addition to the probes.d file also, but that should be fixed, not
removed. Don't we normally reject incomplete patches?
You can't say we don't have many probes so we won't add one. There never
will be many if we do that - its a circular argument.
--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
Re: [pgsql-patches] [HACKERS] [PATCHES] Patch to log usage of temporary files
Simon Riggs [EMAIL PROTECTED] writes: Tom Lane wrote: The TRACE is in the wrong place no? I thought it was going to be after the stat() operation so it could pass the file size. We had that discussion already. If you only pass it after the stat() then you cannot use DTrace, except when you already get a message in the log and therefore don't need DTrace. We may have had the discussion, but apparently you didn't follow it :-(. The point of the proposal was that if you wanted to DTrace temp files you would set log_temp_files to some large value, thus causing the stat() call to occur but no log message to come out. This is surely a lot more efficient than having to have DTrace open the file for itself --- and if you don't care about micro-efficiency, what's wrong with using the logging option? You can't say we don't have many probes so we won't add one. There never will be many if we do that - its a circular argument. I think the real criterion has to be is this probe useful to developers?. I'm entirely uninterested in adding probes that are targeted towards DBAs, as this one would have been --- if we think there's a problem that a DBA would have, we need to offer a more portable solution than that. Which we did, in the form of a logging option, which makes the DTrace probe pretty useless anyway. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [pgsql-patches] [HACKERS] [PATCHES] Patch to log usage of temporary files
Simon Riggs wrote: Also, I dunno much about DTrace, but I had the idea that you can't simply throw a PG_TRACE macro into the source and think you are done --- isn't there a file of probe declarations to add to? Not to mention the documentation of what probes exist. I didn't like the macro in that area anyway. It seems too adhock to just throw it in when we have so few places monitored now. Removed. err... why are we removing it? The patch should have included an addition to the probes.d file also, but that should be fixed, not removed. Don't we normally reject incomplete patches? You can't say we don't have many probes so we won't add one. There never will be many if we do that - its a circular argument. The trace probe was incorrect and kind of at an odd place. I don't think we want to go down the road of throwing trace in everwhere, do we? I would like to see a more systematic approach to it. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [pgsql-patches] Remove of .SECONDARY from SGML Makefile
Peter Eisentraut wrote: Bruce Momjian wrote: To get the new html dependency to work properly, I removed the .SECONDARY tag from the Makefile. SECONDARY prevents missing files from being built. That is not what it does. It does several things, but that is one of the things it does: `.SECONDARY' The targets which `.SECONDARY' depends on are treated as intermediate files, except that they are never automatically deleted. *Note Chains of Implicit Rules: Chained Rules. `.SECONDARY' with no prerequisites marks all file targets mentioned in the makefile as secondary. and intermediate files are defined as: Intermediate files are remade using their rules just like all other files. But intermediate files are treated differently in two ways. The first difference is what happens if the intermediate file does not exist. If an ordinary file B does not exist, and `make' considers - a target that depends on B, it invariably creates B and then updates the target from B. But if B is an intermediate file, then `make' can leave well enough alone. It won't bother updating B, or the ultimate -- target, unless some prerequisite of B is newer than that target or there is some other reason to update that target. The second difference is that if `make' _does_ create B in order to update something else, it deletes B later on after it is no longer needed. Therefore, an intermediate file which did not exist before `make' also does not exist after `make'. `make' reports the deletion to you by printing a `rm -f' command showing which file it is deleting. This is what caused the dependencies not to run. SECONDARY was added by you in 2000. What I don't understand is what else we have lost by removing it: revision 1.27 date: 2000/12/14 22:30:56; author: petere; state: Exp; lines: +2 -1 Merge functions and operators chapters. Lots of updates. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
[pgsql-patches] pthread option of msvc build.
Hi Magnus-san. I am trying simple construction by operating config.pl. It has changed wonderfully now. however, I do not use ecpg, and see the simplest construction. At that time, even pthread might not be needed. It was simple. Please consider this. P.S) I can't catch up with your quick work. However, I will try the debugging execution with VS2005 by the arrangement option. tools/msvc will surely facilitate debugging.! Thanks. Regards, Hiroshi Saito msvc_patch Description: Binary data ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [pgsql-patches] pthread option of msvc build.
Hiroshi Saito wrote:
Hi Magnus-san.
I am trying simple construction by operating config.pl.
It has changed wonderfully now. however, I do not use ecpg, and see the
simplest construction. At that time, even pthread might not be needed.
It was simple.
Good point, but the patch doesn't work. Simply removing the check
doesn't work, it will cause warnings and errors all over when doing a
complete build.
Please try the attached patch which actually disables the building of
ecpg if pthreads is not specified.
//Magnus
Index: src\tools\msvc/Project.pm
===
RCS file: c:/prog/cvsrepo/pgsql/pgsql/src/tools/msvc/Project.pm,v
retrieving revision 1.5
diff -c -r1.5 Project.pm
*** src\tools\msvc/Project.pm 4 Jan 2007 17:58:19 - 1.5
--- src\tools\msvc/Project.pm 11 Jan 2007 18:54:19 -
***
*** 367,377
}
$libs =~ s/ $//;
$libs =~ s/__CFGNAME__/$cfgname/g;
print $f EOF;
Configuration Name=$cfgname|Win32
OutputDirectory=.\\$cfgname\\$self-{name}
IntermediateDirectory=.\\$cfgname\\$self-{name}
ConfigurationType=$cfgtype UseOfMFC=0
ATLMinimizesCRunTimeLibraryUsage=FALSE CharacterSet=2
WholeProgramOptimization=$p-{wholeopt}
Tool Name=VCCLCompilerTool Optimization=$p-{opt}
!
AdditionalIncludeDirectories=src/include;src/include/port/win32;src/include/port/win32_msvc;$self-{solution}-{options}-{pthread};$self-{includes}
PreprocessorDefinitions=WIN32;_WINDOWS;__WINDOWS__;__WIN32__;EXEC_BACKEND;WIN32_STACK_RLIMIT=4194304;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE$self-{defines}$p-{defs}
StringPooling=$p-{strpool}
RuntimeLibrary=$p-{runtime}
DisableSpecificWarnings=$self-{disablewarnings}
--- 367,379
}
$libs =~ s/ $//;
$libs =~ s/__CFGNAME__/$cfgname/g;
+ my $pth = $self-{solution}-{options}-{pthread};
+ $pth = '' unless $pth;
print $f EOF;
Configuration Name=$cfgname|Win32
OutputDirectory=.\\$cfgname\\$self-{name}
IntermediateDirectory=.\\$cfgname\\$self-{name}
ConfigurationType=$cfgtype UseOfMFC=0
ATLMinimizesCRunTimeLibraryUsage=FALSE CharacterSet=2
WholeProgramOptimization=$p-{wholeopt}
Tool Name=VCCLCompilerTool Optimization=$p-{opt}
!
AdditionalIncludeDirectories=src/include;src/include/port/win32;src/include/port/win32_msvc;$pth;$self-{includes}
PreprocessorDefinitions=WIN32;_WINDOWS;__WINDOWS__;__WIN32__;EXEC_BACKEND;WIN32_STACK_RLIMIT=4194304;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE$self-{defines}$p-{defs}
StringPooling=$p-{strpool}
RuntimeLibrary=$p-{runtime}
DisableSpecificWarnings=$self-{disablewarnings}
Index: src\tools\msvc/Solution.pm
===
RCS file: c:/prog/cvsrepo/pgsql/pgsql/src/tools/msvc/Solution.pm,v
retrieving revision 1.5
diff -c -r1.5 Solution.pm
*** src\tools\msvc/Solution.pm 29 Dec 2006 16:49:02 - 1.5
--- src\tools\msvc/Solution.pm 11 Jan 2007 18:52:16 -
***
*** 6,12
sub new {
my $junk = shift;
my $options = shift;
- die Pthreads is required.\n unless $options-{pthread};
my $self = {
projects = {},
options = $options,
--- 6,11
Index: src\tools\msvc/mkvcbuild.pl
===
RCS file: c:/prog/cvsrepo/pgsql/pgsql/src/tools/msvc/mkvcbuild.pl,v
retrieving revision 1.9
diff -c -r1.9 mkvcbuild.pl
*** src\tools\msvc/mkvcbuild.pl 9 Jan 2007 06:00:43 - 1.9
--- src\tools\msvc/mkvcbuild.pl 11 Jan 2007 18:52:32 -
***
*** 90,117
$pgtypes-AddReference($postgres,$libpgport);
$pgtypes-AddIncludeDir('src\interfaces\ecpg\include');
! my $libecpg =
$solution-AddProject('libecpg','dll','interfaces','src\interfaces\ecpg\ecpglib');
! $libecpg-AddDefine('FRONTEND');
! $libecpg-AddIncludeDir('src\interfaces\ecpg\include');
! $libecpg-AddIncludeDir('src\interfaces\libpq');
! $libecpg-AddIncludeDir('src\port');
! $libecpg-AddLibrary('wsock32.lib');
! $libecpg-AddLibrary($config-{'pthread'} . '\pthreadVC2.lib');
! $libecpg-AddReference($libpq,$pgtypes);
!
! my $libecpgcompat =
$solution-AddProject('libecpg_compat','dll','interfaces','src\interfaces\ecpg\compatlib');
! $libecpgcompat-AddIncludeDir('src\interfaces\ecpg\include');
! $libecpgcompat-AddIncludeDir('src\interfaces\libpq');
! $libecpgcompat-AddReference($pgtypes,$libecpg);
!
! my $ecpg =
$solution-AddProject('ecpg','exe','interfaces','src\interfaces\ecpg\preproc');
! $ecpg-AddIncludeDir('src\interfaces\ecpg\include');
! $ecpg-AddIncludeDir('src\interfaces\libpq');
! $ecpg-AddFiles('src\interfaces\ecpg\preproc','pgc.l','preproc.y');
! $ecpg-AddDefine('MAJOR_VERSION=4');
! $ecpg-AddDefine('MINOR_VERSION=2');
! $ecpg-AddDefine('PATCHLEVEL=1');
!
Re: [pgsql-patches] [HACKERS] [PATCHES] Patch to log usage of temporary files
On Thu, Jan 11, 2007 at 12:35:25PM -0500, Tom Lane wrote: I think the real criterion has to be is this probe useful to developers?. I'm entirely uninterested in adding probes that are targeted towards DBAs, as this one would have been --- if we think there's a problem that a DBA would have, we need to offer a more portable solution than that. Which we did, in the form of a logging option, which makes the DTrace probe pretty useless anyway. But the problem with just logging stuff is it's not a monitoring solution. Granted, it's better than nothing, but in a production environment I'd like to have some way to monitor temp file usage/utilization over time. AFAIK dtrace would provide that capability, thought I don't think it'd be unreasonable to have our own counter as well. Perhaps add temp_usage_count and temp_usage_size to pg_stat_database (number of times something spilled to disk and total size, respectively). -- Jim Nasby[EMAIL PROTECTED] EnterpriseDB http://enterprisedb.com 512.569.9461 (cell) ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
[pgsql-patches] vcbuild - conditional regeneration
The attached patch changes vcbuild so the project and solution files are
only regenerated if they are actually changed. This helps when you're
developing in the Visual Studio GUI, because updating the files (even to
the same contents) will cause a reload and re-parse of all the projects
in the solution.
As a bonus, it doesn't trash the project file if it for some reason
aborts half-way through.
The patch includes part of the previously sent patch about pthreads
requirements, because they change the same area. Should be workable
without the other one as well, so it's no problem applying just one of
them if the other one is troublesome.
//Magnus
Index: src/tools/msvc/Project.pm
===
RCS file: c:/prog/cvsrepo/pgsql/pgsql/src/tools/msvc/Project.pm,v
retrieving revision 1.5
diff -c -r1.5 Project.pm
*** src/tools/msvc/Project.pm 4 Jan 2007 17:58:19 - 1.5
--- src/tools/msvc/Project.pm 11 Jan 2007 21:24:38 -
***
*** 252,261
}
# Dump the project
! open(F, $self-{name}.vcproj) || croak(Could not write to
$self-{name}.vcproj\n);
! $self-WriteHeader(*F);
! $self-WriteReferences(*F);
! print F EOF;
Files
EOF
my @dirstack = ();
--- 252,261
}
# Dump the project
!my $dump = '';
! $dump .= $self-WriteHeader();
! $dump .= $self-WriteReferences();
! $dump .= EOF;
Files
EOF
my @dirstack = ();
***
*** 271,277
last if (length($dir) ==
length(join('\\',@dirstack)));
last if (substr($dir,
length(join('\\',@dirstack)),1) eq '\\');
}
! print F ' ' x $#dirstack . /Filter\n;
pop @dirstack;
}
# Now walk forwards and create whatever directories are needed
--- 271,277
last if (length($dir) ==
length(join('\\',@dirstack)));
last if (substr($dir,
length(join('\\',@dirstack)),1) eq '\\');
}
! $dump .= ' ' x $#dirstack . /Filter\n;
pop @dirstack;
}
# Now walk forwards and create whatever directories are needed
***
*** 280,317
$left =~ s/^\\//;
my @pieces = split /\\/, $left;
push @dirstack, $pieces[0];
! print F ' ' x $#dirstack . Filter
Name=\$pieces[0]\ Filter=\\\n;
}
! print F ' ' x $#dirstack .File RelativePath=\$f\;
if ($f =~ /\.y$/) {
my $of = $f;
$of =~ s/\.y$/.c/;
$of =~
s{^src\\pl\\plpgsql\\src\\gram.c$}{src\\pl\\plpgsql\\src\\pl_gram.c};
! print F '' . GenerateCustomTool('Running bison on ' .
$f, 'cmd /V:ON /c src/tools/msvc\pgbison.bat ' . $f, $of) . '/File' . \n;
}
elsif ($f =~ /\.l$/) {
my $of = $f;
$of =~ s/\.l$/.c/;
$of =~
s{^src\\pl\\plpgsql\\src\\scan.c$}{src\\pl\\plpgsql\\src\\pl_scan.c};
! print F '' . GenerateCustomTool('Running flex on ' .
$f, 'src/tools/msvc\pgflex.bat ' . $f,$of) . '/File' . \n;
}
elsif (defined($uniquefiles{$file})) {
# File already exists, so fake a new name
my $obj = $dir;
$obj =~ s/\\/_/g;
! print F FileConfiguration Name=\Debug|Win32\Tool
Name=\VCCLCompilerTool\ ObjectFile=\.\\debug\\$self-{name}\\$obj .
_$file.obj\ //FileConfigurationFileConfiguration
Name=\Release|Win32\Tool Name=\VCCLCompilerTool\
ObjectFile=\.\\release\\$self-{name}\\$obj . _$file.obj\
//FileConfiguration/File\n;
}
else {
$uniquefiles{$file} = 1;
! print F /\n;
}
}
while ($#dirstack = 0) {
! print F ' ' x $#dirstack . /Filter\n;
pop @dirstack;
}
! $self-Footer(*F);
close(F);
}
--- 280,331
$left =~ s/^\\//;
my @pieces = split /\\/, $left;
push @dirstack, $pieces[0];
! $dump .= ' ' x $#dirstack . Filter
Name=\$pieces[0]\ Filter=\\\n;
}
! $dump .= ' ' x $#dirstack .File RelativePath=\$f\;
if ($f =~ /\.y$/) {
my $of = $f;
$of =~ s/\.y$/.c/;
$of =~
s{^src\\pl\\plpgsql\\src\\gram.c$}{src\\pl\\plpgsql\\src\\pl_gram.c};
! $dump .= '' . GenerateCustomTool('Running bison on ' .
$f, 'cmd /V:ON /c
Re: [pgsql-patches] [HACKERS] [PATCHES] Patch to log usage of temporary files
On Thu, 2007-01-11 at 12:37 -0500, Bruce Momjian wrote: The trace probe was incorrect Yes, incomplete, no doubt. On that point you were 100% right to reject. and kind of at an odd place. I don't think we want to go down the road of throwing trace in everwhere, do we? I would like to see a more systematic approach to it. I guess my systematic approach was to add PG_TRACE to all new log points from now on, so we have a choice of which trace/log mechanism to use. I'm happy with everything but the portability of DTrace, however I'm fond of the idea of sowing seeds for the time when other solutions will work also. I've a bag of PG_TRACE trace points to add sometime soon. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [pgsql-patches] [HACKERS] [PATCHES] Patch to log usage of temporary files
On Thu, 2007-01-11 at 12:35 -0500, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Tom Lane wrote: The TRACE is in the wrong place no? I thought it was going to be after the stat() operation so it could pass the file size. We had that discussion already. If you only pass it after the stat() then you cannot use DTrace, except when you already get a message in the log and therefore don't need DTrace. We may have had the discussion, but apparently you didn't follow it :-(. My apologies. You can't say we don't have many probes so we won't add one. There never will be many if we do that - its a circular argument. I think the real criterion has to be is this probe useful to developers?. I'm entirely uninterested in adding probes that are targeted towards DBAs, as this one would have been --- if we think there's a problem that a DBA would have, we need to offer a more portable solution than that. Which we did, in the form of a logging option, which makes the DTrace probe pretty useless anyway. Well, you know my major objection to including DTrace was all to do with portability. I'm happy that the way its been implemented allows other solutions to take advantage of the trace points also. We're working on 8.3 now and by the time that is delivered and perhaps for 2 years hence, i.e. Aug 2009, the software will be in production use. In that period, DTrace will have been ported more widely and I'm hearing that some kind of user space solution for Linux will mature in that time also. If that isn't true then I'll be more interested in some custom tracing solutions built around the PG_TRACE macro concept. My thought is to provide both a log-based trace solution as has been done, plus a hook for PG_TRACE (not just DTrace) at the same time. i.e. each time we enhance the logging infrastructure, take the time to place a trace point there also. Theologically, we both know we see things differently on the DBA v Developer discussion. The only point I would make is that the more information you give the DBA, the more comes back to you as a Developer. You, personally, could not possibly have interacted with as many server set-ups required to highlight the problems and issues you address. It's only because of the info provided by the existing system that you're able to make headway with rare optimizer problems. My perspective is that if you help the DBA you also help the Developer; if you help the Developer only, then the Developer's information is also inevitably restricted. The tip says EXPLAIN ANALYZE is your friend. It's right, and it isn't just talking to DBAs. My feeling is that this is true for all tools/trace mechanisms. I'd rather be sent the info than have to go do it myself on an individual basis. Indirect access isn't the best way, but we harvest a much wider range of information that way. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [pgsql-patches] [HACKERS] [PATCHES] wal_checksum = on (default) | off
On Thu, 2007-01-11 at 17:06 +, Gregory Stark wrote: Having a CRC in WAL but not in the heap seems kind of pointless. Yes... If your hardware is unreliable the corruption could anywhere. Agreed. Other DBMS have one setting for the whole server; I've never seen separate settings for WAL and data. Depending on it to solve multiple problems means we can't offer the option to disable it because it would affect other things as well. What I would like to see is a CRC option that would put CRC checks in every disk page whether heap, index, WAL, control file, etc. I think we would default that to off to match our current setup most closely. Separately we would have a feature in WAL to detect torn pages so that we can reliably detect the end of valid WAL. That would have to always be on. But having it as a separate feature means the CRC could be optional. Your thoughts seem logical to me. It does seem a bigger project than I'd envisaged, but doable, one day. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [pgsql-patches] [PATCHES] Tablespace for temporary objects and sort files
Please, go on with that, I hadn't seen that problem. Indeed, I read Andrew answer to your question and I think it's a nice solution. A Dimecres 10 Gener 2007 05:33, Jaime Casanova va escriure: On 1/9/07, Albert Cervera Areny [EMAIL PROTECTED] wrote: I don't have much time lately so if you're willing to work on it, please do. after doing the index part i revisited the patch again and saw that there is something fundamentally wrong here (sorry for no noticing that before :( ) your are using local backend variables for the iterator so two different backends will use to different variables... because of that if you have 3 temp tablespaces in your temp_tablespaces guc and start 100 backend and every one of them create just one temp table those 100 temp tables will be in the first temp tablespace... :( i will try to fix that as well... unless you want to do it, just tell me... ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [pgsql-patches] [PATCHES] Tablespace for temporary objects and sort files
On 1/11/07, Albert Cervera Areny [EMAIL PROTECTED] wrote: Please, go on with that, I hadn't seen that problem. Indeed, I read Andrew answer to your question and I think it's a nice solution. yes... i'm always trying to kill flies with tanks... ;) i will use Andrew's suggestion... -- regards, Jaime Casanova Programming today is a race between software engineers striving to build bigger and better idiot-proof programs and the universe trying to produce bigger and better idiots. So far, the universe is winning. Richard Cook ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
[pgsql-patches] width_bucket() for float8
Attached is a patch that provides an implementation of width_bucket()
for the float8 data type, per earlier discussion. The implementation is
fairly ugly, but I don't see an easy way to simplify it.
When writing this, I noticed that my previous implementation of
width_bucket() probably doesn't handle NaN correctly:
postgres=# select width_bucket('NaN', 1, 5, 5);
width_bucket
--
6
(1 row)
AFAICS SQL:2003 does not define a NaN value, so it doesn't address how
width_bucket() should behave here. The patch changes width_bucket() so
that ereport(ERROR) is raised if NaN is specified for the operand or the
lower or upper bounds to width_bucket(). I think this is reasonable
behavior -- any objections?
float8 also allows positive and negative infinity. I disallowed infinite
values for the histogram bounds, but allowed it for the operand.
-Neil
Index: doc/src/sgml/func.sgml
===
RCS file: /home/neilc/postgres/cvs_root/pgsql/doc/src/sgml/func.sgml,v
retrieving revision 1.348
diff -c -p -r1.348 func.sgml
*** doc/src/sgml/func.sgml 21 Dec 2006 16:05:12 - 1.348
--- doc/src/sgml/func.sgml 11 Jan 2007 20:21:23 -
***
*** 854,859
--- 854,869
entryliteralwidth_bucket(5.35, 0.024, 10.06, 5)/literal/entry
entryliteral3/literal/entry
/row
+
+ row
+entryliteralfunctionwidth_bucket/function(parameterop/parameter typefloat8/type, parameterb1/parameter typefloat8/type, parameterb2/parameter typefloat8/type, parametercount/parameter typeint/type)/literal/entry
+entrytypeint/type/entry
+entryreturn the bucket to which parameteroperand/ would
+be assigned in an equidepth histogram with parametercount/
+buckets, in the range parameterb1/ to parameterb2//entry
+entryliteralwidth_bucket(5.35, 0.024, 10.06, 5)/literal/entry
+entryliteral3/literal/entry
+ /row
/tbody
/tgroup
/table
Index: src/backend/utils/adt/float.c
===
RCS file: /home/neilc/postgres/cvs_root/pgsql/src/backend/utils/adt/float.c,v
retrieving revision 1.146
diff -c -p -r1.146 float.c
*** src/backend/utils/adt/float.c 6 Jan 2007 20:21:29 - 1.146
--- src/backend/utils/adt/float.c 12 Jan 2007 00:09:26 -
*** float84ge(PG_FUNCTION_ARGS)
*** 2745,2750
--- 2745,2831
PG_RETURN_BOOL(float8_cmp_internal(arg1, arg2) = 0);
}
+ /*
+ * Implements the float8 version of the width_bucket() function
+ * defined by SQL2003. See also width_bucket_numeric().
+ *
+ * 'bound1' and 'bound2' are the lower and upper bounds of the
+ * histogram's range, respectively. 'count' is the number of buckets
+ * in the histogram. width_bucket() returns an integer indicating the
+ * bucket number that 'operand' belongs in for an equiwidth histogram
+ * with the specified characteristics. An operand smaller than the
+ * lower bound is assigned to bucket 0. An operand greater than the
+ * upper bound is assigned to an additional bucket (with number
+ * count+1).
+ */
+ Datum
+ width_bucket_float8(PG_FUNCTION_ARGS)
+ {
+ float8 operand = PG_GETARG_FLOAT8(0);
+ float8 bound1 = PG_GETARG_FLOAT8(1);
+ float8 bound2 = PG_GETARG_FLOAT8(2);
+ int32 count = PG_GETARG_INT32(3);
+ int32 result;
+
+ if (count = 0.0)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_ARGUMENT_FOR_WIDTH_BUCKET_FUNCTION),
+ errmsg(count must be greater than zero)));
+
+ if (isnan(operand) || isnan(bound1) || isnan(bound2))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_ARGUMENT_FOR_WIDTH_BUCKET_FUNCTION),
+ errmsg(operand, lower bound and upper bound cannot be NaN)));
+
+ /* Note that we allow operand to be infinite */
+ if (is_infinite(bound1) || is_infinite(bound2))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_ARGUMENT_FOR_WIDTH_BUCKET_FUNCTION),
+ errmsg(lower and upper bounds must be finite)));
+
+ if (bound1 bound2)
+ {
+ if (operand bound1)
+ result = 0;
+ else if (operand = bound2)
+ {
+ result = count + 1;
+ /* check for overflow */
+ if (result count)
+ ereport(ERROR,
+ (errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
+ errmsg(integer out of range)));
+ }
+ else
+ result = ((float8) count * (operand - bound1) / (bound2 - bound1)) + 1;
+ }
+ else if (bound1 bound2)
+ {
+ if (operand bound1)
+ result = 0;
+ else if (operand = bound2)
+ {
+ result = count + 1;
+ /* check for overflow */
+ if (result count)
+ ereport(ERROR,
+ (errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
+ errmsg(integer out of range)));
+ }
+ else
+ result = ((float8) count * (bound1 - operand) / (bound1 - bound2)) + 1;
+ }
+ else
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_ARGUMENT_FOR_WIDTH_BUCKET_FUNCTION),
+ errmsg(lower bound cannot equal upper
Re: [pgsql-patches] [PATCHES] Tablespace for temporary objects and
On Thu, 2007-01-11 at 21:05 -0500, Jaime Casanova wrote: On 1/11/07, Albert Cervera Areny [EMAIL PROTECTED] wrote: Please, go on with that, I hadn't seen that problem. Indeed, I read Andrew answer to your question and I think it's a nice solution. yes... i'm always trying to kill flies with tanks... ;) Isn't that a little expensive on gas? i will use Andrew's suggestion... -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 Providing the most comprehensive PostgreSQL solutions since 1997 http://www.commandprompt.com/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [pgsql-patches] [PATCHES] Tablespace for temporary objects and sort files
On 1/11/07, Joshua D. Drake [EMAIL PROTECTED] wrote: On Thu, 2007-01-11 at 21:05 -0500, Jaime Casanova wrote: On 1/11/07, Albert Cervera Areny [EMAIL PROTECTED] wrote: Please, go on with that, I hadn't seen that problem. Indeed, I read Andrew answer to your question and I think it's a nice solution. yes... i'm always trying to kill flies with tanks... ;) Isn't that a little expensive on gas? maybe... but... what a beatiful explosion we can get... ;) seriously, attached's a new version of the patch... the patch use temp tablespaces for: - temp tables - temp files (generated by sorts and such) - indexes on temp tables the temp_tablespaces GUC still cannot be set from postgresql.conf, i will keep working on that but i have to understand the code... any comments on this patch? although it's not ready for apply it yet, i think Albert made a good work on it... -- regards, Jaime Casanova Programming today is a race between software engineers striving to build bigger and better idiot-proof programs and the universe trying to produce bigger and better idiots. So far, the universe is winning. Richard Cook ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [pgsql-patches] [PATCHES] Tablespace for temporary objects and sort files
Sorry, patch attached this time...
On 1/12/07, Jaime Casanova [EMAIL PROTECTED] wrote:
On 1/11/07, Joshua D. Drake [EMAIL PROTECTED] wrote:
On Thu, 2007-01-11 at 21:05 -0500, Jaime Casanova wrote:
On 1/11/07, Albert Cervera Areny [EMAIL PROTECTED] wrote:
Please, go on with that, I hadn't seen that problem. Indeed, I read Andrew
answer to your question and I think it's a nice solution.
yes... i'm always trying to kill flies with tanks... ;)
Isn't that a little expensive on gas?
maybe... but... what a beatiful explosion we can get... ;)
seriously, attached's a new version of the patch...
the patch use temp tablespaces for:
- temp tables
- temp files (generated by sorts and such)
- indexes on temp tables
the temp_tablespaces GUC still cannot be set from postgresql.conf, i
will keep working on that but i have to understand the code...
any comments on this patch? although it's not ready for apply it yet,
i think Albert made a good work on it...
--
regards,
Jaime Casanova
Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs and the universe trying
to produce bigger and better idiots.
So far, the universe is winning.
Richard Cook
--
Atentamente,
Jaime Casanova
Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs and the universe trying
to produce bigger and better idiots.
So far, the universe is winning.
Richard Cook
Index: src/backend/commands/indexcmds.c
===
RCS file: /projects/cvsroot/pgsql/src/backend/commands/indexcmds.c,v
retrieving revision 1.152
diff -c -r1.152 indexcmds.c
*** src/backend/commands/indexcmds.c9 Jan 2007 02:14:11 - 1.152
--- src/backend/commands/indexcmds.c12 Jan 2007 05:08:35 -
***
*** 209,215
}
else
{
! tablespaceId = GetDefaultTablespace();
/* note InvalidOid is OK in this case */
}
--- 209,221
}
else
{
! /*
!* if the target table is temporary then use a temp_tablespace
!*/
! if (!rel-rd_istemp)
! tablespaceId = GetDefaultTablespace();
! else
! tablespaceId = GetTempTablespace();
/* note InvalidOid is OK in this case */
}
Index: src/backend/commands/tablecmds.c
===
RCS file: /projects/cvsroot/pgsql/src/backend/commands/tablecmds.c,v
retrieving revision 1.210
diff -c -r1.210 tablecmds.c
*** src/backend/commands/tablecmds.c5 Jan 2007 22:19:26 - 1.210
--- src/backend/commands/tablecmds.c12 Jan 2007 05:08:47 -
***
*** 334,339
--- 334,343
errmsg(tablespace \%s\ does not
exist,
stmt-tablespacename)));
}
+ else if (stmt-relation-istemp)
+ {
+ tablespaceId = GetTempTablespace();
+ }
else
{
tablespaceId = GetDefaultTablespace();
Index: src/backend/commands/tablespace.c
===
RCS file: /projects/cvsroot/pgsql/src/backend/commands/tablespace.c,v
retrieving revision 1.40
diff -c -r1.40 tablespace.c
*** src/backend/commands/tablespace.c 5 Jan 2007 22:19:26 - 1.40
--- src/backend/commands/tablespace.c 12 Jan 2007 05:08:49 -
***
*** 65,73
#include utils/lsyscache.h
! /* GUC variable */
char *default_tablespace = NULL;
static bool remove_tablespace_directories(Oid tablespaceoid, bool redo);
static void set_short_version(const char *path);
--- 65,76
#include utils/lsyscache.h
! /* GUC variables */
char *default_tablespace = NULL;
+ char *temp_tablespaces = NULL;
+ int next_temp_tablespace;
+ int num_temp_tablespaces;
static bool remove_tablespace_directories(Oid tablespaceoid, bool redo);
static void set_short_version(const char *path);
***
*** 930,935
--- 933,1069
return result;
}
+ /*
+ * Routines for handling the GUC variable 'temp_tablespaces'.
+ */
+
+ /* assign_hook: validate new temp_tablespaces, do extra actions as needed */
+ const char *
+ assign_temp_tablespaces(const char *newval, bool doit, GucSource source)
+ {
+ char *rawname;
+ List *namelist;
+ ListCell *l;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(newval);
+
+ /* Parse string into list of identifiers */
+ if (!SplitIdentifierString(rawname, ',', namelist))
+ {
+ /* syntax error in name list */
+
