Re: [HACKERS] Are these groups unauthorized?
On Fri, 28 Jun 2002, Lamar Owen wrote: On Friday 28 June 2002 10:46 am, Tom Lane wrote: Guido Ostkamp [EMAIL PROTECTED] writes: I am sure, a lot of people would be happy, if those groups were officially introduced and hosted on many international newservers. Yup. Are you volunteering to be the proponent who shepherds a vote through the official process? Maybe more like 'martyr' who shepherds a vote through. I remember going through a few votes years ago, and the memories are not fond ones. That's why I never bothered ... I've been admin'ng Usenet for enough years now to know that if you create them and propogate them to your neighbors, they will eventually get propogaated out and created ... there are a few admin out there that are anal about creating 'unauthorized' groups, but most out there just let them pass ... That said, if anyone wants to provide an open NNTP server for the c.d.postgresql.* hierarchy, please let me know and we'll add ou on ... ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Are these groups unauthorized?
Tom Lane [EMAIL PROTECTED] wrote:
Guido Ostkamp [EMAIL PROTECTED] writes:
I am sure, a lot of people would be happy, if those groups were
officially introduced and hosted on many international newservers.
Yup. Are you volunteering to be the proponent who shepherds a vote
through the official process?
No.
If you look closely at the 'comp.databases.*' hierarchy you will find
that most of the databases listed have only one group, with the
exception of the big players like Oracle. That means, the maximum you
would be able to get is a 'comp.databases.postgresql', but not the bunch
of groups which is available here. I don't believe admins here would
agree to throw away all others.
What I recommend to do, is that the names of the groups here gets
changed by stripping of the 'comp.databases' prefix. The group names
would then make up their own main hierarchy ('postgres.*') like it
exists for other stuff or companies as well (like 'microsoft.*') etc.
That would AFAIK no longer violate any rules, and allow webmasters from
outside to host these groups. Only the people reading these groups
would need a small and easy reconfiguration of their subscribed lists
which could be announced by a posting before its done, that's all.
What do you think?
BTW: I see you belong to the core development team. Are you responsible
for running this server news.postgresql.org?
Regards,
Guido
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/users-lounge/docs/faq.html
[HACKERS] Vacuum Daemon
From the ToDo list: Vacuum: * Provide automatic running of vacuum in the background (Tom) As of 7.2 we have lazy vacuum. The next logical step is setting up vacuum to run automatically in the background either as some type of daemon or as something kicked off by the postmaster. I am interested in working on this to do item, although I see it is assigned to Tom right now. First: is this something we still want (I assume it is since its in the todo.). Second: There was some discussion (http://archives.postgresql.org/pgsql-hackers/2002-05/msg00970.php) about this not being neede once UNDO is on place, what is the current view on this? Matthew ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Vacuum Daemon
Matthew T. O'Connor [EMAIL PROTECTED] writes: As of 7.2 we have lazy vacuum. The next logical step is setting up vacuum to run automatically in the background either as some type of daemon or as something kicked off by the postmaster. I am interested in working on this to do item, although I see it is assigned to Tom right now. It's sufficiently far down my to-do list that I'm happy to let someone else do it ;-). Second: There was some discussion (http://archives.postgresql.org/pgsql-hackers/2002-05/msg00970.php) about this not being neede once UNDO is on place, what is the current view on this? I do not think that is the case; and anyway we've pretty much rejected Vadim's notion of going to an Oracle-style UNDO buffer. I don't foresee VACUUM going away anytime soon --- what we need is to make it less obtrusive. 7.2 made some progress in that direction, but we need more. Launching VACUUMs on some automatic schedule, preferably using feedback about where space needs to be reclaimed, seems like a pretty straightforward small-matter-of-programming. The thing that would really be needed to make it unobtrusive is to find a way to run the vacuum processing at low priority, or at least when the system is not heavily loaded. I don't know a good way to do that. Nice'ing the vacuum process won't work because of priority-inversion problems. Making it suspend itself when load gets high might do; but how to detect that in a reasonably portable fashion? regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Vacuum Daemon
On Sat, 2002-06-29 at 20:14, Tom Lane wrote: Matthew T. O'Connor [EMAIL PROTECTED] writes: Second: There was some discussion (http://archives.postgresql.org/pgsql-hackers/2002-05/msg00970.php) about this not being neede once UNDO is on place, what is the current view on this? I do not think that is the case; and anyway we've pretty much rejected Vadim's notion of going to an Oracle-style UNDO buffer. I don't foresee VACUUM going away anytime soon --- what we need is to make it less obtrusive. 7.2 made some progress in that direction, but we need more. Could someone point me to this discussion, or summarize what the problem was? Was his proposal to keep tuple versions in the UNDO AM, or only pointers to them? The referred-to message seems to be about something else. ;jrnield -- J. R. Nield [EMAIL PROTECTED] ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Vacuum Daemon
J. R. Nield [EMAIL PROTECTED] writes: I do not think that is the case; and anyway we've pretty much rejected Vadim's notion of going to an Oracle-style UNDO buffer. Could someone point me to this discussion, or summarize what the problem was? I'm too lazy to dig through the archives at the moment, but the main points were (a) a finite-size UNDO buffer chokes on large transactions and (b) the Oracle approach requires live transaction processing to do the cleanup work that our approach can push off to hopefully-not- time-critical vacuum processing. UNDO per se doesn't eliminate VACUUM anyhow; it only reclaims space from tuples written by aborted transactions. If you want to get rid of VACUUM then you need another way to get rid of the old versions of perfectly good committed tuples that are obsoleted by updates from later transactions. That essentially means you need an overwriting storage manager, which is a concept that doesn't mix well with MVCC. Oracle found a solution to that conundrum, but it's really not obvious to me that their solution is better than ours. Also, they have patents that we'd probably run afoul of if we try to imitate their approach too closely. regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Vacuum Daemon
Tom Lane wrote: J. R. Nield [EMAIL PROTECTED] writes: I do not think that is the case; and anyway we've pretty much rejected Vadim's notion of going to an Oracle-style UNDO buffer. Could someone point me to this discussion, or summarize what the problem was? I'm too lazy to dig through the archives at the moment, but the main points were (a) a finite-size UNDO buffer chokes on large transactions and (b) the Oracle approach requires live transaction processing to do the cleanup work that our approach can push off to hopefully-not- time-critical vacuum processing. UNDO per se doesn't eliminate VACUUM anyhow; it only reclaims space from tuples written by aborted transactions. If you want to get rid of VACUUM then you need another way to get rid of the old versions of perfectly good committed tuples that are obsoleted by updates from later transactions. That essentially means you need an overwriting storage manager, which is a concept that doesn't mix well with MVCC. Oracle found a solution to that conundrum, but it's really not obvious to me that their solution is better than ours. Also, they have patents that we'd probably run afoul of if we try to imitate their approach too closely. Don't forget reclaiming space from transactions that delete tuples. UNDO doesn't help there either. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Vacuum Daemon
On Sat, 2002-06-29 at 21:55, Tom Lane wrote: J. R. Nield [EMAIL PROTECTED] writes: I do not think that is the case; and anyway we've pretty much rejected Vadim's notion of going to an Oracle-style UNDO buffer. Could someone point me to this discussion, or summarize what the problem was? I'm too lazy to dig through the archives at the moment, but the main points were (a) a finite-size UNDO buffer chokes on large transactions Yes this is a good point. Oracle was always lame with its ROLLBACK SEGMENTS. SolidDB (SolidWorks? It's been a while...) used a btree-like structure for this that was not of fixed size. Oracle supposedly moved to the same method in its 9i release, but I don't know the details. I could never figure out how they did this, until I realized that UNDO doesn't need to be in the WAL log. You just use any access method you feel like, and make sure the method is itself protected by REDO. Just insert REDO entries to protect the insert into the UNDO AM, and REDO log when you delete. That makes it easy to have the recovery code be idempotent, to catch the case of a system crash during recovery. and (b) the Oracle approach requires live transaction processing to do the cleanup work that our approach can push off to hopefully-not- time-critical vacuum processing. I'm not sure which way I'm leaning on this. On the one hand, it requires extra work to clean up while the system is live, in addition to writing the undo records, though the cleanup is not necessarily by the same transaction that committed the work (the cleanup needs to be deferred until it's out of an active snapshot anyway). On the other hand, you can clean-up without a full table scan, because you know which tuples need to be changed. This can be a big advantage on gigantic tables. Also, it lets you remove deleted tuples quickly, so the space can be reused, and eliminates the xid wraparound problem. Of course, any kind of undo is worse for performance with bulk inserts/updates, so you either end up committing every few thousand inserts, or you use some special extension to disable undo logging for a bulk load (or if you really want to be able to roll it back, you live with it :-) How slow is it to vacuum a 1 TB database with postgres? Do we even have any users who could test this? Also, I would never advocate that we do what I'm pretty sure Oracle does, and keep old values in the Rollback Segment. Only (RelFileNode, ItemDataPointer) addresses would need to be kept in the UNDO AM, if we went this route. UNDO per se doesn't eliminate VACUUM anyhow; it only reclaims space from tuples written by aborted transactions. If you want to get rid of VACUUM then you need another way to get rid of the old versions of perfectly good committed tuples that are obsoleted by updates from later transactions. That essentially means you need an overwriting storage manager, which is a concept that doesn't mix well with MVCC. Well, you can keep the UNDO records after commit to do a fast incremental vacuum as soon as the transaction that deleted the tuples becomes older than the oldest snapshot. If this is always done whenever an XID becomes that old, then you never need to vacuum, and you never need a full table scan. Because postgres never overwrites (except at vacuum), I think it actually makes us a BETTER candidate for this to be implemented cleanly then with an overwriting storage manager. We will never need to keep tuple values in UNDO! Oracle found a solution to that conundrum, but it's really not obvious to me that their solution is better than ours. Their approach was worse, because they had an overwriting storage manager before they tried to implement it (I'm guessing). :-) Also, they have patents that we'd probably run afoul of if we try to imitate their approach too closely. Given the current state of affairs here in the US, PostgreSQL probably violates hundreds or even thousands of software patents. It probably violates tens of patents that have been upheld in court. The only thing keeping companies from shutting down postgres, linux, OpenOffice, and a hundred other projects is fear of adverse publicity, and the fact that development would move overseas and continue to be a thorn in their side. We'll see how long this lasts, given the fear some vendors have of certain maturing open-source/GPL projects, but I don't think PostgreSQL will be first, since anyone can take this code and become an instant proprietary database vendor! (No, I'm not complaining. Please, nobody start a license fight because of this) -- J. R. Nield [EMAIL PROTECTED] ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Vacuum Daemon
On Saturday 29 June 2002 08:14 pm, Tom Lane wrote: Launching VACUUMs on some automatic schedule, preferably using feedback about where space needs to be reclaimed, seems like a pretty straightforward small-matter-of-programming. The thing that would really be needed to make it unobtrusive is to find a way to run the vacuum processing at low priority, or at least when the system is not heavily loaded. I don't know a good way to do that. Nice'ing the vacuum process won't work because of priority-inversion problems. Making it suspend itself when load gets high might do; but how to detect that in a reasonably portable fashion? Are we sure we want it to be unobtrusive? If vacuum is performed only where and when it's needed, it might be better for overall throughput to have it run even when the system is loaded. Such as a constantly updated table. As for a portable way to identify system load (if this is what we want) I was thinking of looking at the load average (such as the one reported by the top command) but I don't know much about portability issues. Since there appears to be sufficient interest in some solution, I'll start working on it. I would like to hear a quick description of what small-matter-of-programming means. Do you have specific ideas about what how best to get that feedback? Matthew ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
