[HACKERS] Guide to PostgreSQL source tree
Hi all,
For a few months now I've been thinking about whether or not a guide
('line-by-line') to the Postgres source tree would be of any value.
Such a guide would, most probably, trace an 'ultimate' query (ie,
one which requires the use of all source level functionality) through the
source as well as reference appendices guides to underlying functionality
(backend lib, transactions, macros) and client interfaces (inc. FE/BE
protocol, internals of the libs etc), procedural language interfaces, SPI
and any other part of the source I have left out.
The guide would look at all non-trivial functions and code segments in the
source. Examination would involve explanation of complex code (at a
line-by-line level), background information of reasoning behind the
code-level design of important functionality, analysis of algorithms and
what ever else seems helpful to people approaching the source.
I think it would be most useful as a non-commercial project intended for
distribution from the Postgres Web site.
Obviously such a project would take a very long time and would have to
involve more people than myself. So the questions which go unanswered
are: Would such a work be of any real use? Would it be of use to enough
people? Is this the right way to go about introducing people to the
source? Is it desirable to introduce (lots of) people to the source?
For my part, I think such a project would be a good way of countering two
things which are affecting Postgres's popularity. Firstly, in Australia
(and I imagine other parts of the world) University courses dealing with
database/information systems (whether it be as basic as an introduction to
SQL or as detailed as developing complicated/sophisticated data storage
systems) are more often than not sponsored by vendors (Oracle, IBM,
Sybase) or 'consultancy' companies who may as well be sales reps for
vendors. Which ever it is, in the end courses are full of marketing drivel
and very little analysis and exploration of real concepts/problems.
Secondly, where Postgres really out performs proprietary databases is in
its source being open. Problems which cause major functionality problems
and downtime in critical vendor installations could often easily be
resolved if developers had the source. I think a thorough source guide
would go some way to bolstering the appeal of Postgres to such developers
whilst countering some of the arguments for throwing hefty sums of money
at support.
So, what do people think?
Gavin
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
RE: [HACKERS] Progress report on locale safe LIKE indexing
Hiroshi Inoue writes: Hmm * string1 = string2 * doesn't imply * string1 LIKE string2 * ? In the current implementation of LIKE, you're right. The SQL standard allows for the possibility that [d]epending on the collating sequence, two strings may compare as equal even if they are of different lengths or contain different sequences of characters. However, I doubt that this can really happen in practice. For example, in some collating sequences (such as en_US), characters with diacritic marks (accents) are more equal than others, but in the end there's always a tie breaker. Or do you know an example where this really happens? Otherwise the current criterion of LIKE matching unwittingly assumes that there's no locale that has the different definition of '=' from that of ASCII locale. I don't think the current implementation is strictly right. Strictly speaking, it isn't. The SQL standard says that literal characters in the pattern must be matched to the characters in the supplied value according to the collating sequence. (See 8.5 GR 3. d) ii) 4).) However, I strongly doubt that that would actually be a good idea. Pattern matching generally doesn't work this way (cf. POSIX regexp), and since locale-aware comparisons are context-sensitive in some cases I don't even want to think about whether this could actually work when faced with wildcards. -- Peter Eisentraut [EMAIL PROTECTED] http://funkturm.homeip.net/~peter ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Progress report on locale safe LIKE indexing
Tom Lane writes: But the LIKE code does know about multibyte character sets. Is it safe to assume that memcmp-based sorting will not make any mistakes with multibyte characters? Remember that this memcmp-based sorting is the same sorting that texteq will do when locale is turned off. So if there were a problem it would already exist, but I'm sure there isn't one. -- Peter Eisentraut [EMAIL PROTECTED] http://funkturm.homeip.net/~peter ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] Re: [PATCHES] encoding names
On Sun, Aug 19, 2001 at 11:02:57AM +0900, Tatsuo Ishii wrote: 4) Encoding official names are inconsistent. Here are my suggested changes (referring http://www.iana.org/assignments/character-sets, according to Peter's suggestiuon): ALT - IBM866 KOI8 - KOI8_R UNICODE - UTF_8 (Peter's suggestion) Right. But we will still need aliases UNICODE, ALT, KOI8 for back compatibility. Thanks, I try fix all. Karel -- Karel Zak [EMAIL PROTECTED] http://home.zf.jcu.cz/~zakkr/ C, PostgreSQL, PHP, WWW, http://docs.linux.cz, http://mape.jcu.cz ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Re: [PATCHES] encoding names
ALT - IBM866 Just a quick comment: ALT is not necessarily IBM866. It can be any US-ASCII or 26-character-alphabet Latin set, for example IBM819 or ISO8859-1. Is actually quite different from IBM866 in its true meaning, and they shouldn't be aliased together. ALT is used for example, when none of KOI8-R, Windows-1251, or IBM866 are available to a Russian-speaking person to read/write any text, messages and stuff, we use simple English letters to write words in Russian so that pronunciation sort of holds the same. It's something like russian_latin (as an equivalent to greek_latin in the http://www.iana.org/assignments/character-sets spec), and the writing this way reminds Polish or Serbian-Latin a bit. Ok. Let's leave ALT as it is. -- Tatsuo Ishii ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
[HACKERS] Re: [PATCHES] encoding names
4) Encoding official names are inconsistent. Here are my suggested changes (referring http://www.iana.org/assignments/character-sets, according to Peter's suggestiuon): ALT - IBM866 KOI8 - KOI8_R UNICODE - UTF_8 (Peter's suggestion) Right. But we will still need aliases UNICODE, ALT, KOI8 for back compatibility. Sure. Thanks, I try fix all. Thanks! But we seem to leave ALT as it is (Serguei's suggestion). -- Tatsuo Ishii ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Guide to PostgreSQL source tree
- Original Message -
From: Gavin Sherry [EMAIL PROTECTED]
Sent: Sunday, August 19, 2001 3:40 AM
Hi all,
For a few months now I've been thinking about whether or not a guide
('line-by-line') to the Postgres source tree would be of any value.
[snip]
So, what do people think?
Such a guide would be nice to have handy as a reference, but how
are you going to keep up with frequent code changes
and the new code added? Keeping this thing up-to-date
is an enormous effort even if you're not the only
one involved.
And what's wrong in automated guide generation? Like
if put specially formatted comments (responsibility of the author of the piece)
for the guide in the source, and run through the source tree
your guide generator from time to time (like on every beta, RC, and release perhaps?).
This way the guide maintenance will be much easier, but the quality will solely
depend on comments supplied by the author.
My two Canadian cents.
Serguei
---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])
[HACKERS] Re: Multiple parameters on aggregates?
mlw wrote: Just a suggestion, how much work would it be to accept multiple parameters on aggregate functions? For instance: select fubar(field1, field2) from table one group by field1; The reason I think that this is useful is that for some statistical operations, often times there is extra per record data that can affect how you calculate a value. This would also be VERY helpful in a lot of OLAP type processing! create function aggfunct( int4, int4, int4 ) returns int4 as '/usr/local/lib/pglib.so', 'aggfunct' language 'c' ; create function aggterm( int4 ) returns int4 as '/usr/local/lib/pglib.so', 'aggterm' language 'c' ; create aggregate agg1 ( basetype = integer, sfunc1 = aggfunct, stype1 = integer, finalfunc = aggterm, initcond1 = 0 ); In the above example, parameters 1 and 2 to aggfunct are the standard aggregate parameters are required by the create aggrigate syntax. At query time, however, the additional parameter(s) is used in addition. As an example of a more complex example, one could do something like this: select mycube_agg(region, date, sales, product) from salesinfo group by region; -- 5-4-3-2-1 Thunderbirds are GO! http://www.mohawksoft.com ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] Guide to PostgreSQL source tree
Serguei Mokhov [EMAIL PROTECTED] writes: And what's wrong in automated guide generation? Even more to the point, what's wrong with looking at the source code? (Why would you want a line by line guide if you're not looking at the source code, anyway?) We could probably do with more extensive high-level documentation than we have, to point people to the parts of the code that they need to read for a particular purpose. But I agree with Sergei that it's hopeless to try to divorce low-level documentation from the code itself. One thing that I find absolutely essential for dealing with any large project is a full-text indexer (I use Glimpse, but I think there are others out there). Being able to quickly look at every use of a particular identifier goes a long way towards answering questions. regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] A fixed user id for the postgres user?
We've had some problem reports that the current practice of initdb assigning to the postgres user the same usesysid as the user id of the Unix user running initdb has caused some clashes. (Imagine this scenario: A few years ago you installed BluePants Linux 5.0, created a user for PostgreSQL, id 501, created a database. Later you created a few real users, which get uids 502, 503, etc. Then you pg_dumpall that database (which saves the sysid). Now you install BluePants Linux 7.0 on a new box, create a new user for PostgreSQL, which turns out to be 502, because foolishly you created a user for TheirSQL first. So now you replay your pg_dumpall and you have two users with id 502. Boom.) One idea to resolve this, by getting rid of the usesysid column in favour of the oid column has fallen by the wayside (for some valid reasons), so the problem remains. I think the simplest fix would be to assign a fixed usesysid of 1. There still is the possibility of changing that with an initdb option, as there always has been. (We could also ensure that CREATE USER never assigns ids below, say, 10, so that if for who knows what reason we decide to add more users into the bootstrap installation we have some room.) Comments? -- Peter Eisentraut [EMAIL PROTECTED] http://funkturm.homeip.net/~peter ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Guide to PostgreSQL source tree
On Sun, 19 Aug 2001, Tom Lane wrote: One thing that I find absolutely essential for dealing with any large project is a full-text indexer (I use Glimpse, but I think there are others out there). Being able to quickly look at every use of a particular identifier goes a long way towards answering questions. Agreed -- you can't find your way around PostgreSQL without such a program. Personally, I use Source Navigator which you can grab at http://sources.redhat.com/sourcenav/ . The really useful thing about source navigator is that it parses the source into functions, variables, etc. rather than just indexing it all as text. This means when you are looking at a source file with it, you can do neat things like click on a function call and then see things like the declaration and a x-ref tree. Very handy. Neil -- Neil Padgett Red Hat Canada Ltd. E-Mail: [EMAIL PROTECTED] 2323 Yonge Street, Suite #300, Toronto, ON M4P 2C9 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
[HACKERS] Locale by default?
It occurred to me that a server with locale features that is started in the C locale is going to behave the same as a server without locale features. The exception are a few extra memory moving operations. (I sincerely hope that all systems' libcs have optimized paths for the C locale.) So we could get rid of this --enable-locale switch altogether. Given our international user base, this would be an appropriate step and move the locale support out of the cumbersome secondary feature compartment. What do you think? -- Peter Eisentraut [EMAIL PROTECTED] http://funkturm.homeip.net/~peter ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Locale by default?
Hi Peter, Any idea of how many extra memory moving operations that would be? :-) Regards and best wishes, Justin Clift Peter Eisentraut wrote: It occurred to me that a server with locale features that is started in the C locale is going to behave the same as a server without locale features. The exception are a few extra memory moving operations. (I sincerely hope that all systems' libcs have optimized paths for the C locale.) So we could get rid of this --enable-locale switch altogether. Given our international user base, this would be an appropriate step and move the locale support out of the cumbersome secondary feature compartment. What do you think? -- Peter Eisentraut [EMAIL PROTECTED] http://funkturm.homeip.net/~peter ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED]) -- My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there. - Indira Gandhi ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] A fixed user id for the postgres user?
Peter Eisentraut [EMAIL PROTECTED] writes: I think the simplest fix would be to assign a fixed usesysid of 1. Slightly more flexible: make the ID number an initdb option, with a default of 1. This would let people do it the old way if they wanted. Doesn't seem very critical though. regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Locale by default?
Peter Eisentraut [EMAIL PROTECTED] writes: (I sincerely hope that all systems' libcs have optimized paths for the C locale.) So we could get rid of this --enable-locale switch altogether. Some experimental evidence to support the claim that --enable-locale has zero cost would be good before taking this step. If any hotspots turn up, we could possibly do runtime checks: if (locale_is_c()) strcmp() else strcoll() regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Locale by default?
If it's of any assistance, I'm working with the Open Source Database Benchmark guys (osdb.sourceforge.net) to get an AS3AP-based benchmark for PostgreSQL 7.1.x+ up-and-running reliably. It's working on my Mandrake Linux 8.0 system here, but I need the main OSDB guy to get back from holidays to review and commit things to their CVS. ETA of around a week from right now. :) My point is, if we've got decent benchmarking software (and we can actually freely use it), we can do real-world validation tests when considering things like Peter's suggestion. Sounds good to me. Regards and best wishes, Justin Clift Tom Lane wrote: Peter Eisentraut [EMAIL PROTECTED] writes: (I sincerely hope that all systems' libcs have optimized paths for the C locale.) So we could get rid of this --enable-locale switch altogether. Some experimental evidence to support the claim that --enable-locale has zero cost would be good before taking this step. If any hotspots turn up, we could possibly do runtime checks: if (locale_is_c()) strcmp() else strcoll() regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html -- My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there. - Indira Gandhi ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
RE: [HACKERS] User locks code
Well, ability to lock only unlocked rows in select for update is useful, of course. But uniq features of user'locks are: 1. They don't interfere with normal locks hold by session/transaction. 2. Share lock is available. 3. User can lock *and unlock objects* inside transaction, which is not (and will not be) available with locks held by transactions. They are interesting too and proposed implementation will not impact lock manager (just additional 4 bytes in LOCKTAG = same size of LOCKTAG on machines with 8 bytes alignment). An interesting method would be to allow users to simply avoid locked rows: SELECT * FROM queue FOR UPDATE LIMIT 1 UNLOCKED; Unlocked, return immediately, whatever could be used as a keyword to avoid rows that are locked (skipping over them). For update locks the row of course. Currently for the above type of thing I issue an ORDER BY random() which avoids common rows enough, the queue agent dies if queries start taking too long (showing it's waiting for other things) and tosses up new copies if it goes a while without waiting at all (showing increased load). -- Rod Taylor This message represents the official view of the voices in my head - Original Message - From: Mikheev, Vadim [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Friday, August 17, 2001 2:48 PM Subject: [HACKERS] User locks code 1. Just noted this in contrib/userlock/README.user_locks: User locks, by Massimo Dal Zotto [EMAIL PROTECTED] Copyright (C) 1999, Massimo Dal Zotto [EMAIL PROTECTED] This software is distributed under the GNU General Public License either version 2, or (at your option) any later version. Well, anyone can put code into contrib with whatever license he/she want but user locks package includes interface functions in contrib *and* changes in our lock manager, ie changes in backend code. I wonder if backend' part of package is covered by the same license above? And is it good if yes? 2. Not good implementation, imho. It's too complex (separate lock method table, etc). Much cleaner would be implement this feature the same way as transactions wait other transaction commit/abort: by locking objects in pseudo table. We could get rid of offnum and lockmethod from LOCKTAG and add struct Oid RelId; Oid ObjId; } userObjId; to objId union of LOCKTAG. This way user could lock whatever object he/she want in specified table and note that we would be able to use table access rights to control if user allowed to lock objects in table - missed in 1. One could object that 1. is good because user locks never wait. I argue that never waiting for lock is same bad as always waiting. Someday we'll have time-wait etc features for general lock method and everybody will be happy -:) Comments? Vadim P.S. I could add 2. very fast, no matter if we'll keep 1. or not. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
[HACKERS] Tool Search
We're about to start up Postgresql in production, and I am looking for a few tools, so I do not have to reinvent the wheel. I'm looking for tools that: 1) Grep through log files, looking for particulary nasty items, that should get sent to me in email or to a cell phone. I noticed you can use the syslog daemon, which would be great for sorting. Anyone have a sort list? 2) Probably, we'll use Bib Brother or Whats Up Gold to monitor system resources, like cpu, memory, and disk space. Anyone have suggestions? 3) Interesting tools any self-respecting DBA should have, that looks for particular quirks in Postgresql. For example, in Informix, we were always checking fragmentation and space issues. Naomi Walker Chief Information Officer Eldorado Computing, Inc. 602-604-3100 [EMAIL PROTECTED] ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Progress report on locale safe LIKE indexing
Peter Eisentraut wrote: Hiroshi Inoue writes: Hmm * string1 = string2 * doesn't imply * string1 LIKE string2 * ? In the current implementation of LIKE, you're right. The SQL standard allows for the possibility that "[d]epending on the collating sequence, two strings may compare as equal even if they are of different lengths or contain different sequences of characters." However, I doubt that this can really happen in practice. For example, in some collating sequences (such as en_US), characters with diacritic marks (accents) are "more equal" than others, but in the end there's always a tie breaker. Or do you know an example where this really happens? I can see the examples in a documentation M$ SQL Server though I can't try it in reality. For example ignore case(low/high) ignore accents I don't think they are strange as collating sequences. You are establishing a pretty big mechanism and I think you should clarify the assumption. Please tell me the assumption. I can think of the followings. 1) Because the current implementaion of LIKE isn't locale-aware, we should be compatible with it for ever. 2) strcoll(str1, str2) == 0 means strcmp(str1, str2) == 0 in any locale. regards, Hiroshi Inoue ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
[HACKERS] Re: [ADMIN] DB size
Hi all. I'm not a good DB admin , thats why I'm posting to this list !
How can I figure out the size of a database or table ???
It was easier in older versions of postgresql to perform a `du -h` on
the directory that corresponded to the database name. However now they
are named by some type of object id?
Would it be possible for an item to be added to the to do list to perform
a report on a database?
I.e:
\d{p|S|l|r} list permissions/system tables/lobjects/ report
Report will contain:
1. Last vacuum time.
2. Space the database is using on the file system.
3. Some other juicy statistics?
Thanks.
---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Locale by default?
It occurred to me that a server with locale features that is started in the C locale is going to behave the same as a server without locale features. The exception are a few extra memory moving operations. (I sincerely hope that all systems' libcs have optimized paths for the C locale.) So we could get rid of this --enable-locale switch altogether. Given our international user base, this would be an appropriate step and move the locale support out of the cumbersome secondary feature compartment. What do you think? I wouldn't object it if there is a way to disable locale support. We in Japan are always troubled by borken Japanese locales on some systems. I'm afraid to hear more complains if there is no way to disable the locale support. Moreover, collation of locales for Japanese are broken on all platforms as far as I know. I'm not sure about other Asian languages though. -- Tatsuo Ishii ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
[HACKERS] Suggestion for To Do List - Client timeout please.
Possibly create a timeout for psql. pg_dump, pg_restore and other clients. If they can not connect to a certain host within a certain period it will quit with an error. I have psql's still running for 6 days from crontab that could not connect to a bogus IP address. I checked the idocs and searched on google for pg_dump timeout. Thankyou. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
