Re: [HACKERS] Call for 7.5 feature completion
From the FAQ (http://www.drbd.org/316.html): Q: Can XFS be used with DRBD? A: XFS uses dynamic block size, thus DRBD 0.7 or later is needed. Hope we're talking about the same project. ;) Cheers! On Tue, 2004-05-18 at 00:16, Mario Weilguni wrote: Well that seems to be part of the problem. ext3 does not scale well at all under load. You should probably upgrade to a better FS (like XFS). I am not saying that your point isn't valid (it is) but upgrading to a better FS will help you. Thanks for the info, but I've already noticed that. XFS is no option since it does not work with drbd, but jfs seems to be quite good. Regards, Mario Weilguni ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings -- Greg Copeland, Owner [EMAIL PROTECTED] Copeland Computer Consulting 940.206.8004 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Call for 7.5 feature completion
On Tue, 2004-05-18 at 09:30, Andrew Sullivan wrote: On Sun, May 16, 2004 at 02:46:38PM -0400, Jan Wieck wrote: fact that checkpoints, vacuum runs and pg_dumps bog down their machines to the state where simple queries take several seconds care that much for any Win32 port? Do you think it is a good sign for those who have Yes. I am one such person, but from the marketing side of things, I understand perfectly well what failing to deliver on Win32 again will cost. It's not important to _me_, but that doesn't mean it's unimportant to the project. My primary fear about delivering Win32 with all of these other great features is that, IMO, there is a higher level of risk associated with these advanced features. At the same time, this will be the first trial for many Win32 users. Should there be some problems, in general, or worse, specific to Win32 users as it relates to these new features, it could cost some serious Win32/PostgreSQL community points. A troubled release which is experienced by Win32 users is going to be a battle cry for MySQL. I've been quietly hoping that these great new features would become available a release before Win32 was completed. That way, a shake down would occur before the Win32 audience got a hold of it. Which, in turn, should make for a great Win32 experience. I guess my point is, if these other features don't make it into 7.5 and Win32 does, that might still be a good thing for the potential Win32 market. Granted, if I was a developer on one of these big features and it didn't make it, I too would be fairly disappointed. Yet, once we get a release out with Win32, it should help give everyone a feel for the ability to actually support this new audience and platform. If there is a large influx of users compounded by problems, I suspect it's again, going to reflect poorly on the PostgreSQL community. ...just some ramblings -- Greg Copeland, Owner [EMAIL PROTECTED] Copeland Computer Consulting 940.206.8004 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] [GENERAL] Performance while loading data and indexing
I tend to agree with this though I have nothing to back up it with. My impression is that XFS does very well for large files. Accepting that as fact?, my impression is that XFS historically does well for database's. Again, I have nothing to back that up other than hear-say and conjecture. Greg On Thu, 2002-09-26 at 15:55, Hans-Jürgen Schönig wrote: I have seen various benchmarks where XFS seems to perform best when it comes to huge amounts of data and many files (due to balanced internal b+ trees). also, XFS seems to be VERY mature and very stable. ext2/3 don't seem to be that fast in most of the benchmarks. i did some testing with reiser some time ago. the problem is that it seems to restore a very historic consistent snapshot of the data. XFS seems to be much better in this respect. i have not tested JFS yet (but on this damn AIX beside me) from my point of view i strongly recommend XFS (maybe somebody from RedHat should think about it). Hans Neil Conway wrote: Bruce Momjian [EMAIL PROTECTED] writes: The paper does recommend ext3, but the differences between file systems are very small. Well, I only did a very rough benchmark (a few runs of pgbench), but the results I found were drastically different: ext2 was significantly faster (~50%) than ext3-writeback, which was in turn significantly faster (~25%) than ext3-ordered. Also, though ext3 is slower, turning fsync off should make ext3 function similar to ext2. Why would that be? Cheers, Neil -- *Cybertec Geschwinde u Schoenig* Ludo-Hartmannplatz 1/14, A-1160 Vienna, Austria Tel: +43/1/913 68 09; +43/664/233 90 75 www.postgresql.at http://www.postgresql.at, cluster.postgresql.at http://cluster.postgresql.at, www.cybertec.at http://www.cybertec.at, kernel.cybertec.at http://kernel.cybertec.at ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: [HACKERS] [GENERAL] Performance while loading data and indexing
On Thu, 2002-09-26 at 09:52, Shridhar Daithankar wrote: My friend argues for ext2 to eliminate journalling overhead but I favour reiserfs personally having used it in pgbench with 10M rows on paltry 20GB IDE disk for 25 tps.. We will be attempting raiserfs and/or XFS if required. I know how much speed difference exists between resiserfs and ext2. Would not be surprised if everythng just starts screaming in one go.. I'm not sure about reiserfs or ext3 but with XFS, you can create your log on another disk. Also worth noting is that you can also configure the size and number of log buffers. There are also some other performance type enhancements you can fiddle with if you don't mind risking time stamp consistency in the event of a crash. If your setup allows for it, you might want to consider using XFS in this configuration. While I have not personally tried moving XFS' log to another device, I've heard that performance gains can be truly stellar. Assuming memory allows, twiddling with the log buffering is said to allow for large strides in performance as well. If you do try this, I'd love to hear back about your results and impressions. Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] [GENERAL] Performance while loading data and indexing
On Thu, 2002-09-26 at 11:41, Bruce Momjian wrote: Shridhar Daithankar wrote: I might have found the bottleneck, although by accident. Mysql was running out of space while creating index. So my friend shut down mysql and tried to move things by hand to create links. He noticed that even things like cp were terribly slow and it hit us.. May be the culprit is the file system. Ext3 in this case. I just added a file system and multi-cpu section to my performance tuning paper: http://www.ca.postgresql.org/docs/momjian/hw_performance/ The paper does recommend ext3, but the differences between file systems are very small. If you are seeing 'cp' as slow, I wonder if it may be something more general, like poorly tuned hardware or something. You can use 'dd' to throw some data around the file system and see if that is showing slowness; compare those numbers to another machine that has different hardware/OS. Also, though ext3 is slower, turning fsync off should make ext3 function similar to ext2. That would be an interesting test if you suspect ext3. I'm curious as to why you recommended ext3 versus some other (JFS, XFS). Do you have tests which validate that recommendation or was it a simple matter of getting the warm fuzzies from familiarity? Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] [GENERAL] Performance while loading data and indexing
On Thu, 2002-09-26 at 11:41, Bruce Momjian wrote: Shridhar Daithankar wrote: I might have found the bottleneck, although by accident. Mysql was running out of space while creating index. So my friend shut down mysql and tried to move things by hand to create links. He noticed that even things like cp were terribly slow and it hit us.. May be the culprit is the file system. Ext3 in this case. I just added a file system and multi-cpu section to my performance tuning paper: http://www.ca.postgresql.org/docs/momjian/hw_performance/ The paper does recommend ext3, but the differences between file systems are very small. If you are seeing 'cp' as slow, I wonder if it may be something more general, like poorly tuned hardware or something. You can use 'dd' to throw some data around the file system and see if that is showing slowness; compare those numbers to another machine that has different hardware/OS. That's a good point. Also, if you're using IDE, you do need to verify that you're using DMA and proper PIO mode if at possible. Also, big performance improvements can be seen by making sure your IDE bus speed has been properly configured. The drivetweak-gtk and hdparm utilities can make huge difference in performance. Just be sure you know what the heck your doing when you mess with those. Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] [GENERAL] Performance while loading data and indexing
On Thu, 2002-09-26 at 16:03, Neil Conway wrote: Bruce Momjian [EMAIL PROTECTED] writes: Wow. That leaves no good Linux file system alternatives. PostgreSQL just wants an ordinary file system that has reliable recovery from a crash. I'm not really familiar with the reasoning behind ext2's reputation as recovering poorly from crashes; if we fsync a WAL record to disk before we lose power, can't we recover reliably, even with ext2? Well, I have experienced data loss from ext2 before. Also, recovery from crashes on large file systems take a very, very long time. I can't imagine anyone running a production database on an ext2 file system having 10's or even 100's of GB. Ouch. Recovery would take forever! Even recovery on small file systems (2-8G) can take extended periods of time. Especially so on IDE systems. Even then manual intervention is not uncommon. While I can't say that x, y or z is the best FS to use on Linux, I can say that ext2 is probably an exceptionally poor choice from a reliability and/or uptime perspective. Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] How to REINDEX in high volume environments?
On Sat, 2002-09-28 at 02:16, Shridhar Daithankar wrote: On 28 Sep 2002 at 17:08, Justin Clift wrote: Have moved the indexes to another drive, then created symlinks to them. Ran a benchmark against the database, REINDEX'd the tables, VACUUM FULL ANALYZE'd, prepared to re-run the benchmark again and guess what? The indexes were back on the original drive. Is there a way to allow REINDEX to work without having this side affect? Pre-creating a bunch of dangling symlinks doesn't work (tried that, it gives a ERROR: cannot create accounts_pkey: File exists on FreeBSD 4.6.2 when using the REINDEX). Looks like we should have a subdirectory in database directory which stores index. May be transaction logs, indexes goes in separte directory which can be symlinked. Linking a directory is much simpler solution than linking a file. I suggest we have per database transaction log and indexes created in separate subdirectories for each database. Furhter given that large tables are segmented after one GB size, a table should have it's own subdirectory optionally.. At the cost of few inodes, postgresql would gain much more flexibility and hence tunability.. May be TODO for 7.4? Anyone? Very neat idea! Sounds like an excellent way of gaining lots of granularity! I can't even think of a reason not to use the directory per table scheme all the time. Perhaps simply allowing for a script/tool that will automatically perform such a physical table migration to a distinct directory would be in order too. Either way, sounds like a good idea. Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] version mismatch detection doesn't work
It was I that originally brought the topic up. I don't really remember the exact details but I do seem to recall that the author thought it was a horrid idea. Basically and poorly paraphrased the response was that everyone should use select version() after they connect and if they don't know to do that or simply forget, that's tough. I also seem to recall that even the prospect of having some slash command that showed psql and back end version was considered a waste and a bad/redundant idea. So, as it stands, only the psql version is displayed. I still think it makes so much more sense to simply state something like, Welcome to psql 7.3b1, the PostgreSQL interactive terminal. You are currently connected with a 7.1.1 server named 'foobar'. It's simple and makes the information very obvious. It also helps re-enforce the name of the server that you've connected with. I should clarify, the host name par is not something I originally asked about but does seem to make sense. I honestly could care less about the exact text as making the information obviously available is all that I care really about. Personally, I never understood how making even marginally redundant information readily and obviously available, especially when it can prevent some potential peril, is a bad idea. But, for each is own. ;) Greg On Sat, 2002-09-28 at 11:28, Alvaro Herrera wrote: Tom Lane dijo: Alvaro Herrera [EMAIL PROTECTED] writes: Seems the functionality to detect old versions of the postmaster with newer psql doesn't work. What functionality? psql has never had such a test. I think you are thinking of pg_dump. No, I was thinking of psql. There was a discussion some time ago about mismatching versions; I don't know where I got the idea that the conclusion had been that if versions mismatched, psql would barf. (The conclusion was to add the version number to psql.) -- Alvaro Herrera (alvherre[a]atentus.com) No hay ausente sin culpa ni presente sin disculpa (Prov. frances) ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly signature.asc Description: This is a digitally signed message part
Re: [HACKERS] 7.2.3?
Should an advisory be issued for production sites to not perform a vacuum full with a notice that a bug fix will be coming shortly? Greg On Sat, 2002-09-28 at 13:45, Justin Clift wrote: Bruce Momjian wrote: I have seen no discussion on whether to go ahead with a 7.2.3 to add several serious fixes Tom has made to the code in the past few days. This will allow production sites to run the 7.2 series and also do VACUUM FULL won't it? If so, then the idea is already pretty good. :-) Which other fixes would be included? Regards and best wishes, Justin Clift Are we too close to 7.3 for this to be worthwhile? Certainly there will be people distributing 7.2.X for some time as 7.3 stabilizes. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html -- My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there. - Indira Gandhi ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: [HACKERS] [GENERAL] Performance while loading data and indexing
Hey, excellent. Thanks! Based on that, it appears that XFS is a pretty good FS to use. For me, the real surprise was how well reiserfs performed. Greg On Thu, 2002-10-03 at 18:09, Mike Benoit wrote: Some of you may be interested in this seemingly exhaustive benchmark between ext2/3, ReiserFS, JFS, and XFS. http://www.osdl.org/presentations/lwe-jgfs.pdf ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Threaded Sorting
I wouldn't hold your breath for any form of threading. Since PostgreSQL is process based, you might consider having a pool of sort processes which address this but I doubt you'll get anywhere talking about threads here. Greg On Fri, 2002-10-04 at 02:46, Hans-Jürgen Schönig wrote: Did anybody think about threaded sorting so far? Assume an SMP machine. In the case of building an index or in the case of sorting a lot of data there is just one backend working. Therefore just one CPU is used. What about starting a thread for every temporary file being created? This way CREATE INDEX could use many CPUs. Maybe this is worth thinking about because it will speed up huge databases and enterprise level computing. Best regards, Hans-Jürgen Schönig -- *Cybertec Geschwinde u Schoenig* Ludo-Hartmannplatz 1/14, A-1160 Vienna, Austria Tel: +43/1/913 68 09; +43/664/233 90 75 www.postgresql.at http://www.postgresql.at, cluster.postgresql.at http://cluster.postgresql.at, www.cybertec.at http://www.cybertec.at, kernel.cybertec.at http://kernel.cybertec.at ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Threaded Sorting
On Fri, 2002-10-04 at 09:40, Hans-Jürgen Schönig wrote: I had a brief look at the code used for sorting. It is very well documented so maybe it is worth thinking about a parallel algorithm. When talking about threads: A pool of processes for sorting? Maybe this could be useful but I doubt if it the best solution to avoid overhead. Somewhere in the TODO it says that there will be experiments with a threaded backend. This make me think that threads are not a big no no. Hans That was a fork IIRC. Threading is not used in baseline PostgreSQL nor is there any such plans that I'm aware of. People from time to time ask about threads for this or that and are always told what I'm telling you. The use of threads leads to portability issues not to mention PostgreSQL is entirely built around the process model. Tom is right to dismiss the notion of adding additional CPUs to something that is already I/O bound, however, the concept it self should not be dismissed. Applying multiple CPUs to a sort operation is well accepted and understood technology. At this point, perhaps Tom or one of the other core developers having insight in this area would be willing to address how readily such a mechanism could could be put in place. Also, don't be so fast to dismiss what the process model can do. There is not reason to believe that having a process pool would not be able to perform wonderful things if implemented properly. Basically, the notion would be that the backend processing the query would solicit assistance from the sort pool if one or more processes were available. At that point, several methods could be employed to divide the work. Some form of threshold would also have to be created to prevent the pool from being used when a single backend is capable of addressing the need. Basically the idea is, you only have the pool assist with large tuple counts and then, only when resources are available and resource are available from within the pool. By doing this, you avoid additional overhead for small sort efforts and gain when it matters the most. Regards, Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Threaded Sorting
On Fri, 2002-10-04 at 10:37, Hans-Jürgen Schönig wrote: My concern was that a process model might be a bit too slow for that but if we had processes in memory this would be wonderful thing. Yes, that's the point of having a pool. The idea is not only do you avoid process creation and destruction which is notoriously expensive on many platforms, they would sit idle until signaled to begin working on it's assigned sort operation. Ideally, these would be configurable options which would include items such as, pool size (maximum number of processes in the pool), max concurrency level (maximum number of process from the pool which can contribute to a single backend) and tuple count threshold (threshold which triggers solicition for assistance from the sort pool). Using it for small amounts of data is pretty useless - I totally agree but when it comes to huge amounts of data it can be useful. It is a mechanism for huge installations with a lot of data. Hans Agreed. Thus the importance of being able to specify some type of meaningful threshold. Any of the core developers wanna chime in here on this concept? Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Threaded Sorting
On Fri, 2002-10-04 at 12:26, Bruce Momjian wrote: Added to TODO: * Allow sorting to use multiple work directories Why wouldn't that fall under the table space effort??? Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Potential Large Performance Gain in WAL synching
On Fri, 2002-10-04 at 18:03, Neil Conway wrote: Curtis Faith [EMAIL PROTECTED] writes: It looks to me like BufferAlloc will simply result in a call to BufferReplace smgrblindwrt write for md storage manager objects. This means that a process will block while the write of dirty cache buffers takes place. I think Tom was suggesting that when a buffer is written out, the write() call only pushes the data down into the filesystem's buffer -- which is free to then write the actual blocks to disk whenever it chooses to. In other words, the write() returns, the backend process can continue with what it was doing, and at some later time the blocks that we flushed from the Postgres buffer will actually be written to disk. So in some sense of the word, that I/O is asynchronous. Isn't that true only as long as there is buffer space available? When there isn't buffer space available, seems the window for blocking comes into play?? So I guess you could say it is optimally asynchronous and worse case synchronous. I think the worse case situation is one which he's trying to address. At least that's how I interpret it. Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Threaded Sorting
I see. I just always assumed that it would be done as part of table space effort as it's such a defacto feature. I am curious as to why no one has commented on the other rather obvious performance enhancement which was brought up in this thread. Allowing for parallel sorting seems rather obvious and is a common enhancement yet seems to of been completely dismissed as people seem to be fixated on I/O. Go figure. Greg On Fri, 2002-10-04 at 14:02, Bruce Momjian wrote: Greg Copeland wrote: -- Start of PGP signed section. On Fri, 2002-10-04 at 12:26, Bruce Momjian wrote: Added to TODO: * Allow sorting to use multiple work directories Why wouldn't that fall under the table space effort??? Yes, but we make it a separate item so we are sure that is implemented as part of tablespaces. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Threaded Sorting
Well, that's why I was soliciting developer input as to exactly what goes on with sorts. From what I seem to be hearing, all sorts result in temp files being created and/or used. If that's the case then yes, I can understand the fixation. Of course that opens the door for it being a horrible implementation. If that's not the case, then parallel sorts still seem like a rather obvious route to look into. Greg On Fri, 2002-10-04 at 14:15, Bruce Momjian wrote: Greg Copeland wrote: -- Start of PGP signed section. I see. I just always assumed that it would be done as part of table space effort as it's such a defacto feature. I am curious as to why no one has commented on the other rather obvious performance enhancement which was brought up in this thread. Allowing for parallel sorting seems rather obvious and is a common enhancement yet seems to of been completely dismissed as people seem to be fixated on I/O. Go figure. We think we are fixated on I/O because we think that is where the delay is. Is there a reason we shouldn't think that? -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Threaded Sorting
On Fri, 2002-10-04 at 14:31, Bruce Momjian wrote: We use tape sorts, ala Knuth, meaning we sort in memory as much as possible, but when there is more data than fits in memory, rather than swapping, we write to temp files then merge the temp files (aka tapes). Right, which is what I originally assumed. On lower end systems, that works great. Once you allow that people may actually have high-end systems with multiple CPUs and lots of memory, wouldn't it be nice to allow for huge improvements on large sorts? Basically, you have two ends of the spectrum. One, where you don't have enough memory and become I/O bound. The other is where you have enough memory but are CPU bound; where potentially you have extra CPUs to spare. Seems to me they are not mutually exclusive. Unless I've missed something, the ideal case is to never use tapes for sorting. Which is saying, you're trying to optimize an already less an ideal situation (which is of course good). I'm trying to discuss making it a near ideal use of available resources. I can understand why addressing the seemingly more common I/O bound case would receive priority, however, I'm at a loss as to why the other would be completely ignored. Seems to me, implementing both would even work in a complimentary fashion on the low-end cases and yield more breathing room for the high-end cases. What am I missing for the other case to be completely ignored? Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] [GENERAL] Large databases, performance
On Thu, 2002-10-03 at 10:56, Shridhar Daithankar wrote: Well, we were comparing ext3 v/s reiserfs. I don't remember the journalling mode of ext3 but we did a 10 GB write test. Besides converting the RAID to RAID- 0 from RAID-5 might have something to do about it. There was a discussion on hackers some time back as in which file system is better. I hope this might have an addition over it.. Hmm. Reiserfs' claim to fame is it's low latency with many, many small files and that it's journaled. I've never seem anyone comment about it being considered an extremely fast file system in an general computing context nor have I seen any even hint at it as a file system for use in heavy I/O databases. This is why Reiserfs is popular with news and squid cache servers as it's almost an ideal fit. That is, tons of small files or directories contained within a single directory. As such, I'm very surprised that reiserfs is even in the running for your comparison. Might I point you toward XFS, JFS, or ext3, ? As I understand it, XFS and JFS are going to be your preferred file systems for for this type of application with XFS in the lead as it's tool suite is very rich and robust. I'm actually lacking JFS experience but from what I've read, it's a notch or two back from XFS in robustness (assuming we are talking Linux here). Feel free to read and play to find out for your self. I'd recommend that you start playing with XFS to see how the others compare. After all, XFS' specific claim to fame is high throughput w/ low latency on large and very large files. Furthermore, they even have a real time mechanism that you can further play with to see how it effects your throughput and/or latencies. Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Threaded Sorting
On Fri, 2002-10-04 at 15:07, Tom Lane wrote: the sort comparison function can be anything, including user-defined code that does database accesses or other interesting stuff. This This is something that I'd not considered. would mean that the sort auxiliary process would have to adopt the database user identity of the originating process, and quite possibly absorb a whole lot of other context information before it could correctly/safely execute the comparison function. That pushes the overhead up a lot more. Significantly! Agreed. Still, if you want to try it out, feel free ... this is an open-source project, and if you can't convince other people that an idea is worth implementing, that doesn't mean you can't implement it yourself and prove 'em wrong. No Tom, my issue wasn't if I could or could not convince someone but rather that something has been put on the table requesting additional feedback on it's feasibility but had been completely ignored. Fact is, I knew I didn't know enough about the implementation details to even attempt to convince anyone of anything. I simply wanted to explore the idea or rather the feasibility of the idea. In theory, it's a great idea. In practice, I had no idea, thus my desire to seek additional input. As such, it seems a practical implementation may prove difficult. I now understand. Thank you for taking the take to respond in a manner that satisfies my curiosity. That's all I was looking for. :) Best Regards, Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Proposed LogWriter Scheme, WAS: Potential Large
On Sat, 2002-10-05 at 14:46, Curtis Faith wrote: 2) aio_write vs. normal write. Since as you and others have pointed out aio_write and write are both asynchronous, the issue becomes one of whether or not the copies to the file system buffers happen synchronously or not. Actually, I believe that write will be *mostly* asynchronous while aio_write will always be asynchronous. In a buffer poor environment, I believe write will degrade into a synchronous operation. In an ideal situation, I think they will prove to be on par with one another with a slight bias toward aio_write. In less than ideal situations where buffer space is at a premium, I think aio_write will get the leg up. The kernel doesn't need to know anything about platter rotation. It just needs to keep the disk write buffers full enough not to cause a rotational latency. Which is why in a buffer poor environment, aio_write is generally preferred as the write is still queued even if the buffer is full. That means it will be ready to begin placing writes into the buffer, all without the process having to wait. On the other hand, when using write, the process must wait. In a worse case scenario, it seems that aio_write does get a win. I personally would at least like to see an aio implementation and would be willing to even help benchmark it to benchmark/validate any returns in performance. Surely if testing reflected a performance boost it would be considered for baseline inclusion? Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Proposed LogWriter Scheme, WAS: Potential Large
On Sun, 2002-10-06 at 11:46, Tom Lane wrote: I can't personally get excited about something that only helps if your server is starved for RAM --- who runs servers that aren't fat on RAM anymore? But give it a shot if you like. Perhaps your analysis is pessimistic. I do suspect my analysis is somewhat pessimistic too but to what degree, I have no idea. You make a good case on your memory argument but please allow me to further kick it around. I don't find it far fetched to imagine situations where people may commit large amounts of memory for the database yet marginally starve available memory for file system buffers. Especially so on heavily I/O bound systems or where sporadicly other types of non-database file activity may occur. Now, while I continue to assure myself that it is not far fetched I honestly have no idea how often this type of situation will typically occur. Of course, that opens the door for simply adding more memory and/or slightly reducing the amount of memory available to the database (thus making it available elsewhere). Now, after all that's said and done, having something like aio in use would seemingly allowing it to be somewhat more self-tuning from a potential performance perspective. Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Analysis of ganged WAL writes
On Sun, 2002-10-06 at 18:07, Tom Lane wrote: CPU loading goes from 80% idle at 1 client to 50% idle at 5 clients to 10% idle at 10 or more. So this does seem to be a nice win, and unless I hear objections I will apply it ... Wow Tom! That's wonderful! On the other hand, maybe people needed the extra idle CPU time that was provided by the unpatched code. ;) Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Proposed LogWriter Scheme, WAS: Potential Large
On Mon, 2002-10-07 at 10:38, Antti Haapala wrote: Browsed web and came across this piece of text regarding a Linux-KAIO patch by Silicon Graphics... Ya, I have read this before. The problem here is that I'm not aware of which AIO implementation on Linux is the forerunner nor do I have any idea how it's implementation or performance details defer from that of other implementations on other platforms. I know there are at least two aio efforts underway for Linux. There could yet be others. Attempting to cite specifics that only pertain to Linux and then, only with a specific implementation which may or may not be in general use is questionable. Because of this I simply left it as saying that I believe my analysis is pessimistic. Anyone have any idea of Red Hat's Advanced Server uses KAIO or what? Preliminary experience with KAIO have shown over 35% improvement in database performance tests. Unit tests (which only perform I/O) using KAIO and Raw I/O have been successful in achieving 93% saturation with 12 disks hung off 2 X 40 MB/s Ultra-Wide SCSI channels. We believe that these encouraging results are a direct result of implementing a significant part of KAIO in the kernel using split-phase I/O while avoiding or minimizing the use of any globally contented locks. The problem here is, I have no idea what they are comparing to (worse case read/writes which we know PostgreSQL *mostly* isn't suffering from). If we assume that PostgreSQL's read/write operations are somewhat optimized (as it currently sounds like they are), I'd seriously doubt we'd see that big of a difference. On the other hand, I'm hoping that if an aio postgresql implementation does get done we'll see something like a 5%-10% performance boost. Even still, I have nothing to pin that on other than hope. If we do see a notable performance increase for Linux, I have no idea what it will do for other platforms. Then, there are all of the issues that Tom brought up about bloat/uglification and maintainability. So, while I certainly do keep those remarks in my mind, I think it's best to simply encourage the effort (or something like it) and help determine where we really sit by means of empirical evidence. Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Analysis of ganged WAL writes
On Mon, 2002-10-07 at 16:06, Curtis Faith wrote: Well, too bad. If you haven't gotten your commit record down to disk, then *you have not committed*. This is not negotiable. (If you think it is, then turn off fsync and quit worrying ;-)) At this point, I think we've come full circle. Can we all agree that this concept is a *potential* source of improvement in a variety of situations? If we can agree on that, perhaps we should move to the next stage in the process, validation? How long do you think it would take to develop something worthy of testing? Do we have known test cases which will properly (in)validate the approach that everyone will agree to? If code is reasonably clean so as to pass the smell test and shows a notable performance boost, will it be seriously considered for inclusion? If so, I assume it would become a configure option (--with-aio)? Regards, Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Dirty Buffer Writing [was Proposed LogWriter Scheme]
On Mon, 2002-10-07 at 15:28, Bruce Momjian wrote: This is the trickle syncer. It prevents bursts of disk activity every 30 seconds. It is for non-fsync writes, of course, and I assume if the kernel buffers get low, it starts to flush faster. Doesn't this also increase the likelihood that people will be running in a buffer-poor environment more frequently that I previously asserted, especially in very heavily I/O bound systems? Unless I'm mistaken, that opens the door for a general case of why an aio implementation should be looked into. Also, on a side note, IIRC, linux kernel 2.5.x has a new priority elevator which is said to be MUCH better as saturating disks than ever before. Once 2.6 (or whatever it's number will be) is released, it may not be as much of a problem as it seems to be for FreeBSD (I think that's the one you're using). Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Analysis of ganged WAL writes
Well, I was thinking that aio may not be available on all platforms, thus the conditional compile option. On the other hand, wouldn't you pretty much want it either on or off for all instances? I can see that it would be nice for testing though. ;) Greg On Mon, 2002-10-07 at 16:23, Justin Clift wrote: Greg Copeland wrote: snip If so, I assume it would become a configure option (--with-aio)? Or maybe a GUC use_aio ? :-) Regards and best wishes, Justin Clift Regards, Greg Name: signature.asc signature.asc Type: application/pgp-signature Description: This is a digitally signed message part -- My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there. - Indira Gandhi signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Analysis of ganged WAL writes
On Tue, 2002-10-08 at 04:15, Zeugswetter Andreas SB SD wrote: Can the magic be, that kaio directly writes from user space memory to the disk ? Since in your case all transactions A-E want the same buffer written, the memory (not it's content) will also be the same. This would automatically write the latest possible version of our WAL buffer to disk. *Some* implementations allow for zero-copy aio. That is a savings. On heavily used systems, it can be a large savings. The problem I can see offhand is how the kaio system can tell which transaction can be safely notified of the write, or whether the programmer is actually responsible for not changing the buffer until notified of completion ? That's correct. The programmer can not change the buffer contents until notification has completed for that outstanding aio operation. To do otherwise results in undefined behavior. Since some systems do allow for zero-copy aio operations, requiring the buffers not be modified, once queued, make a lot of sense. Of course, even on systems that don't support zero-copy, changing the buffered data prior to write completion just seems like a bad idea to me. Here's a quote from SGI's aio_write man page: If the buffer pointed to by aiocbp-aio_buf or the control block pointed to by aiocbp changes or becomes an illegal address prior to asynchronous I/O completion then the behavior is undefined. Simultaneous synchronous operations using the same aiocbp produce undefined results. And on SunOS we have: The aiocbp argument points to an aiocb structure. If the buffer pointed to by aiocbp-aio_buf or the control block pointed to by aiocbp becomes an illegal address prior to asynchronous I/O completion, then the behavior is undefined. and For any system action that changes the process memory space while an asynchronous I/O is outstanding to the address range being changed, the result of that action is undefined. Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Dirty Buffer Writing [was Proposed LogWriter Scheme]
Bruce, Is there remarks along these lines in the performance turning section of the docs? Based on what's coming out of this it would seem that stressing the importance of leaving a notable (rule of thumb here?) amount for general OS/kernel needs is pretty important. Greg On Tue, 2002-10-08 at 09:50, Tom Lane wrote: (This is, BTW, one of the reasons for discouraging people from pushing Postgres' shared buffer cache up to a large fraction of total RAM; starving the kernel of disk buffers is just plain not a good idea.) signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Bison 1.50 was released
Can we please hold off until bison 1.50 becomes a defacto? It will be a matter of weeks before distros offer this as an upgrade package let alone months before distros offer this as a standard. Seems like these changes are ideal for a release after next (7.5/7.6) as enough time will of gone by for it to be much more commonly found. By not jumping on the wagon now, it will also allow more time for bugs in the wild to be caught and fixed before we force it onto the masses. Greg On Thu, 2002-10-10 at 02:05, Michael Meskes wrote: Hi, I just learned that bison 1.50 was released on Oct. 5th and it indeed compiles ecpg just nicely on my machine. Could we please install this on our main machine and merge the ecpg.big branch back into main? Michael -- Michael Meskes [EMAIL PROTECTED] Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL! ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Bison 1.50 was released
Oh, that's right. I had forgotten that it wasn't for general PostgreSQL use. Since it's a ecpg deal only, I guess I remove my objection. Greg On Thu, 2002-10-10 at 09:18, Tom Lane wrote: Greg Copeland [EMAIL PROTECTED] writes: Can we please hold off until bison 1.50 becomes a defacto? We don't have a whole lot of choice, unless you prefer releasing a broken or crippled ecpg with 7.3. In practice this only affects people who pull sources from CVS, anyway. If you use a tarball then you'll get prebuilt bison output. regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html signature.asc Description: This is a digitally signed message part
Re: [HACKERS] MySQL vs PostgreSQL.
On Fri, 2002-10-11 at 08:20, Antti Haapala wrote: Quoted from one page Because we couldn't get vacuum() to work reliable with PostgreSQL 7.1.1, I have little respect for the MySQL advocacy guys. They purposely spread misinformation. They always compare their leading edge alpha software against Postgres' year+ old stable versions. In some cases, I've seen them compare their alpha (4.x) software against 7.0. Very sad that these people can't even attempt to be honest. In the case above, since they are comparing 4.x, they should be comparing it to 7.x at least. It's also very sad that their testers don't seem to even understand something as simple as cron. If they can't understand something as simple as cron, I fear any conclusions they may arrive at throughout their testing (destined to be incorrect/invalid). MySQL supports data compression between front and back ends. This could be easily implemented, or is it already supported? Mammoth has such a feature...or at least it's been in development for a while. If I understood them correctly, it will be donated back to core sometime in the 7.5 or 7.7 series. Last I heard, their results were absolutely wonderful. I think all the other statements were misleading in the sense, that they compared their newest product with PostgreSQL 7.1.1. Ya, historically, they go out of their way to ensure unfair comparisons. I have no respect for them. They could be provided one... ;-) In other words, they need a list of features that they can one day hope to add to MySQL. Upgrading MySQL Server is painless. When you are upgrading MySQL Server, you don't need to dump/restore your data, as you have to do with most PostgreSQL upgrades. Ok... this is true, but not so hard - yesterday I installed 7.3b2 onto my linux box. Of course PostgreSQL isn't yet as fast as it could be. ;) I consider this par for the course. This is something I've had to do with Sybase, Oracle and MSSQL. Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Peer to peer replication of Postgresql databases
Well, not scalable doesn't have to mean not good. That's why I asked. Considering this is one of the problems with mosix clusters (process migration and associated restrictions) and the nature of PostgreSQL's implementation I'm not sure what other result may of been expected. Because of that, I wasn't sure if something else was being implied. Greg On Fri, 2002-10-11 at 08:40, Shridhar Daithankar wrote: On 11 Oct 2002 at 8:30, Greg Copeland wrote: I'd be curious to hear in a little more detail what constitutes not good for postgres on a mosix cluster. On Fri, 2002-10-11 at 06:15, Anuradha Ratnaweera wrote: On Fri, Oct 11, 2002 at 04:29:53PM +0530, Shridhar Daithankar wrote: Have already tested postgres on a mosix cluster, and as expected results are not good. (although mosix does the correct thing in keeping all the database backend processes on one node). Well, I guess in kind of replication we are talking here, the performance will be enhanced only if separate instances of psotgresql runs on separate machine. Now if mosix kernel applies some AI and puts all of them on same machine, it isn't going to be any good for the purpose replication is deployed. I guess that's what she meant.. Bye Shridhar -- User n.: A programmer who will believe anything you tell him. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Vacuum improvement
That a good idea. That way, if your database slows during specific windows in time, you can vacuum larger sizes, etc. Seemingly would help you better manage your vacuuming against system loading. Greg On Tue, 2002-10-15 at 19:22, Gavin Sherry wrote: Hi all, I'm thinking that there is an improvement to vacuum which could be made for 7.4. VACUUM FULLing large, heavily updated tables is a pain. There's very little an application can do to minimise dead-tuples, particularly if the table is randomly updated. Wouldn't it be beneficial if VACUUM could have a parameter which specified how much of the table is vacuumed. That is, you could specify: VACUUM FULL test 20 precent; Yes, terrible syntax but regardless: this would mean that we could spread the vacuum out and not, possibly, be backing up queues. ANALYZE could be modified, if necessary. Thoughts? Gavin ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Postgresql and multithreading
On Wed, 2002-10-16 at 01:27, Shridhar Daithankar wrote: Well, slow adoption rate is attributed to 'apache 1.3.x is good enough for us' syndrome, as far as I can see from news. Once linux distros start shipping with apache 2.x series *only*, the upgrade cycle will start rolling, I guess. I think that's part of it. I think the other part is that by the time you're getting to huge r/s numbers, typical web site bandwidth is already used up. So, what's the point in adding more breathing room when you don't have the bandwidth to use it anyways. Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Vacuum improvement
On Wed, 2002-10-16 at 02:29, Gavin Sherry wrote: On 16 Oct 2002, Hannu Krosing wrote: On Wed, 2002-10-16 at 05:22, Gavin Sherry wrote: Hi all, I'm thinking that there is an improvement to vacuum which could be made for 7.4. VACUUM FULLing large, heavily updated tables is a pain. There's very little an application can do to minimise dead-tuples, particularly if the table is randomly updated. Wouldn't it be beneficial if VACUUM could have a parameter which specified how much of the table is vacuumed. That is, you could specify: VACUUM FULL test 20 precent; What about VACUUM FULL test WORK 5 SLEEP 50; meaning to VACUUM FULL the whole table, but to work in small chunks and relaese all locks and let others access the tables between these ? Great idea. I think this could work as a complement to the idea I had. To answer Tom's question, how would we know what we've vacuumed, we could store the range of tids we've vacuumed in pg_class. Or, we could store the block offset of where we left off vacuuming before and using stats, run for another X% of the heap. Is this possible? Why couldn't you start your % from the first rotten/dead tuple? Just reading through trying to find the first tuple to start counting from wouldn't hold locks would it? That keeps you from having to track stats and ensures that X% of the tuples will be vacuumed. Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Vacuum improvement
But doesn't the solution I offer present a possible work around? The table wouldn't need to be locked (I think) until the first dead tuple were located. After that, you would only keep the locks until you've scanned X% of the table and shrunk as needed. The result, I think, results in incremental vacuuming with shorter duration locks being held. It's not ideal (locks) but may shorten the duration behind help by locks. I'm trying to figure out if the two approaches can't be combined somehow. That is, a percent with maybe even a max lock duration? Greg On Wed, 2002-10-16 at 11:33, David Walker wrote: Vacuum full locks the whole table currently. I was thinking if you used a similar to a hard drive defragment that only 2 rows would need to be locked at a time. When you're done vacuum/defragmenting you shorten the file to discard the dead tuples that are located after your useful data. There might be a need to lock the table for a little while at the end but it seems like you could reduce that time greatly. I had one table that is heavily updated and it grew to 760 MB even with regular vacuuming. A vacuum full reduced it to 1.1 MB. I am running 7.2.0 (all my vacuuming is done by superuser). On Wednesday 16 October 2002 09:30 am, (Via wrote: On Wed, 2002-10-16 at 02:29, Gavin Sherry wrote: On 16 Oct 2002, Hannu Krosing wrote: On Wed, 2002-10-16 at 05:22, Gavin Sherry wrote: Hi all, I'm thinking that there is an improvement to vacuum which could be made for 7.4. VACUUM FULLing large, heavily updated tables is a pain. There's very little an application can do to minimise dead-tuples, particularly if the table is randomly updated. Wouldn't it be beneficial if VACUUM could have a parameter which specified how much of the table is vacuumed. That is, you could specify: VACUUM FULL test 20 precent; What about VACUUM FULL test WORK 5 SLEEP 50; meaning to VACUUM FULL the whole table, but to work in small chunks and relaese all locks and let others access the tables between these ? Great idea. I think this could work as a complement to the idea I had. To answer Tom's question, how would we know what we've vacuumed, we could store the range of tids we've vacuumed in pg_class. Or, we could store the block offset of where we left off vacuuming before and using stats, run for another X% of the heap. Is this possible? Why couldn't you start your % from the first rotten/dead tuple? Just reading through trying to find the first tuple to start counting from wouldn't hold locks would it? That keeps you from having to track stats and ensures that X% of the tuples will be vacuumed. Greg ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Memory leaks
On Tue, 2002-10-22 at 22:28, Tom Lane wrote: Greg Copeland [EMAIL PROTECTED] writes: So again, I'm not really sure it they are meaningful at this point. psql might well have some internal leaks; the backend memory-context design doesn't apply to it. Okay. Thanks. I'll probably take another look at it a little later and report back if I find anything of value. Does that mean they are not using palloc/pfree stuff? Not everywhere. plpgsql is full of malloc's and I think the other PL modules are too --- and that's not to mention the allocation policies of the perl, tcl, etc, language interpreters. We could use a thorough review of that whole area. Okay. I've started looking at plpython to better understand it's memory needs. I'm seeing a mix of mallocs and PLy_malloc. The PLy version is basically malloc which also checks and reports on memory allocation errors. Anyone know if the cases where malloc was used was purposely done so for performance reasons or simply the flavor or the day? I thinking for starters, the plpython module could be normalized to use the PLy_malloc stuff across the board. Then again, I still need to spend some more time on it. ;) Well, the thing that really got my attention is that dmalloc is reporting frees on null pointers. AFAIK that would dump core on many platforms (it sure does here...), so I don't think I believe it without seeing chapter and verse. I actually expected it to do that here on my x86-Linux platform but a quick check showed that it was quiet happy with it. What platforms are you using -- just curious? But if you can point out where it's really happening, then we must fix it. I'll trying track this down some more this coming week to see if this is really occurring. After thinking about it, I'm not sure why dmalloc would ever report a free on null if it were not actually occurring. After all, any call to free should still be showing some memory address (valid or otherwise). Off the top of my head, I can't think of an artifact that would cause it to falsely report it. Greg ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Memory leaks
On Wed, 2002-10-23 at 08:48, Tom Lane wrote: Greg Copeland [EMAIL PROTECTED] writes: Okay. I've started looking at plpython to better understand it's memory needs. I'm seeing a mix of mallocs and PLy_malloc. The PLy version is basically malloc which also checks and reports on memory allocation errors. Anyone know if the cases where malloc was used was purposely done so for performance reasons or simply the flavor or the day? Probably either oversight or the result of different people's different coding styles. My local copy has this changed to PLy stuff now. Testing shows it's good...then again, I didn't really expect it to change anything. I'll submit patches later. I thinking for starters, the plpython module could be normalized to use the PLy_malloc stuff across the board. Then again, I still need to spend some more time on it. ;) Consistency is good. What I'd wonder about, though, is whether you shouldn't be using palloc ;-). malloc, with or without a PLy_ wrapper, doesn't provide any leverage to help you get rid of stuff when you don't want it anymore. Ya, I'm currently looking to see how the memory is being used and why. I'm trying to better understand it's life cycle. You implying that even the short term memory should be using the palloc stuff? What about long term? Blanket statement that pretty much all the PLy stuff should really be using palloc? Well, the thing that really got my attention is that dmalloc is reporting frees on null pointers. AFAIK that would dump core on many platforms (it sure does here...), I have to take that back: I was thinking about pfree() not free(). The ANSI C spec says that free(NULL) is a legal no-op, and there are Oh really. I didn't realize that. I've been using the if( ptr ) stuff for so long I didn't realize I didn't need to anymore. Thanks for the update. That was, of course, the cause for alarm. It's probably pointless to try to convince people to change that coding style. Well at this late time, I think it's safe to say that it's not causing problems for anyone on any of the supported platforms. So I'll not waste time looking for it even though I happen think it's a poor practice just the same. Thanks, Greg ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] PREPARE / EXECUTE
If you were using them that frequently, couldn't you just keep a persistent connection? If it's not used that often, wouldn't the overhead of preparing the query following a new connection become noise? Greg On Wed, 2002-10-23 at 09:24, Hans-Jürgen Schönig wrote: First of all PREPARE/EXECUTE is a wonderful thing to speed up things significantly. I wonder if there is a way to store a parsed/rewritten/planned query in a table so that it can be loaded again. This might be useful when it comes to VERY complex queries ( 10 tables). I many applications the situation is like that: a. The user connects to the database. b. The user sends various different queries to the server (some might be the same) c. The user disconnects. If there was a way to store execution plans in a table the user could load the execution plans of the most time consuming stuff into the backend without parsing and optimizing it every time he authenticates. Does it sound useful to anybody? Is it possible to do it or are there some technical problems? Maybe this is worth thinking about. Hans -- *Cybertec Geschwinde u Schoenig* Ludo-Hartmannplatz 1/14, A-1160 Vienna, Austria Tel: +43/1/913 68 09; +43/664/233 90 75 www.postgresql.at http://www.postgresql.at, cluster.postgresql.at http://cluster.postgresql.at, www.cybertec.at http://www.cybertec.at, kernel.cybertec.at http://kernel.cybertec.at ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] PREPARE / EXECUTE
Could you use some form of connection proxy where the proxy is actually keeping persistent connections but your application is making transient connections to the proxy? I believe this would result in the desired performance boost and behavior. Now, the next obvious question...anyone know of any proxy apps available for postgresql? Regards, Greg On Wed, 2002-10-23 at 11:04, Hans-Jürgen Schönig wrote: The idea is not to have it accross multiple backends and having it in sync with the tables in the database. This is not the point. My problem is that I have seen many performance critical applications sending just a few complex queries to the server. The problem is: If you have many queries where the relation time planner/time executor is very high (eg. complex joins with just one value as the result). These applications stay the same for a long time (maybe even years) and so there is no need to worry about new tables and so forth - maybe there is not even a need to worry about new data. In these cases we could speed up the database significantly just by avoiding the use of the planner: An example: I have a join across 10 tables + 2 subselects across 4 tables on the machine I use for testing: planner: 12 seconds executor: 1 second The application will stay the same forever. I could be 10 times faster if there was a way to load the execution plan into the backend. There is no way to use a persistent connection (many clients on different machines, dynamic IPs, etc. ...) There is no way to have an invalid execution plan because there are no changes (new tables etc.) in the database. Also: If people execute a prepared query and it fails they will know why - queries will fail if people drop a table even if these queries are not prepared. A new feature like the one we are discussing might be used rarely but if people use it they will benefit A LOT. If we had a simple ASCII interface to load the stuff into the planner people could save MANY cycles. When talking about tuning it is nice to gain 10% or even 20% but in many cases it does not solve a problem - if a problem can be reduced by 90% it is a REAL gain. Gaining 10% can be done by tweaking the database a little - gaining 1000% cannot be done so it might be worth thinking about it even it the feature is only used by 20% of those users out there. 20% of all postgres users is most likely more than 15.000 people. Again; it is not supposed to be a every-day solution. It is a solution for applications staying the same for a very long time. Hans Tom Lane wrote: =?ISO-8859-1?Q?Hans-J=FCrgen_Sch=F6nig?= [EMAIL PROTECTED] writes: I wonder if there is a way to store a parsed/rewritten/planned query in a table so that it can be loaded again. The original version of the PREPARE patch used a shared-across-backends cache for PREPAREd statements. We rejected that for a number of reasons, one being the increased difficulty of keeping such a cache up to date. I think actually storing the plans on disk would have all the same problems, but worse. regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED]) -- *Cybertec Geschwinde u Schoenig* Ludo-Hartmannplatz 1/14, A-1160 Vienna, Austria Tel: +43/1/913 68 09; +43/664/233 90 75 www.postgresql.at http://www.postgresql.at, cluster.postgresql.at http://cluster.postgresql.at, www.cybertec.at http://www.cybertec.at, kernel.cybertec.at http://kernel.cybertec.at ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] Memory leaks
I've started playing a little with Postgres to determine if there were memory leaks running around. After some very brief checking, I'm starting[1] to think that the answer is yes. Has anyone already gone through a significant effort to locate and eradicate memory leaks? Is this done periodically? If so, what tools are others using? I'm currently using dmalloc for my curiosity. [1] Not sure yet as I'm really wanting to find culumative leaks rather than one shot allocations which are simply never freed prior to process termination. Regards, Greg Copeland ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Memory leaks
On Tue, 2002-10-22 at 17:09, Tom Lane wrote: Greg Copeland [EMAIL PROTECTED] writes: I've started playing a little with Postgres to determine if there were memory leaks running around. After some very brief checking, I'm starting[1] to think that the answer is yes. Has anyone already gone through a significant effort to locate and eradicate memory leaks? Yes, this has been dealt with before. What tools, aside from noggin v1.0, did they use? Do we know? Have you read src/backend/utils/mmgr/README? Yes...but it's been some time since I last opened it. It was because I knew there were some caveats that I wasn't attempting to claim for sure that there were leaks. I then moved on to psql, again, just for fun. Here, I'm thinking that I started to find some other leaks...but again, I've not spent any real time on it. So again, I'm not really sure it they are meaningful at this point. AFAIK the major problems these days are not in the backend as a whole, but in the lesser-used PL language modules (cf recent discussions). Ya, that's what made me wonder about the topic on a larger scale. plpgsql has some issues too, I suspect, but not as bad as pltcl etc. Possibly the best answer is to integrate the memory-context notion into those modules; if they did most of their work in a temp context that could be freed once per PL statement or so, the problems would pretty much go away. Interesting. Having not looked at memory management schemes used in the pl implementations, can you enlighten me by what you mean by integrate the memory-context notion? Does that mean they are not using palloc/pfree stuff? It's fairly difficult to get anywhere with standard leak-tracking tools, since they don't know anything about palloc. What's worse, it is *not* a bug for a routine to palloc space it never pfrees, if it knows that it's palloc'ing in a short-lived memory context. The fact that a context may be released with much still-allocated memory in it is not a bug but a feature; but try teaching that to any standard leak checker... regards, tom lane Well, the thing that really got my attention is that dmalloc is reporting frees on null pointers. While that may be safe on specific platforms, IIRC, it's actually undefined. Again, this is as reported by dmalloc so I've yet to actually track it down to determine if it's possibly something else going on (magic voodoo of some heap manager, etc). Greg ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] idle connection timeout ...
On Fri, 2002-10-25 at 00:52, Marc G. Fournier wrote: Ya, I've thought that one through ... I think what I'm more looking at is some way of 'limiting' persistent connections, where a server opens n connections during a spike, which then sit idle indefinitely since it was one fo those 'slashdot effect' kinda spikes ... Is there any way of the 'master process' *safely/accurately* knowing, through the shared memory link, the # of connections currently open to a particular database? So that a limit could be set on a per db basis, say as an additional arg to pg_hba.conf? Well, if you're application is smart enough to know it needs to dynamically add connections, it should also be smart enough to tear them down after some idle period. I agree with Tom. I think that sounds like application domain. Greg ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Peer to peer replication of Postgresql databases
I'd be curious to hear in a little more detail what constitutes not good for postgres on a mosix cluster. Greg On Fri, 2002-10-11 at 06:15, Anuradha Ratnaweera wrote: On Fri, Oct 11, 2002 at 04:29:53PM +0530, Shridhar Daithankar wrote: Well, I don't think adding support for multiple slaves to usogres would be that problematic. Of course if you want to load balance your application queries, application has to be aware of that. I will not do sending requests to a mosix cluster anyway. Have already tested postgres on a mosix cluster, and as expected results are not good. (although mosix does the correct thing in keeping all the database backend processes on one node). Anuradha -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) Remember: Silly is a state of Mind, Stupid is a way of Life. -- Dave Butler ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Postgresql and multithreading
On Thu, 2002-10-17 at 22:20, Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Let me add one more thing on this thread. This is one email in a long list of Oh, gee, you aren't using that wizz-bang new sync/thread/aio/raid/raw feature discussion where someone shows up and wants to know why. Does anyone know how to address these, efficiently? Simple: respond to 'em all with a one-line answer: convince us why we should use it. The burden of proof always seems to fall on the wrong end in these discussions. regards, tom lane That may be easier said that done. If you don't know what the objections are, it's hard to argue your case. If you do know and understand the objections, chances are you already know the code very well and/or have the mailing lists for a very long time. This basically means, you don't want to hear from anyone unless they are one with the code. That seems and sounds very anti-open source. After it's all said and done, I think you guys are barking up the wrong tree. Open Source is all about sharing ideas. Many times I've seen ideas expressed here that were not exact hits yet help facilitate discussion, understanding on the topics in general and in some cases may even spur other ideas or associated code fixes/improvements. When I first started on this list, I was scolded rather harshly for not asking all of my questions on the list. Originally, I was told to ask reasonable questions so that everyone can learn. Now, it seems, that people don't want to answer questions at all as it's bothering the developers. Commonly asked items, such as threading, seems like they are being addressed rather well without core developer participation. Right now, I'm not seeing any down sides to what's currently in place. If the core developers still feel like they are spending more time then they like, then perhaps those that following the mailing list can step forward a little more to address general questions and defer when needed. The topic, such as threading, was previously addressed yet people still followed up on the topic. Perhaps those that don't want to be bothered should allow more time for others to address the topic and leave it alone once it has been addressed. That alone seems like it would be a huge time saver for the developers and a better use of resources. Greg signature.asc Description: This is a digitally signed message part
Re: [HACKERS] pgAdmin III (Was: Request for supported platforms)
Since you're using wxWindows, I *HIGHLY* recommend obtaining a license to wxDesigner from http://www.roebling.de/. It allows for very rapid GUI design. It also understands various sizers and makes it SOOO much easier to make use of them. Once you understand sizers, you'll love them but they are somewhat hard to use without a tool to assist. Also, for the about box, I also suggest that you make use of a wxHTML dialog. The cool thing about doing this is that you can create links and even place nice images within the about box. Plus, maintaining the about box can be easily done via an HTML resource file versus having to update code and recompile. If you have questions on wxWindows and/or wxPython, please let me know. I've been a long time user. Also, if you're willing, I also have some constructive criticism on the code. Since this is obviously going off topic as it relates to PostgreSQL, it probably wouldn't be appropriate to followup on the mailing list. Best Regards, Greg Copeland On Wed, 2002-10-30 at 02:19, Dave Page wrote: -Original Message- From: Greg Copeland [mailto:greg;copelandconsulting.net] Sent: 30 October 2002 01:08 To: Dave Page Subject: Re: [HACKERS] Request for supported platforms C++? Really? What GUI toolkit is being used? Just curious. wxWindows. The CVS is online if anyone is interested in taking a peek - it's at http://cvs.pgadmin.org/. The code is in the pgadmin3 module. Please bear in mind though, that Mark I have only been using C++ for about a month (though things are coming along very quickly), so there may be some nasties in our code. Any constructive comments from any of the -hackers would be more than welcome. Regards, Dave. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED] ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Auto Vacuum Daemon (again...)
On Fri, 2002-11-29 at 07:19, Shridhar Daithankar wrote: On 29 Nov 2002 at 7:59, Matthew T. O'Connor wrote: On Thursday 28 November 2002 23:26, Shridhar Daithankar wrote: On 28 Nov 2002 at 10:45, Tom Lane wrote: This is almost certainly a bad idea. vacuum is not very processor-intensive, but it is disk-intensive. Multiple vacuums running at once will suck more disk bandwidth than is appropriate for a background operation, no matter how sexy your CPU is. I can't see any reason to allow more than one auto-scheduled vacuum at a time. Hmm.. We would need to take care of that as well.. Not sure what you mean by that, but it sounds like the behaviour of my AVD (having it block until the vacuum command completes) is fine, and perhaps preferrable. Right.. But I will still keep option open for parallel vacuum which is most useful for reusing tuples in shared buffers.. And stale updated tuples are what causes performance drop in my experience.. You know.. just enough rope to hang themselves..;-) Right. This is exactly what I was thinking about. If someone shoots their own foot off, that's their problem. The added flexibility seems well worth it. Greg ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Auto Vacuum Daemon (again...)
On Fri, 2002-11-29 at 06:59, Matthew T. O'Connor wrote: On Thursday 28 November 2002 23:26, Shridhar Daithankar wrote: On 28 Nov 2002 at 10:45, Tom Lane wrote: Matthew T. O'Connor [EMAIL PROTECTED] writes: interesting thought. I think this boils down to how many knobs do we need to put on this system. It might make sense to say allow upto X concurrent vacuums, a 4 processor system might handle 4 concurrent vacuums very well. This is almost certainly a bad idea. vacuum is not very processor-intensive, but it is disk-intensive. Multiple vacuums running at once will suck more disk bandwidth than is appropriate for a background operation, no matter how sexy your CPU is. I can't see any reason to allow more than one auto-scheduled vacuum at a time. Hmm.. We would need to take care of that as well.. Not sure what you mean by that, but it sounds like the behaviour of my AVD (having it block until the vacuum command completes) is fine, and perhaps preferrable. I can easily imagine larger systems with multiple CPUs and multiple disk and card bundles to support multiple databases. In this case, I have a hard time figuring out why you'd not want to allow multiple concurrent vacuums. I guess I can understand a recommendation of only allowing a single vacuum, however, should it be mandated that AVD will ONLY be able to perform a single vacuum at a time? Greg ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] 7.4 Wishlist
On Tue, 2002-12-10 at 09:36, Stephen L. wrote: 6. Compression between client/server interface like in MySQL Mammoth is supposed to be donating their compression efforts back to this project, or so I've been told. I'm not exactly sure of their time-line as I've slept since my last conversation with them. The initial feedback that I've gotten back from them on this subject is that the compression has been working wonderfully for them with excellent results. IIRC, in their last official release, they announced their compression implementation. So, I'd think that it would be available for 7.4 of 7.5 time frame. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [mail] Re: [HACKERS] 7.4 Wishlist
On Tue, 2002-12-10 at 11:25, Al Sutton wrote: Would it be possible to make compression an optional thing, with the default being off? I'm not sure. You'd have to ask Command Prompt (Mammoth) or wait to see what appears. What I originally had envisioned was a per database and user permission model which would better control use. Since compression can be rather costly for some use cases, I also envisioned it being negotiated where only the user/database combo with permission would be able to turn it on. I do recall that compression negotiation is part of the Mammoth implementation but I don't know if it's a simple capability negotiation or part of a larger scheme. The reason I originally imagined a user/database type approach is because I would think only a subset of a typical installation would be needing compression. As such, this would help prevent users from arbitrarily chewing up database CPU compressing data because: o datasets are uncompressable or poorly compresses o environment cpu is at a premium o is in a bandwidth rich environment I'm in a position that many others may be in where the link between my app server and my database server isn't the bottleneck, and thus any time spent by the CPU performing compression and decompression tasks is CPU time that is in effect wasted. Agreed. This is why I'd *guess* that Mammoth's implementation does not force compression. If a database is handling numerous small queries/updates and the request/response packets are compressed individually, then the overhead of compression and decompression may result in slower performance compared to leaving the request/response packets uncompressed. Again, this is where I'm gray on their exact implementation. It's possible they implemented a compressed stream even though I'm hoping they implemented a per packet compression scheme (because adaptive compression becomes much more capable and powerful; in both algorithmically and logistical use). An example of this would be to avoid any compression on trivially sized result sets. Again, this is another area where I can imagine some tunable parameters. Just to be on the safe side, I'm cc'ing Josh Drake at Command Prompt (Mammoth) to see what they can offer up on it. Hope you guys don't mind. Greg - Original Message - From: Greg Copeland [EMAIL PROTECTED] To: Stephen L. [EMAIL PROTECTED] Cc: PostgresSQL Hackers Mailing List [EMAIL PROTECTED] Sent: Tuesday, December 10, 2002 4:56 PM Subject: [mail] Re: [HACKERS] 7.4 Wishlist On Tue, 2002-12-10 at 09:36, Stephen L. wrote: 6. Compression between client/server interface like in MySQL Mammoth is supposed to be donating their compression efforts back to this project, or so I've been told. I'm not exactly sure of their time-line as I've slept since my last conversation with them. The initial feedback that I've gotten back from them on this subject is that the compression has been working wonderfully for them with excellent results. IIRC, in their last official release, they announced their compression implementation. So, I'd think that it would be available for 7.4 of 7.5 time frame. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [mail] Re: [HACKERS] 7.4 Wishlist
On Tue, 2002-12-10 at 13:38, Bruce Momjian wrote: I haven't heard anything about them contributing it. Doesn't mean it will not happen, just that I haven't heard it. This was in non-mailing list emails that I was told this by Joshua Drake at Command Prompt. Of course, that doesn't have to mean it will be donated for sure but nonetheless, I was told it will be. Here's a quote from one of the emails. I don't think I'll be too far out of line posting this. On August 9, 2002, Joshua Drake said, One we plan on releasing this code to the developers after 7.3 comes out. We want to be good members of the community but we have to keep a slight commercial edge (wait to you see what we are going to do to vacuum). Obviously, I don't think that was official speak, so I'm not holding them to the fire, nonetheless, that's what was said. Additional follow ups did seem to imply that they were very serious about this and REALLY want to play nice as good shared source citizens. I am not excited about per-db/user compression because of the added complexity of setting it up, and even set up, I can see cases where some queries would want it, and others not. I can see using GUC to control this. If you enable it and the client doesn't support it, it is a no-op. We have per-db and per-user settings, so GUC would allow such control if you wish. I never thought beyond the need for what form an actual implementation of this aspect would look like. The reason for such a concept would be to simply limit the number of users that can be granted compression. If you have a large user base all using compression or even a small user base where very large result sets are common, I can imagine your database server becoming CPU bound. The database/user thinking was an effort to allow the DBA to better manage the CPU effect. Ideally, it would be a tri-valued parameter, that is ON, OFF, or AUTO, meaning it would determine if there was value in the compression and do it only when it would help. Yes, that makes sense and was something I had originally envisioned. Simply stated, some installations may never want compression while others may want it for every connection. Beyond that, I believe there needs to be something of a happy medium where a DBA can better control who and what is taking his CPU away (e.g. only that one remote location being fed via ISDN). If GUC can fully satisfy, I certainly won't argue against it. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Auto Vacuum Daemon (again...)
On Tue, 2002-12-10 at 13:09, scott.marlowe wrote: On 10 Dec 2002, Rod Taylor wrote: Perhaps a more appropriate rule would be 1 AVD per tablespace? Since PostgreSQL only has a single tablespace at the moment But Postgresql can already place different databases on different data stores. I.e. initlocation and all. If someone was using multiple SCSI cards with multiple JBOD or RAID boxes hanging off of a box, they would have the same thing, effectively, that you are talking about. So, someone out there may well be able to use a multiple process AVD right now. Imagine m databases on n different drive sets for large production databases. That's right. I always forget about that. So, it seems, regardless of the namespace effort, we shouldn't be limiting the number of concurrent AVD's. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] PQnotifies() in 7.3 broken?
Seems like a mistake was made. Let's (don't ya love how that sounds like I'm actually involved in the fix? ;) fix it sooner rather than later. Just curious, after a release, how come the numbers are not automatically bumped to ensure this type thing gets caught sooner rather than later? Is it possible to automate this as part of the build process so that they get grabbed from some version information during the build? Greg On Tue, 2002-12-10 at 17:36, Bruce Momjian wrote: OK, seeing that we don't have a third number, do people want me to increment the interface numbers for 7.3.1, or just wait for the increment in 7.4? --- Peter Eisentraut wrote: Tom Lane writes: It is not real clear to me whether we need a major version bump, rather than a minor one. We *do* need to signal binary incompatibility. Who can clarify the rules here? Strictly speaking, it's platform-dependent, but our shared library code plays a bit of abuse with it. What it comes down to is: If you change or remove an interface, increment the major version number. If you add an interface, increment the minor version number. If you did neither but changed the source code at all, increment the third version number, if we had one. To be thoroughly amused, read the libtool source. Grep for 'version_type'. -- Peter Eisentraut [EMAIL PROTECTED] -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] [mail] Re: 7.4 Wishlist
This has been brought up a couple of times now. Feel free to search the old archives for more information. IIRC, it would of made the implementation more problematic, or so I think it was said. When I originally brought the topic (compression) up, it was not well received. As such, it may of been thought that additional effort on such an implementation would not be worth the return on a feature which most seemingly didn't see any purpose in supporting in the first place. You need to keep in mind that many simply advocated using a compressing ssh tunnel. Seems views may of changed some since then so it may be worth revisiting. Admittedly, I have no idea what would be required to move the toast data all the way through like that. Any idea? Implementing a compression stream (which seems like what was done for Mammoth) or even packet level compression were both something that I could comfortably put my arms around in a timely manner. Moving toast data around wasn't. Greg On Tue, 2002-12-10 at 18:45, Kyle wrote: Without getting into too many details, why not send toast data to non-local clients? Seems that would be the big win. The data is already compressed, so the server wouldn't pay cpu time to recompress anything. And since toast data is relatively large anyway, it's the stuff you'd want to compress before putting it on the wire anyway. If this is remotely possible let me know, I might be interested in taking a look at it. -Kyle Bruce Momjian wrote: I am not excited about per-db/user compression because of the added complexity of setting it up, and even set up, I can see cases where some queries would want it, and others not. I can see using GUC to control this. If you enable it and the client doesn't support it, it is a no-op. We have per-db and per-user settings, so GUC would allow such control if you wish. Ideally, it would be a tri-valued parameter, that is ON, OFF, or AUTO, meaning it would determine if there was value in the compression and do it only when it would help. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] [INTERFACES] Patch for DBD::Pg pg_relcheck problem
Perhaps compression should be added to the list of protocol changes. This way, we can allow for per packet evaluation for compression. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting On Tue, 2002-12-10 at 21:50, Bruce Momjian wrote: Tom Lane wrote: Ian Barwick [EMAIL PROTECTED] writes: Sounds good to me. Is it on the todo-list? (Couldn't see it there). Probably not; Bruce for some reason has resisted listing protocol change desires as an identifiable TODO category. There are a couple of threads in the pghackers archives over the last year or so that discuss the different things we want to do, though. (Improving the error-reporting framework and fixing the COPY protocol are a couple of biggies I can recall offhand.) Listing protocol changes seemed too low-level for the TODO list, but I have kept the email messages. Today I updated the TODO list and added a section for them. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] PQnotifies() in 7.3 broken?
But it's something they should of already had to do. We're just paying late for old sins. ;) Greg On Thu, 2002-12-12 at 23:34, Bruce Momjian wrote: Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: OK, so what do we do with 7.3.1. Increment major or minor? Major. I thought you did it already? I did only minor, which I knew was safe. Do folks realize this will require recompile of applications by 7.3 users moving to 7.3.1? That seems very drastic, and there have been very few problem reports about the NOTIFY change. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Big 7.4 items
On Fri, 2002-12-13 at 04:53, Hannu Krosing wrote: On Fri, 2002-12-13 at 06:22, Bruce Momjian wrote: I wanted to outline some of the big items we are looking at for 7.4: Point-In-Time Recovery (PITR) J. R. Nield did a PITR patch late in 7.3 development, and Patrick MacDonald from Red Hat is working on merging it into CVS and adding any missing pieces. Patrick, do you have an ETA on that? How hard would it be to extend PITR for master-slave (hot backup) repliaction, which should then amount to continuously shipping logs to slave and doing nonstop PITR there :) It will never be usable for multi-master replication, but somehow it feels that for master-slave replication simple log replay would be most simple and robust solution. I'm curious, what would be the recovery strategy for PITR master-slave replication should the master fail (assuming hot fail over/backup)? A simple dump/restore? Are there/is there any facilities in PorstgreSQL for PITR archival which prevents PITR logs from be recycled (or perhaps, simply archived off)? What about PITR streaming to networked and/or removable media? -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Big 7.4 items
I must of miscommunicated here as you're describing PITR replication. I'm asking about a master failing and the slaving picking up. Now, some n-time later, how do you recover your master system to be back in sync with the slave. Obviously, I'm assuming some level of manual recovery. I'm wondering what the general approach would be. Consider that on the slave which is now the active server (master dead), it's possible that the slave's PITR's will be recycled before the master can come back up. As such, unless there is a, an archival process for PITR or b, a method of streaming PITR's off for archival, the odds of using PITR to recover the master (resync if you will) seem greatly reduced as you will be unable to replay PITR on the master for synchronization. Greg On Mon, 2002-12-16 at 08:02, Shridhar Daithankar wrote: On Monday 16 December 2002 07:26 pm, you wrote: I'm curious, what would be the recovery strategy for PITR master-slave replication should the master fail (assuming hot fail over/backup)? A simple dump/restore? Are there/is there any facilities in PorstgreSQL for PITR archival which prevents PITR logs from be recycled (or perhaps, simply archived off)? What about PITR streaming to networked and/or removable media? In asynchrounous replication, WAL log records are fed to anoter host, which replays those transactions to sync the data. This way it does not matter if WAL log is recycled as it is already replicated someplace else.. HTH Shridhar ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Big 7.4 items
On Mon, 2002-12-16 at 08:20, Shridhar Daithankar wrote: On Monday 16 December 2002 07:43 pm, you wrote: Consider that on the slave which is now the active server (master dead), it's possible that the slave's PITR's will be recycled before the master can come back up. As such, unless there is a, an archival process for PITR or b, a method of streaming PITR's off for archival, the odds of using PITR to recover the master (resync if you will) seem greatly reduced as you will be unable to replay PITR on the master for synchronization. I agree. Since we are talking about features in future release, I think it should be added to TODO if not already there. I don't know about WAL numbering but AFAIU, it increments and old files are removed once there are enough WAL files as specified in posgresql.conf. IIRC there are some perl based replication project exist already which use this feature. The problem with this is that most people, AFAICT, are going to size WAL based on their performance/sizing requirements and not based on theoretical estimates which someone might make to allow for a window of failure. That is, I don't believe increasing the number of WAL's is going to satisfactorily address the issue. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Password security question
On Tue, 2002-12-17 at 10:49, mlw wrote: Christopher Kings-Lynne wrote: Hi guys, Just a thought - do we explicitly wipe password strings from RAM after using them? I just read an article (by MS in fact) that illustrates a cute problem. Imagine you memset the password to zeros after using it. There is a good chance that the compiler will simply remove the memset from the object code as it will seem like it can be optimised away... Just wondering... Chris Could you post that link? That seems wrong, an explicit memset certainly changes the operation of the code, and thus should not be optimized away. I'd like to see the link too. I can imagine that it would be possible for it to optimize it away if there wasn't an additional read/write access which followed. In other words, why do what is more or less a no-op if it's never accessed again. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Password security question
On Tue, 2002-12-17 at 11:11, Ken Hirsch wrote: http://msdn.microsoft.com/library/en-us/dncode/html/secure10102002.asp ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED] Thanks. Seems I hit the nail on the head. ;) -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Update on replication
On Tue, 2002-12-17 at 20:55, Neil Conway wrote: On Tue, 2002-12-17 at 21:33, Greg Copeland wrote: I do agree, GBorg needs MUCH higher visibility! I'm just curious: why do we need GBorg at all? Does it offer anything that SourceForge, or a similar service does not offer? Especially given that (a) most other OSS projects don't have a site for related projects (unless you count something like CPAN, which is totally different) (b) GBorg is completely unknown to anyone outside the PostgreSQL community and even to many people within it... Part I can answer, part I can not. Since I'm not the one that pushed the projects to that site, I can't answer that part of the equation. Addressing the part of your question that I think I can, I do like the concept of one-stop-shopping for all PostgreSQL needs. All Things ProgreSQL is a pretty neat concept. Of course, it rather defeats the whole purpose if no one, including potential developers, have no idea it exists. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] v7.3.1 tar ready ... please check it ...
On Wed, 2002-12-18 at 08:53, Dave Page wrote: -Original Message- From: Marc G. Fournier [mailto:[EMAIL PROTECTED]] Sent: 18 December 2002 14:51 To: Robert Treat Cc: [EMAIL PROTECTED] Subject: Re: [HACKERS] v7.3.1 tar ready ... please check it ... On Wed, 18 Dec 2002, Robert Treat wrote: Is this going to be announced to a wider press audience? Has anyone gone over the list of things to do when we release to make sure things like the websites getting updated or perhaps getting rpm builds coordinated has been done? No, we don't do that with minor releases ... nothing has changed that needs to be announced, other then a few bugs fixed ... Maybe we should? The more publicity the better etc... Regards, Dave ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED] I agree it should be considered. It helps build mind share, which I think we can all agree is somewhat lacking for PostgreSQL compared to MySQL. It helps build the impression that PostgreSQL doesn't just sit idle between major releases. It allows a potential user base to see PostgreSQL more frequently and build interest. It let's people know that PostgreSQL is constantly being improved. Mind share is a powerful thing and as any advertiser will tell you, press releases is one of the best ways to get the word out. Greg -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] v7.3.1 Bundled and Released ...
On Sun, 2002-12-22 at 13:12, Marc G. Fournier wrote: Last night, we packaged up v7.3.1 of PostgreSQL, our latest stable release. Purely meant to be a bug fix release, this one does have one major change, in that the major number of the libpq library was increased, which means that everyone is encouraged to recompile their clients along with this upgrade. This release can be found on all the mirrors, and on the root ftp server, under: ftp://ftp.postgresql.org/pub/source/v7.3.1 Please report all bugs to [EMAIL PROTECTED] Hmm. For some reason I'm not seeing a 7.3.1 tag in CVS. Do you guys do something else for sub-releases? Case in point: cvs [server aborted]: no such tag REL7_3_1_STABLE It's still early here so I may be suffering from early morning brain rot. ;) Regards, Greg -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [GENERAL] [HACKERS] v7.3.1 Bundled and Released ...
Just a reminder, there still doesn't appear to be a 7.3.1 tag. This is from the HISTORY file. symbolic names: REL7_3_STABLE: 1.182.0.2 REL7_2_3: 1.153.2.8 REL7_2_STABLE: 1.153.0.2 REL7_2: 1.153 Notice 7.3 stable but nothing about 7.3.x! I also see a 7.2.3, etc., just as one would expect but nothing about 7.3 dot releases. I'm still getting, cvs [server aborted]: no such tag REL7_3_1_STABLE. Something overlooked here? Regards, Greg Copeland On Mon, 2002-12-23 at 09:57, Bruce Momjian wrote: Greg Copeland wrote: On Sun, 2002-12-22 at 13:12, Marc G. Fournier wrote: Last night, we packaged up v7.3.1 of PostgreSQL, our latest stable release. Purely meant to be a bug fix release, this one does have one major change, in that the major number of the libpq library was increased, which means that everyone is encouraged to recompile their clients along with this upgrade. This release can be found on all the mirrors, and on the root ftp server, under: ftp://ftp.postgresql.org/pub/source/v7.3.1 Please report all bugs to [EMAIL PROTECTED] Hmm. For some reason I'm not seeing a 7.3.1 tag in CVS. Do you guys do something else for sub-releases? Case in point: cvs [server aborted]: no such tag REL7_3_1_STABLE It's still early here so I may be suffering from early morning brain rot. ;) There should be a 7.3.1 tag, but you can use the 7_3 branch to pull 7.3.1. Of course, that will shift as we patch for 7.3.2. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] complie error on windows
If you run, gcc, at the prompt (preferably the one you're trying to run configure from), do you get something like, gcc: No input files or do you get, gcc: command not found? If you get the later (or something like it), you need to include it in your path, just as it's telling you to do. If you get the former, something would appear to be off. Greg On Fri, 2003-01-03 at 09:43, Claiborne, Aldemaco Earl (Al) wrote: Hi, I am trying to install postgresql-7.3 on windows and I keep getting the following error despite having downloaded a compiler. Can anyone tell me what I am not doing right? I am a newbie to postgres and development. My ultimate goal is to create a data driven application utilizing the J2EE architecture. $ ./configure checking build system type... i686-pc-cygwin checking host system type... i686-pc-cygwin checking which template to use... win checking whether to build with 64-bit integer date/time support... no checking whether to build with recode support... no checking whether NLS is wanted... no checking for default port number... 5432 checking for default soft limit on number of connections... 32 checking for gcc... no checking for cc... no configure: error: no acceptable C compiler found in $PATH Thanks, Al ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Threads
On Fri, 2003-01-03 at 14:47, mlw wrote: Please no threading threads!!! Ya, I'm very pro threads but I've long since been sold on no threads for PostgreSQL. AIO on the other hand... ;) Your summary so accurately addresses the issue it should be a whole FAQ entry on threads and PostgreSQL. :) Drawbacks to a threaded model: (1) One thread screws up, the whole process dies. In a multiple process application this is not too much of an issue. (2) Heap fragmentation. In a long uptime application, such as a database, heap fragmentation is an important consideration. With multiple processes, each process manages its own heap and what ever fragmentation that exists goes away when the connection is closed. A threaded server is far more vulnerable because the heap has to manage many threads and the heap has to stay active and unfragmented in perpetuity. This is why Windows applications usually end up using 2G of memory after 3 months of use. (Well, this AND memory leaks) These are things that can't be stressed enough. IMO, these are some of the many reasons why applications running on MS platforms tend to have much lower application and system up times (that and resources leaks which are inherent to the platform). BTW, if you do much in the way of threaded coding, there is libHorde which is a heap library for heavily threaded, memory hungry applications. It excels in performance, reduces heap lock contention (maintains multiple heaps in a very thread smart manner), and goes a long way toward reducing heap fragmentation which is common for heavily memory based, threaded applications. (3) Stack space. In a threaded application they are more limits to stack usage. I'm not sure, but I bet PostgreSQL would have a problem with a fixed size stack, I know the old ODBC driver did. Most modern thread implementations use a page guard on the stack to determine if it needs to grow or not. Generally speaking, for most modern platforms which support threading, stack considerations rarely become an issue. (5) Lastly, why bother? Seriously? Process creation time is an issue true, but its an issue with threads as well, just not as bad. Anyone who is looking for performance should be using a connection pooling mechanism as is done in things like PHP. I have done both threaded and process servers. The threaded servers are easier to write. The process based severs are more robust. From an operational point of view, a select foo from bar where x y will take he same amount of time. I agree with this, however, using threads does open the door for things like splitting queries and sorts across multiple CPUs. Something the current process model, which was previously agreed on, would not be able to address because of cost. Example: select foo from bar where x y order by foo ;, could be run on multiple CPUs if the sort were large enough to justify. After it's all said and done, I do agree that threading just doesn't seem like a good fit for PostgreSQL. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Threads
On Fri, 2003-01-03 at 14:52, Dann Corbit wrote: -Original Message- (1) One thread screws up, the whole process dies. In a multiple process application this is not too much of an issue. If you use C++ you can try/catch and nothing bad happens to anything but the naughty thread. That doesn't protect against the type of issues he's talking about. Invalid pointer reference is a very common snafu which really hoses threaded applications. Not to mention resource leaks AND LOCKED resources which are inherently an issue on Win32. Besides, it's doubtful that PostgreSQL is going to be rewritten in C++ so bringing up try/catch is pretty much an invalid argument. (2) Heap fragmentation. In a long uptime application, such as a database, heap fragmentation is an important consideration. With multiple processes, each process manages its own heap and what ever fragmentation that exists goes away when the connection is closed. A threaded server is far more vulnerable because the heap has to manage many threads and the heap has to stay active and unfragmented in perpetuity. This is why Windows applications usually end up using 2G of memory after 3 months of use. (Well, this AND memory leaks) Poorly written applications leak memory. Fragmentation is a legitimate concern. And well written applications which attempt to safely handle segfaults, etc., often leak memory and lock resources like crazy. On Win32, depending on the nature of the resources, once this happens, even process termination will not free/unlock the resources. (4) Lock Contention. The various single points of access in a process have to be serialized for multiple threads. heap allocation, deallocation, etc all have to be managed. In a multple process model, these resources would be separated by process contexts. Semaphores are more complicated than critical sections. If anything, a shared memory approach is more problematic and fragile, especially when porting to multiple operating systems. And critical sections lead to low performance on SMP systems for Win32 platforms. No task can switch on ANY CPU for the duration of the critical section. It's highly recommend by MS as the majority of Win32 applications expect uniprocessor systems and they are VERY fast. As soon as multiple processors come into the mix, critical sections become a HORRIBLE idea if any soft of scalability is desired. Is it a FAQ? If not, it ought to be. I agree. I think mlw's list of reasons should be added to a faq. It terse yet says it all! -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Threads
On Fri, 2003-01-03 at 19:34, Tom Lane wrote: Serguei Mokhov [EMAIL PROTECTED] writes: (1) One thread screws up, the whole process dies. In a multiple process application this is not too much of an issue. (1) is an issue only for user-level threads. Umm. No. User or system level threads, the statement is true. If a thread kills over, the process goes with it. Furthermore, on Win32 platforms, it opens a whole can of worms no matter how you care to address it. Uh, what other kind of thread have you got in mind here? I suppose the lack-of-cross-thread-protection issue would go away if our objective was only to use threads for internal parallelism in each backend instance (ie, you still have one process per connection, but internally it would use multiple threads to process subqueries in parallel). Several have previously spoken about a hybrid approach (ala Apache). IIRC, it was never ruled out but it was simply stated that no one had the energy to put into such a concept. Of course that gives up the hope of faster connection startup that has always been touted as a major reason to want Postgres to be threaded... regards, tom lane Faster startup, should never be the primary reason as there are many ways to address that issue already. Connection pooling and caching are by far, the most common way to address this issue. Not only that, but by definition, it's almost an oxymoron. If you really need high performance, you shouldn't be using transient connections, no matter how fast they are. This, in turn, brings you back to persistent connections or connection pools/caches. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Threads
On Fri, 2003-01-03 at 21:39, mlw wrote: Connection time should *never* be in the critical path. There, I've said it!! People who complain about connection time are barking up the wrong tree. Regardless of the methodology, EVERY OS has issues with thread creation, process creation, the memory allocation, and system manipulation required to manage it. Under load this is ALWAYS slower. I think that if there is ever a choice, do I make startup time faster? or Do I make PostgreSQL not need a dump/restore for upgrade the upgrade problem has a much higher impact to real PostgreSQL sites. Exactly. Trying to speed up something that shouldn't be in the critical path is exactly what I'm talking about. I completely agree with you! -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [GENERAL] [HACKERS] v7.3.1 Bundled and Released ...
On Sat, 2003-01-04 at 04:27, Peter Eisentraut wrote: Greg Copeland writes: Just a reminder, there still doesn't appear to be a 7.3.1 tag. There is a long tradition of systematically failing to tag releases in this project. Don't expect it to improve. Well, I thought I remembered from the release team thread that it was said there was a punch list of things that are done prior to actually releasing. If not, it certainly seems like we need one. If there is one, tagging absolutely needs to be on it. If we have one and this is already on the list, seems we need to be eating our own food. ;) -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Threads
On Sat, 2003-01-04 at 06:59, Kaare Rasmussen wrote: Umm. No. User or system level threads, the statement is true. If a thread kills over, the process goes with it. Furthermore, on Win32 Hm. This is a database system. If one of the backend processes dies unexpectedly, I'm not sure I would trust the consistency and state of the others. Or maybe I'm just being chicken. I'd call that being wise. That's the problem with using threads. Should a thread do something naughty, the state of the entire process is in question. This is true regardless if it is a user mode, kernel mode, or hybrid thread implementation. That's the power of using the process model that is currently in use. Should it do something naughty, we bitch and complain politely, throw our hands in the air and exit. We no longer have to worry about the state and validity of that backend. This creates a huge systemic reliability surplus. This is also why the concept of a hybrid thread/process implementation keeps coming to the surface on the list. If you maintain the process model and only use threads for things that ONLY relate to the single process (single session/connection), should a thread cause a problem, you can still throw you hands in the air and exit just as is done now without causing problems for, or questioning the validity of, other backends. The cool thing about such a concept is that it still opens the door for things like parallel sorts and queries as it relates to a single backend. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Upgrading rant.
On Sat, 2003-01-04 at 09:53, Tom Lane wrote: Oliver Elphick [EMAIL PROTECTED] writes: On Sat, 2003-01-04 at 02:17, Tom Lane wrote: There isn't any simple way to lock *everyone* out of the DB and still allow pg_upgrade to connect via the postmaster, and even if there were, the DBA could too easily forget to do it. I tackled this issue in the Debian upgrade scripts. I close the running postmaster and open a new postmaster using a different port, so that normal connection attempts will fail because there is no postmaster running on the normal port. That's a good kluge, but still a kluge: it doesn't completely guarantee that no one else connects while pg_upgrade is trying to do its thing. I am also concerned about the consequences of automatic background activities. Even the periodic auto-CHECKPOINT done by current code is not obviously safe to run behind pg_upgrade's back (it does make WAL entries). And the auto-VACUUM that we are currently thinking of is even less obviously safe. I think that someday, running pg_upgrade standalone will become *necessary*, not just a good safety feature. regards, tom lane I thought there was talk of adding a single user/admin only mode. That is, where only the administrator can connect to the database. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Upgrading rant.
On Sat, 2003-01-04 at 22:37, Tom Lane wrote: You're missing the point: I don't want to lock out everyone but the super-user, I want to lock out everyone, period. Superusers are just as likely to screw up pg_upgrade as anyone else. BTW: $ postmaster -N 1 -c superuser_reserved_connections=1 postmaster: superuser_reserved_connections must be less than max_connections. $ Well, first, let me say that the above just seems wrong. I can't think of any valid reason why reserved shouldn't be allowed to equal max. I also assumed that pg_update would be attempting to connect as the superuser. Therefore, if you only allow a single connection from the superuser and pg_upgrade is using it, that would seem fairly hard to mess things up. On top of that, that's also the risk of someone being a superuser. They will ALWAYS have the power to hose things. Period. As such, I don't consider that to be a valid argument. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [GENERAL] [HACKERS] v7.3.1 Bundled and Released ...
On Sun, 2003-01-05 at 06:41, Dan Langille wrote: On Sat, 4 Jan 2003, Tom Lane wrote: Marc G. Fournier [EMAIL PROTECTED] writes: I never considered tag'ng for minor releases as having any importance, since the tarball's themselves provide the 'tag' ... branches give us the ability to back-patch, but tag's don't provide us anything ... do they? Well, a tag makes it feasible for someone else to recreate the tarball, given access to the CVS server. Dunno how important that is in the real world --- but I have seen requests before for us to tag release points. FWIW, in the real world, a release doesn't happen if it's not taqged. Agreed! Any tarballs, rpms, etc., should be made from the tagged source. Period. If rpm's are made from a tarball that is made from tagged source, that's fine. Nonetheless, any official release (major or minor) should always be made from the resulting tagged source. This does two things. First, it validates that everything has been properly tagged. Two, it ensures that there are not any localized files or changes which might become part of a tarball/release which are not officially part of the repository. I can't stress enough that a release should never happen unless source has been tagged. Releases should ALWAYS be made from a checkout based on tags. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Threads
On Mon, 2003-01-06 at 05:36, Shridhar Daithankar wrote: On 6 Jan 2003 at 12:22, Ulrich Neumann wrote: Hello all, If someone is interested in the code I can send a zip file to everyone who wants. I suggest you preserver your work. The reason I suggested thread are mainly two folds. 1) Get I/O time used fuitfully AIO may address this without the need for integrated threading. Arguably, from the long thread that last appeared on the topic of AIO, some hold that AIO doesn't even offer anything beyond the current implementation. As such, it's highly doubtful that integrated threading is going to offer anything beyond what a sound AIO implementation can achieve. 2) Use multiple CPU better. Multiple processes tend to universally support multiple CPUs better than does threading. On some platforms, the level of threading support is currently only user mode implementations which means no additional CPU use. Furthermore, some platforms where user-mode threads are defacto, they don't even allow for scheduling bias resulting is less work being accomplished within the same time interval (work slice must be divided between n-threads within the process, all of which run on a single CPU). It will not require as much code cleaning as your efforts might had. However your work will be very useful if somebody decides to use thread in any fashion in core postgresql. I was hoping for bit more optimistic response given that what I suggested was totally optional at any point of time but very important from performance point. Besides the change would have been gradual as required.. Speaking for my self, I probably would of been more excited if the offered framework had addressed several issues. The short list is: o Code needs to be more robust. It shouldn't be calling exit directly as, I believe, it should be allowing for PostgreSQL to clean up some. Correct me as needed. I would of also expected the code of adopted PostgreSQL's semantics and mechanisms as needed (error reporting, etc). I do understand it was an initial attempt to simply get something in front of some eyes and have something to talk about. Just the same, I was expecting something that we could actually pull the trigger with. o Code isn't very portable. Looked fairly okay for pthread platforms, however, there is new emphasis on the Win32 platform. I think it would be a mistake to introduce something as significant as threading without addressing Win32 from the get-go. o I would desire a more highly abstracted/portable interface which allows for different threading and synchronization primitives to be used. Current implementation is tightly coupled to pthreads. Furthermore, on platforms such as Solaris, I would hope it would easily allow for plugging in its native threading primitives which are touted to be much more efficient than pthreads on said platform. o Code is not commented. I would hope that adding new code for something as important as threading would be commented. o Code is fairly trivial and does not address other primitives (semaphores, mutexs, conditions, TSS, etc) portably which would be required for anything but the most trivial of threaded work. This is especially true in such an application where data IS the application. As such, you must reasonably assume that threads need some form of portable serialization primitives, not to mention mechanisms for non-trivial communication. o Does not address issues such as thread signaling or status reporting. o Pool interface is rather simplistic. Does not currently support concepts such as wake pool, stop pool, pool status, assigning a pool to work, etc. In fact, it's not altogether obvious what the capabilities intent is of the current pool implementation. o Doesn't seem to address any form of thread communication facilities (mailboxes, queues, etc). There are probably other things that I can find if I spend more than just a couple of minutes looking at the code. Honestly, I love threads but I can see that the current code offering is not much more than a token in its current form. No offense meant. After it's all said and done, I'd have to see a lot more meat before I'd be convinced that threading is ready for PostgreSQL; from both a social and technological perspective. Regards, -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] IPv6 patch
On Mon, 2003-01-06 at 15:29, Peter Eisentraut wrote: (2) A socket type is explicitly enabled for the server to use, and if creation fails, server startup fails. It seems that the current code falls back to IPv4 if IPv6 fails. IIRC, it allows it to fall back to IPv4 in case it's compiled for IPv6 support but the kernel isn't compiled to support IPv6. If that is the case, admittedly, you seem to have a point. If someone compiles in v6 support and their system doesn't have v6 support and it's been requested via run-time config, it's should fail just like any other. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] IPv6 patch
On Mon, 2003-01-06 at 15:43, Bruce Momjian wrote: Greg Copeland wrote: On Mon, 2003-01-06 at 15:29, Peter Eisentraut wrote: (2) A socket type is explicitly enabled for the server to use, and if creation fails, server startup fails. It seems that the current code falls back to IPv4 if IPv6 fails. IIRC, it allows it to fall back to IPv4 in case it's compiled for IPv6 support but the kernel isn't compiled to support IPv6. If that is the case, admittedly, you seem to have a point. If someone compiles in v6 support and their system doesn't have v6 support and it's been requested via run-time config, it's should fail just like any other. Yes, right now, it is kind of a mystery when it falls back to IPv4. It does print a message in the server logs: LOG: server socket failure: getaddrinfo2() using IPv6: hostname nor servname provided, or not known LOG: IPv6 support disabled --- perhaps the kernel does not support IPv6 LOG: IPv4 socket created It appears right at the top because creating the socket is the first thing it does. A good question is once we have a way for the user to control IPv4/6, what do we ship as a default? IPv4-only? Both, and if both, do we fail on a kernel that doesn't have IPv6 enabled? So you're saying that by using the IPv6 address family and you bind to an IPv6 address (or even ANY interface), you still get v4 connections on the same bind/listen/accept sequence? I'm asking because I've never done v6 stuff. Regards, -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] IPv6 patch
On Mon, 2003-01-06 at 15:59, Bruce Momjian wrote: Greg Copeland wrote: It appears right at the top because creating the socket is the first thing it does. A good question is once we have a way for the user to control IPv4/6, what do we ship as a default? IPv4-only? Both, and if both, do we fail on a kernel that doesn't have IPv6 enabled? So you're saying that by using the IPv6 address family and you bind to an IPv6 address (or even ANY interface), you still get v4 connections on the same bind/listen/accept sequence? I'm asking because I've never done v6 stuff. Yes, it listens on both. The original author, Nigel, tested in using both IPv4 and IPv6, and the #ipv6 IRC channel and google postings seem to indicate that too. What I am not sure how to do is say _only_ IPv4. Wouldn't you just use an IPv4 address family when creating your socket? -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] IPv6 patch
On Mon, 2003-01-06 at 16:17, Bruce Momjian wrote: Greg Copeland wrote: On Mon, 2003-01-06 at 15:59, Bruce Momjian wrote: Greg Copeland wrote: It appears right at the top because creating the socket is the first thing it does. A good question is once we have a way for the user to control IPv4/6, what do we ship as a default? IPv4-only? Both, and if both, do we fail on a kernel that doesn't have IPv6 enabled? So you're saying that by using the IPv6 address family and you bind to an IPv6 address (or even ANY interface), you still get v4 connections on the same bind/listen/accept sequence? I'm asking because I've never done v6 stuff. Yes, it listens on both. The original author, Nigel, tested in using both IPv4 and IPv6, and the #ipv6 IRC channel and google postings seem to indicate that too. What I am not sure how to do is say _only_ IPv4. Wouldn't you just use an IPv4 address family when creating your socket? Sorry, I meant only IPv6. I found this. It seems to imply that you need different sockets to do what you want to do. You might snag a copy of the latest openldap code and look at slapd to see what it's doing. At any rate, here's what I found that pointed me at it: slapd compiled with --enable-ipv6 does only listen to the IPv6 connections. I suspect this is tested on an operating system that receives IPv4 connections to the IPv6 socket as well, although this is not not the case for OpenBSD (nor FreeBSD or NetBSD ,IIRC). slapd needs to listen to both IPv4 and IPv6 on separate sockets. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Next platform query: Alphaservers under VMS?
IIRC, they too have a POSIX layer available. Greg On Tue, 2003-01-07 at 02:44, Justin Clift wrote: Hi guys, Also received a through the Advocacy website asking if anyone has ported PostgreSQL to the AlphaServers under VMS. Anyone know if we run on VMS? Last time I touched VMS (about 10 years ago) it wasn't all that Unix-like. :-) Regards and best wishes, Justin Clift -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Threads
On Tue, 2003-01-07 at 02:00, Shridhar Daithankar wrote: On 6 Jan 2003 at 6:48, Greg Copeland wrote: 1) Get I/O time used fuitfully AIO may address this without the need for integrated threading. Arguably, from the long thread that last appeared on the topic of AIO, some hold that AIO doesn't even offer anything beyond the current implementation. As such, it's highly doubtful that integrated threading is going to offer anything beyond what a sound AIO implementation can achieve. Either way, a complete aio or threading implementation is not available on major platforms that postgresql runs. Linux definitely does not have one, last I checked. There are two or three significant AIO implementation efforts currently underway for Linux. One such implementation is available from the Red Hat Server Edition (IIRC) and has been available for some time now. I believe Oracle is using it. SGI also has an effort and I forget where the other one comes from. Nonetheless, I believe it's going to be a hard fought battle to get AIO implemented simply because I don't think anyone, yet, can truly argue a case on the gain vs effort. If postgresql is not using aio or threading, we should start using one of them, is what I feel. What do you say? I did originally say that I'd like to see an AIO implementation. Then again, I don't current have a position to stand other than simply saying it *might* perform better. ;) Not exactly a position that's going to win the masses over. was expecting something that we could actually pull the trigger with. That could be done. I'm sure it can, but that's probably the easiest item to address. o Code isn't very portable. Looked fairly okay for pthread platforms, however, there is new emphasis on the Win32 platform. I think it would be a mistake to introduce something as significant as threading without addressing Win32 from the get-go. If you search for pthread in thread.c, there are not many instances. Same goes for thread.h. From what I understand windows threading, it would be less than 10 minutes job to #ifdef the pthread related part on either file. It is just that I have not played with windows threading and nor I am inclined to...;-) Well, the method above is going to create a semi-ugly mess. I've written thread abstraction layers which cover OS/2, NT, and pthreads. Each have subtle distinction. What really needs to be done is the creation of another abstraction layer which your current code would sit on top of. That way, everything contained within is clear and easy to read. The big bonus is that as additional threading implementations need to be added, only the low-level abstraction stuff needs to modified. Done properly, each thread implementation would be it's own module requiring little #if clutter. As you can see, that's a fair amount of work and far from where the code currently is. o I would desire a more highly abstracted/portable interface which allows for different threading and synchronization primitives to be used. Current implementation is tightly coupled to pthreads. Furthermore, on platforms such as Solaris, I would hope it would easily allow for plugging in its native threading primitives which are touted to be much more efficient than pthreads on said platform. Same as above. If there can be two cases separated with #ifdef, there can be more.. But what is important is to have a thread that can be woken up as and when required with any function desired. That is the basic idea. Again, there's a lot of work in creating a well formed abstraction layer for all of the mechanics that are required. Furthermore, different thread implementations have slightly different semantics which further complicates things. Worse, some types of primitives are simply not available with some thread implementations. That means those platforms require it to be written from the primitives that are available on the platform. Yet more work. o Code is fairly trivial and does not address other primitives (semaphores, mutexs, conditions, TSS, etc) portably which would be required for anything but the most trivial of threaded work. This is especially true in such an application where data IS the application. As such, you must reasonably assume that threads need some form of portable serialization primitives, not to mention mechanisms for non-trivial communication. I don't get this. Probably I should post a working example. It is not threads responsibility to make a function thread safe which is changed on the fly. The function has to make sure that it is thread safe. That is altogether different effort.. You're right, it's not the thread's responsibility, however, it is the threading toolkit's. In this case, you're offering to be the toolkit which functions across two platforms, just for starters. Reasonably, you should expect a third to quickly follow
Re: [HACKERS] Threads
On Tue, 2003-01-07 at 12:21, Greg Stark wrote: Greg Copeland [EMAIL PROTECTED] writes: That's the power of using the process model that is currently in use. Should it do something naughty, we bitch and complain politely, throw our hands in the air and exit. We no longer have to worry about the state and validity of that backend. You missed the point of his post. If one process in your database does something nasty you damn well should worry about the state of and validity of the entire database, not just that one backend. I can assure you I did not miss the point. No idea why you're continuing to spell it out. In this case, it appears the quotation is being taken out of context or it was originally stated in an improper context. Are you really sure you caught the problem before it screwed up the data in shared memory? On disk? This whole topic is in need of some serious FUD-dispelling and careful analysis. Here's a more calm explanation of the situation on this particular point. Perhaps I'll follow up with something on IO concurrency later. Hmmm. Not sure what needs to be dispelled since I've not seen any FUD. The point in consideration here is really memory isolation. Threads by default have zero isolation between threads. They can all access each other's memory even including their stack. Most of that memory is in fact only needed by a single thread. Again, this has been covered already. Processes by default have complete memory isolation. However postgres actually weakens that by doing a lot of work in a shared memory pool. That memory gets exactly the same protection as it would get in a threaded model, which is to say none. Again, this has all been covered, more or less. You're comments seem to imply that you did not fully read what has been said on the topic thus far or that you misunderstood something that was said. Of course, it's also possible that I may of said something out of it's proper context which may be confusing you. I think it's safe to say I don't have any further comment unless something new is being brought to the table. Should there be something new to cover, I'm happy to talk about it. At this point, however, it appears that it's been beat to death already. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] [Npgsql-general] Get function OID and function
San someone point me to what exactly is planned for the protocol/networking stuff? Networking/protocols is one of my fortes and I believe that I could actually help here. Regards, Greg On Tue, 2003-01-07 at 09:01, Tom Lane wrote: Dave Page [EMAIL PROTECTED] writes: Sorry, don't know. Can anyone on pgsql-hackers tell us the purpose of the FunctionCall message? It's used to invoke the fast path function call code (src/backend/tcop/fastpath.c). libpq's large-object routines use this, but little else does AFAIK. The current protocol is sufficiently broken (see comments in fastpath.c) that I'd not really encourage people to use it until we can fix it --- hopefully that will happen in 7.4. regards, tom lane PS: what in the world is [EMAIL PROTECTED] ... is that a real mailing list, and if so why? It sounds a bit, um, duplicative. ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED]) -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] redo error?
On Tue, 2003-01-07 at 22:58, Tom Lane wrote: It also logged that it was killed with signal 9, although I didn't kill it! Is there something weird going on here? Is this Linux? The Linux kernel seems to think that killing randomly-chosen processes with SIGKILL is an appropriate response to running out of memory. I cannot offhand think of a more brain-dead behavior in any OS living or dead, but that's what it does. Just FYI, I believe the 2.6.x series of kernels will rectify this situation. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] \d type queries - why not views in system catalog?!?
Oh! That's an excellent idea. Seemingly addresses the issue and has value-add. I'm not aware of any gotchas here. Is there something that is being overlooked? Greg On Mon, 2003-01-13 at 14:50, Robert Treat wrote: On Mon, 2003-01-13 at 11:28, [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 NotDashEscaped: You need GnuPG to verify this message Wouldn't it be easier (and more portable, see 7.3/7.2 system catalogs vs. psql) to have views for that? Do I miss a point here? Putting the \d commands into views has been on the TODO list for a long time: I think it is actually the only psql-related item left, until we change the backend protocol to indicate transaction state. I don't think a view would have helped with the psql 7.2/7.3 change: a lot more changed than simply the underlying SQL. Some of the the backslash commands are not amenable to putting inside a view, as they actually compromise multiple SQL calls and some logic in the C code, but a few could probably be made into views. Could whomever added that particular TODO item expand on this? One idea I've always thought would be nice would be to make full fledged C functions out of the \ commands and ship them with the database. This way the \ commands could just be alias to select myfunc(). This would help out all of us who write GUI interfaces since we would have standard functions we could call upon, and would also help with backward compatibility since \dv could always call select list_views(), which would already be included with each server. One of the reasons that this was not feasible in the past was that we needed functions that could return multiple rows and columns easily. Now that we have that in 7.3, it might be worth revisiting. Robert Treat ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED] -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] \d type queries - why not views in system catalog?!?
Views or C-functions, I think the idea is excellent. It's the concept that I really like. Greg On Mon, 2003-01-13 at 15:00, Dave Page wrote: -Original Message- From: Greg Copeland [mailto:[EMAIL PROTECTED]] Sent: 13 January 2003 20:56 To: Robert Treat Cc: [EMAIL PROTECTED]; PostgresSQL Hackers Mailing List Subject: Re: [HACKERS] \d type queries - why not views in system catalog?!? Oh! That's an excellent idea. Seemingly addresses the issue and has value-add. I'm not aware of any gotchas here. Is there something that is being overlooked? Why use functions instead of views? Most UIs will want to format the output as they see fit so a recordset would be the appropriate output. Yes, a function could do this, but surely views would be simpler to implement and maintain. Regards, Dave. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] C++ coding assistance request for a visualisation
Have you tried IBM's OSS visualization package yet? Sorry, I don't seem to recall the name of the tool off the top of my head (Data Explorer??) but it uses OpenGL (IIRC) and is said to be able to visualize just about anything. Anything is said to include simple data over time to complex medical CT scans. Greg On Wed, 2003-01-22 at 12:19, Justin Clift wrote: Hi guys, Is there anyone here that's good with C++ and has a little bit of time to add PostgreSQL support to a project? There is a 4D visualisation program called Flounder: http://www.enel.ucalgary.ca/~vigmond/flounder/ And it does some pretty nifty stuff. It takes in data sets (x, y, z, time) and displays then graphically, saving them to image files if needed, and also creating the time sequences as animations if needed. Was looking at it from a performance tuning tool point of view. i.e. Testing PostgreSQL performance with a bunch of settings, then stuffing the results into a database, and then using something like Flounder for visualising it. It seems pretty simple, and Flounder seems like it might be the right kind of tool for doing things like this. Was emailing with Edward Vigmond, the author of it, and he seems to think it'd be pretty easy to implement too. Now, I'm not a C++ coder, and as short of time as anyone, so I was wondering if there is anyone here who'd be interested in helping out here. :-) Regards and best wishes, Justin Clift -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Threads
On Thu, 2003-01-23 at 09:12, Steve Wampler wrote: On Sat, 4 Jan 2003, Christopher Kings-Lynne wrote: Also remember that in even well developed OS's like FreeBSD, all a process's threads will execute only on one CPU. I doubt that - it certainly isn't the case on Linux and Solaris. A thread may *start* execution on the same CPU as it's parent, but native threads are not likely to be constrained to a specific CPU with an SMP OS. You are correct. When spawning additional threads, should an idle CPU be available, it's very doubtful that the new thread will show any bias toward the original thread's CPU. Most modern OS's do run each thread within a process spread across n-CPUs. Those that don't are probably attempting to modernize as we speak. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] C++ coding assistance request for a
On Wed, 2003-01-22 at 23:40, Justin Clift wrote: Justin Clift wrote: Greg Copeland wrote: Have you tried IBM's OSS visualization package yet? Sorry, I don't seem to recall the name of the tool off the top of my head (Data Explorer??) but it uses OpenGL (IIRC) and is said to be able to visualize just about anything. Anything is said to include simple data over time to complex medical CT scans. Cool. Just found it... IBM Open Visualization Data Explorer: http://www.research.ibm.com/dx/ That seems to be a very outdated page for it. The new pages for it (in case anyone else is interested) are at: http://www.opendx.org :-) Regards and best wishes, Justin Clift Yep! That's the stuff! Sorry I wasn't more specific. Just been a while since I'd looked at it. I'd love to know how well it works out for you. Especially love to see any pretty pictures you create with it. ;) Regards, Greg -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] postgresql.org
Should it be saying, Temporarily Unavailable? Regards, -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] plpython fails its regression test
On Thu, 2003-01-30 at 16:39, Tom Lane wrote: In CVS tip, if you run make installcheck in src/pl/plpython, the test fails with a number of diffs between the expected and actual output. I'm not sure if plpython is broken, or if it's just that someone changed the behavior and didn't bother to update the test's expected files (the test files don't seem to have been maintained since they were first installed). Comments? Could this have anything to do with the changes I made to the python stuff to get it to support longs (IIRC)? It's been a while now so I don't recall exactly what got changed. I do remember that I chanced some test code to ensure it tested the newly fixed data type. Regards, -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [mail] Re: [HACKERS] Windows Build System
On Fri, 2003-01-31 at 07:22, Christopher Browne wrote: But it's not /nearly/ that straightforward. If you look at the downloads that MySQL AB provides, they point you to a link that says Windows binaries use the Cygwin library. Which apparently means that this feature is not actually a feature. Unlike PostgreSQL, which is run under the Cygwin emulation, MySQL runs as a native Windows application (with Cygwin emulation). Apparently those are not at all the same thing, even though they are both using Cygwin... I'm confused as to whether you are being sarcastic or truly seem to think there is a distinction here. Simple question, does MySQL require the cygwin dll's (or statically linked to) to run? If the answer is yes, then there is little question that they are as emulated as is the current PostgreSQL/Win32 effort. Care to expand on exactly what you believe the distinction is? ...or did I miss the humor boat? :( Regards, -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Linux.conf.au 2003 Report
On Fri, 2003-01-31 at 13:04, Kurt Roeckx wrote: On Thu, Jan 30, 2003 at 08:21:09PM -0600, Greg Copeland wrote: It doesn't help the confusion that many OS's try to confuse programmers by exposing a single socket interface, etc. Simple fact remains, IPv6 is not IPv4. It's a good things that the socket interface can actually work with all protocol! It doesn't only work with AF_INET, but also AF_UNIX, and probably others. It's a good things that things like socket(), bind(), connect() don't need to be replaced by other things. That's actually not what I was talking about. Please see the recent IPv6 support thread as an example. The fact that an OS allows IPv4 connections to be completed even though you explicitly requested IPv6 protocol, only adds to much confusion (one of many such oddities which some OS's allow). Heck, along those lines, they should allow NCP connections to come through too. Or, how about UDP traffic on TCP sockets. If I wanted IPv4 traffic, I'll ask for it. Likewise of IPv6. My point being, too many people are in a hurry to confuse/combine the two when they are very clearly two distinct protocols, each having distinct needs. The faster people treat them as such, the quicker things will become better for everyone. The fact that some OS's attempt to blur the API lines and underlying semantics between the two protocols only further confuses things as it falsely leads people to believe that they are more or less the same protocol. Regards, -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [mail] Re: [HACKERS] Windows Build System
On Thu, 2003-01-30 at 13:56, Dave Page wrote: When properly configured, Windows can be reliable, maybe not as much as Solaris or HPUX but certainly some releases of Linux (which I use as well). You don't see Oracle or IBM avoiding Windows 'cos it isn't stable enough. I'm not jumping on one side or the other but I wanted to make clear on something. The fact that IBM or Oracle use windows has absolutely zero to do with reliability or stability. They are there because the market is willing to spend money on their product. Let's face it, the share holders of each respective company would come unglued if the largest software audience in the world were completely ignored. Simple fact is, your example really is pretty far off from supporting any view. Bluntly stated, both are in that market because they want to make money; they're even obligated to do so. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [mail] Re: [HACKERS] Windows Build System
On Thu, 2003-01-30 at 14:27, Dave Page wrote: -Original Message- From: Tom Lane [mailto:[EMAIL PROTECTED]] Sent: 30 January 2003 15:56 To: Hannu Krosing Cc: Vince Vielhaber; Dave Page; Ron Mayer; [EMAIL PROTECTED] Subject: Re: [mail] Re: [HACKERS] Windows Build System In the pull-the-plug case you have to worry about what is on disk at any given instant and whether you can make all the bits on disk consistent again. (And also about whether your filesystem can perform the equivalent exercise for its own metadata; which is why we are questioning Windows here. I've never (to my knowledge) lost any data following a powerfail or system crash on a system using NTFS - that has always seemed pretty solid to me. By comparison, I have lost data on ext2 filesystems on a couple of occasions. More info at: http://www.ntfs.com/data-integrity.htm http://www.pcguide.com/ref/hdd/file/ntfs/relRec-c.html Obviously this goes out of the window is the user chooses to run on FAT/FAT32 partitions. I think that it should be made *very* clear in any future documentation that the user is strongly advised to use only NTFS filesystems. I realise this is not proof that it actually works of course... I have lost entire directory trees (and all associated data) on NTFS before. NTFS was kind enough to detect an inconsistency during boot and repaired the file system by simply removing any and all references to the top level damaged directory (on down). Sure, the file system was in a known good state following the repair but the 2-days to recover from it, pretty much stunk! I would also like to point out that this damage/repair occurred on a RAID-5 box (hardware, not software). As the repairs placed the file system back into known good state, the raid hardware was happy to obey. Guess what, it did! :( Make no mistake about it. You can easily lose large amounts of data on NTFS. You also compared NTFS with ext2. That's not exactly fair. Better you should compare NTFS with ext3, XFS, JFS, ReiserFS. It's a better, more fair comparison, as now we're talking about the same category of file system. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [mail] Re: [HACKERS] Windows Build System
On Fri, 2003-01-31 at 19:22, Dann Corbit wrote: For MySQL: There is no Cygwin needed. Period. Any idea as to why we seem to be getting such a conflicting story here? By several accounts, it does. Now, your saying it doesn't. What the heck is going on here. Not that I'm doubting you. I'm just trying to figure out which side of the coin is the shinny one. ;) There's a tool that comes with either the resource kit or the VC++ stuff that will tell you information like what ldd does. I don't recall the name of the tool. Can anyone comment if cygwin (or equivalent) is being linked in (statically or dynamically)? -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [mail] Re: [HACKERS] Windows Build System
On Fri, 2003-01-31 at 19:22, Dann Corbit wrote: For MySQL: There is no Cygwin needed. Period. Sorry to followup again, but I did want to point out something. I'm assuming you actually installed it. Please take note that the cygwin dll is normally installed into one of the window's directories (system, windows, etc). My point being, just because you didn't find it in the mysql directory, doesn't mean it wasn't installed system-wide. Not saying it does or doesn't do this. Just offering something else that may need to be looked at. Regards, -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
