Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
Lamar Owen [EMAIL PROTECTED] writes: Now, J Random slides in the new OS CD on a backup of his main server, and upgrades. RedHat 7.2's installer is very smart -- if no packages are left that use glibc 2.0, it doesn't install the compat-libs necessary for glibc 2.0 apps to run. Actually, glibc is a bad example of things to break - it has versioned symbols, so postgresql is pretty likely to continue working (barring doing extremely low-level stuff, like doing weird things to the loader or depend on buggy behaviour (like Oracle did)). Postgresql doesn't use C++ either (which is a horrible mess wrt. binary compatibility - there is no such thing, FTTB). However, if it depended on kernel specific behaviour (like things in /proc, which may or may not have changed its output format) it could break. -- Trond Eivind Glomsrød Red Hat, Inc.
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
Bruce Momjian wrote: However, the dependency upon the new version of the OS being able to run the old executables could be a killer in the future if executable compatibility is removed -- after all, an upgrade might not be from the immediately prior version of the OS. That is a tough one. I see your point. How would the RPM do this anyway? It is running the same version of the OS right? Did they move the data files from the old OS to the new OS and now they want to upgrade? Hmm. Well, let's suppose the following: J. Random User has a database server that has been running smoothly for ages on RedHat 5.2, running PostgreSQL 6.3.2. He has had no reason to upgrade since -- while MVCC was a nice feature, he was really waiting for OUTER JOIN before upgrading, as his server is lightly loaded and won't benefit greatly from MVCC. Likewise, he's not upgraded from RedHat 5.2, because until RedHat got the 2.4 kernel into a distribution, he wasn't ready to upgrade, as he needs improved NFS performance, available in Linux kernel 2.4. And he wasn't about to go with a version of GCC that doesn't exist. So he skips the whole RedHat 6.x series -- he doesn't want to mess with kernel 2.2 in any form, thanks to its abyssmal NFS performance. So he waits on RedHat 7.2 to be released -- around October 2001 (if the typical RedHat schedule holds). At this point, PostgreSQL 7.2.1 is the standards bearer, with OUTER JOIN support that he craves, and robust WAL for excellent recoverability, amongst other Neat Features(TM). Now, by the time of RedHat 7.2, kernel 2.4 is up to .15 or so, with gcc 3.0 freshly (and officially) released, and glibc 2.2.5 finally fixing the problems that had plagued both pre-2.2 glibc's AND the earliest 2.2 glibc's -- but, the upshot is that glibc 2.0 compatibility is toast. Now, J Random slides in the new OS CD on a backup of his main server, and upgrades. RedHat 7.2's installer is very smart -- if no packages are left that use glibc 2.0, it doesn't install the compat-libs necessary for glibc 2.0 apps to run. The PostgreSQL RPMset's server subpackage preinstall script runs about two-thirds of the way through the upgrade, and backs up the old 6.3.2 executables necessary to pull a dump. The old 6.3.2 rpm database entries are removed, and, as far as the system is concerned, no dependency upon glibc 2.0 remains, so no compat-libs get installed. J Random checks out the new installation, and finds a conspicuous log message telling him to read /usr/share/doc/postgresql-7.2.1/README.rpm. He does so, and runs the (fixed by then) postgresql-dump script, which attempts to start an old backend and do a pg_dumpall -- but, horrors, the old postmaster can't start, glibc 2.0 is gone and glibc 2.2 blows core loaded under postmaster-6.3.2. ARGGHHH That's the scenario I have nightmares about. Really. Yes, but if we added capabilities every time someone wanted something so it worked better in their environment, this software would be a mess, right? Yes, it would. I'll work on a patch, and we'll see what it looks like. been e-mails from users of other OS's -- even those that compile from source -- expressing a desire for a smoother upgrade cycle. The RPM's, Yes, we all agree upgrades should be smoother. The problem is that the cost/benefit analysis always pushed us away from improving it. I understand. I'm looking at some point in time in the future doing a 'postgresql-upgrade' RPM that would include pre-built postmasters and other binaries necessary to dump any previous version PostgreSQL (since about 6.2.1 or so -- 6.2.1 was the first RedHat official PostgreSQL RPM, although there were 6.1.1 RPM's before that, and there is still a postgres95-1.09 RPM out there), linked to the current libs for that RPM's OS release. It would be a large RPM (and the source RPM for it would be _huge_, containing entire tarballs for at least 6.2.1, 6.3.2, 6.4.2, 6.5.3, and 7.0.3). But, this may be the only way to make this work barring a real migration utility. And, BTW, welcome back from the summit. I heard that there was a little 'excitement' there :-). Yes, it was very nice. I will post a summary to announce/general today. Good. And a welcome back to Tom as well, as he went too, IIRC. -- Lamar Owen WGCR Internet Radio 1 Peter 4:11
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
Lamar Owen [EMAIL PROTECTED] writes: All I'm saying is that regression should be _runnable_ in all modes without needing anything but a shell and the PostgreSQL binary installation. I think this'd be mostly a waste of effort. IMHO, 99% of the problems the regression tests might expose will be exposed if they are run against the RPMs by the RPM maker. (Something we have sometimes failed to do in the past ;-).) The regress tests are not that good at detecting environment-specific problems; in fact, they go out of their way to suppress environmental differences. So I don't see any strong need to support regression test running in binary distributions. Especially not if we have to kluge around a lack of essential tools. regards, tom lane
RE: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
I'm looking at some point in time in the future doing a 'postgresql-upgrade' RPM that would include pre-built postmasters and other binaries necessary to dump any previous version PostgreSQL (since about 6.2.1 or so -- 6.2.1 was the first RedHat official PostgreSQL RPM, although there were 6.1.1 RPM's before that, and there is still a postgres95-1.09 RPM out there), linked to the current libs for that RPM's OS release. It would be a large RPM (and the source RPM for it would be _huge_, containing entire tarballs for at least 6.2.1, 6.3.2, 6.4.2, 6.5.3, and 7.0.3). But, this may be the only way to make this work barring a real migration utility. How about instead of one huge utility that does everything for all combinations, you release specific utilities for different upgrades. Example, one RPM for 6.5.3 - 7.0.3 and another RPM for 6.4.2 - 7.0.2. It would result in more RPMs but they would be smaller and probably easier to maintain.
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
Lamar Owen writes: In the environment of the general purpose OS upgrade, the RPM's installation scripts cannot fire up a backend, nor can it assume one is running or is not running, nor can the RPM installation scripts fathom from the run-time environment whether they are being run from a command line or from the OS upgrade (except on Linux Mandrake, which allows such usage). I don't understand why this is so. It seems perfectly possible that some %preremovebeforeupdate starts a postmaster, runs pg_dumpall, saves the file somewhere, then the %postinstallafterupdate runs the inverse operation. Disk space is not a valid objection, you'll never get away without 2x storage. Security is not a problem either. Are you not upgrading in proper dependency order or what? Everybody does dump, remove, install, undump; so can the RPMs. Okay, so it's not as great as a new KDE starting up and asking "may I update your configuration files?", but understand that the storage format is optimized for performance, not easy processing by external tools or something like that. -- Peter Eisentraut [EMAIL PROTECTED] http://yi.org/peter-e/
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
[Since I've rested over the weekend, I hope I don't come across this morning as an angry old snarl, like some of my previous posts on this subject unfortunately have been.] Bruce Momjian wrote: * Location-agnostic installation. Documentation (which I'll be happy to contribute) on that. Peter E is already working in this area. Getting the installation that 'make install' spits out massaged into an FHS compliant setup is the majority of the RPM's spec file. Well, we certainly don't want to make changes that make things harder or more confusing for non-RPM installs. How are they affected here? They wouldn't be. Peter E has seemingly done an excellent job in this area. I say seemingly because I haven't built an RPM from the 7.1 branch yet, but from what he has posted, he seems to understand the issue. Many thanks, Peter. * Upgrades that don't require an ASCII database dump for migration. This can either be implemented as a program to do a pg_dump of an arbitrary version of data, or as a binary migration utility. Currently, I'm I really don't see the issue here. At the risk of being redundant, here goes. As I've explained before, the RPM upgrade environment, thanks to our standing with multiple distributions as being shipped as a part of the OS, could be run as part of a general-purpose OS upgrade. In the environment of the general purpose OS upgrade, the RPM's installation scripts cannot fire up a backend, nor can it assume one is running or is not running, nor can the RPM installation scripts fathom from the run-time environment whether they are being run from a command line or from the OS upgrade (except on Linux Mandrake, which allows such usage). Thus, if a system administrator upgrades a system, or if an end user who has a pgaccess-customized data entry system for things as mundane as an address list or recipe book, there is no opportunity to do a dump. The dump has to be performed _after_ the RPM upgrade. Now, this is far from optimal, I know. I _know_ that the user should take pains with their data. I know that there should be a backup. I also know that a user of PostgreSQL should realize that 'this is just the way it is done' and do things Our Way. I also know that few new users will do it 'Our Way'. No other package that I am aware of requires the manual intervention that PostgreSQL does, with the possible exception of upgrading to a different file system -- but that is something most new users won't do, and is something that is more difficult to automate. However, over the weekend, while resting (I did absolutely NO computer work this weekend -- too close to burnout), I had a brainstorm. A binary migration tool does not need to be written, if a concession to the needs of some users who just simply want to upgrade can be made. Suppose we can package old backends (with newer network code to connect to new clients). Suppose further that postmaster can be made intelligent enough to fire up old backends for old data, using PG_VERSION as a key. Suppose a NOTICE can be fired off warning the user that 'The Database is running in Compatibility Mode -- some features may not be available. Please perform a dump of your data, reinitialize the database, and restore your data to access new features of version x.y'. I'm highly considering doing just that from a higher level. It will not be nearly as smooth, but doable. Of course, that increases maintenance work, and I know it does. But I'm trying to find a middle ground here, since providing a true migration utility (even if it just produces a dump of the old data) seems out of reach at this time. We are currently forcing something like a popular word processing program once did -- it's proprietary file format changed. It was coded so that it could not even read the old files. But both the old and the new versions could read and write an interchange format. People who blindly upgraded their word processor were hit with a major problem. There was even a notice in the README -- which could be read after the program was installed. While the majority of us use PostgreSQL as a server behind websites and other clients, there will be a large number of new users who want to use it for much more mundane tasks. Like address books, or personal information management, or maybe even tax records. Frontends to PostgreSQL, thanks to PostgreSQL's advanced features, are likely to span the gamut -- we already have OnShore TimeSheet for time tracking and payroll, as one example. And I even see database-backed intranet-style web scripts being used on a client workstation for these sorts of things. I personally do just that with my home Linux box -- I have a number of AOLserver dynamic pages that use PostgreSQL for many mundane tasks (a multilevel sermon database is one). While I don't need handholding in the upgrade process, I have provided support to users that do -- who are astonished at the way we upgrade.
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
Peter Eisentraut wrote: Lamar Owen writes: Getting the installation that 'make install' spits out massaged into an FHS compliant setup is the majority of the RPM's spec file. ./configure --prefix=/usr --sysconfdir=/etc Off you go... (I'll refrain from commenting further on the FHS.) I know alot of people don't like LSB/FHS, but, like it or not, I have to work with it. And, many many thanks for putting in the work on the configuration as you have. * Upgrades that don't require an ASCII database dump for migration. Let me ask you this question: When any given RPM-based Linux distribution will update their system from ext2 to, say, ReiserFS across the board, how are they going to do it? Sincere question. Like the TRS-80 model III, whose TRSDOS 1.3 could not read the TRS-80 Model I's disks, written on TRSDOS 2.3 (TRSDOS's versioning was absolutely horrendous). TRSDOS 1.3 included a CONVERT utility that could read files from the old filesystem. I'm sure that the newer distributions using ReiserFS as the primary filesystem will include legacy Ext2/3 support, at least for read-only, for many versions to come. And that's my big beef -- a newer version of PostgreSQL can't even pg_dump an old database. If that single function was supported, I would have no problem with the upgrade whatsoever. * A less source-centric mindset. Let's see, how to explain? The regression tests are a good example. You need make. You need the source installed, configured, and built in the usual location. This is not an excuse, but almost every package behaves this way. Test suites are designed to be run after "make all" and before "make install". When you ship a binary package then you're saying to users "I did the building and installation (and presumably everything else that the authors recommend along the way) for you." Yes, and I do just that. Regression testing is a regular part of my build process here. RPM packages usually don't work very well on systems that are not exactly like the one they were built on Boy, don't I know it.~;-/ Getting the regression tests to work from anywhere is not very hard, but it's not the most interesting project for most people. :-) I know. I'll probably do it myself, as that is something I _can_ do. I think I may have a solution for the library versioning problem. Rather than symlink libpq.so-libpq.so.2-libpq.so.2.x, I'll copy libpq.so.2.1 to libpq.so.2 and symlink libpq.so to that. I'd still claim that if RPM thinks it's smarter than the dynamic loader, then it's broken. All the shared libraries on Linux have a symlink from more general to more specific names. PostgreSQL can't be the first to hit this problem. RPM is getting it's .so dependency list straight from the mouth of the dynamic loader itself. RPM uses shell scripts, customizable for each system on which RPM runs, to determine the automatic dependencies -- those shell scripts run the dynamic loader to get the list of requires. So, the dynamic loader itself is providing the list. -- Lamar Owen WGCR Internet Radio 1 Peter 4:11
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
Lamar Owen writes: Getting the installation that 'make install' spits out massaged into an FHS compliant setup is the majority of the RPM's spec file. ./configure --prefix=/usr --sysconfdir=/etc Off you go... (I'll refrain from commenting further on the FHS.) * Upgrades that don't require an ASCII database dump for migration. Let me ask you this question: When any given RPM-based Linux distribution will update their system from ext2 to, say, ReiserFS across the board, how are they going to do it? Sincere question. * A less source-centric mindset. Let's see, how to explain? The regression tests are a good example. You need make. You need the source installed, configured, and built in the usual location. This is not an excuse, but almost every package behaves this way. Test suites are designed to be run after "make all" and before "make install". When you ship a binary package then you're saying to users "I did the building and installation (and presumably everything else that the authors recommend along the way) for you." RPM packages usually don't work very well on systems that are not exactly like the one they were built on, so this seems to be a fair assumption. Getting the regression tests to work from anywhere is not very hard, but it's not the most interesting project for most people. :-) I think I may have a solution for the library versioning problem. Rather than symlink libpq.so-libpq.so.2-libpq.so.2.x, I'll copy libpq.so.2.1 to libpq.so.2 and symlink libpq.so to that. I'd still claim that if RPM thinks it's smarter than the dynamic loader, then it's broken. All the shared libraries on Linux have a symlink from more general to more specific names. PostgreSQL can't be the first to hit this problem. -- Peter Eisentraut [EMAIL PROTECTED] http://yi.org/peter-e/
[HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
[ Blind CC to general added for comment below.] [Taken off GENERAL, added HACKERS to cc:] Bruce Momjian wrote: He's meaning the libpq version for dynamic link loading. Is the libpq.so lib changing versions (like the change from 6.5.x to 7.0.x changed from libpq.so.2.0 to libpq.so.2.1, which broke binary RPM compatibility for other RPM's linked against libpq.so.2.0, which failed when libpq.so.2.1 came on the scene). I think the answer is no, but I haven't checked the details yet. I usually up the .so version numbers before entering beta. That way, they get marked as newer than older versions. May I ask: is it necessary? Have there been version-bumping changes to libpq since 7.0.x? (With the rate that necessary improvement is happening to PostgreSQL, probably). No, only major releases have bumps. But, enough rant. That _is_ I believe what Trond was asking about. We have been bitten before with people installing the PHP from RedHat 6.2 after installing the PostgreSQL 7.0.x RPMset -- and dependency failures wreaked havoc. So, PostgreSQL 7.1 is slated to be libpq.so.2.2, then? Actually, Bruce, it would do me and Trond a great favor if a list of what so's are getting bumped and to what version were to be posted. At least we can plan for a transition at that point. See pgsql/src/tools/RELEASE_CHANGES. I edit interfaces/*/Makefile and increase the minor number for every interface by one. Let me add one thing on this RPM issue. There has been a lot of talk recently about RPM's, and what they should do, and what they don't do, and who should be blamed. Unfortunately, much of the discussion has been very unproductive and more like 'venting'. I really don't appreciate people 'venting' on these lists, especially since we have _nothing_ to do with RPM's. All we do is make the PostgreSQL software. If people want to discuss RPM's on the ports list, or want to create a new list just about RPM's, that's OK, but venting is bad, and venting on a list that has nothing to do with RPM's is even worse. What would be good is for someone to constructively make a posting about the known problems, and come up with acceptible solutions. Asking us to fix it really isn't going to help because we don't deal with RPM's here, and we don't have enough free time to make significant changes to meet the needs of RPM's. Also, remember we support many Unix platforms, and Linux is only one of them. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026
[HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
[Taken off GENERAL, added HACKERS to cc:] Bruce Momjian wrote: He's meaning the libpq version for dynamic link loading. Is the libpq.so lib changing versions (like the change from 6.5.x to 7.0.x changed from libpq.so.2.0 to libpq.so.2.1, which broke binary RPM compatibility for other RPM's linked against libpq.so.2.0, which failed when libpq.so.2.1 came on the scene). I think the answer is no, but I haven't checked the details yet. I usually up the .so version numbers before entering beta. That way, they get marked as newer than older versions. May I ask: is it necessary? Have there been version-bumping changes to libpq since 7.0.x? (With the rate that necessary improvement is happening to PostgreSQL, probably). Let me explain: RPM's contain a plethora of dependency information, some of which is added manually, but most of which is generated automatically. These dependencies are based on which 'soname' is needed to satisfy dynamic linking requirements, interpreter requirements, etc. With version numbers as part of the name, a change in version numbers changes the dependency. Unfortunately RPM deems a dependency upon libpq.so.2.0 to not be fulfilled by libpq.so.2.1 (how _can_ it know? A client linked to 2.0 might fail if 2.1 were to be loaded under it (hypothetically)). Now, that doesn't directly effect the PostgreSQL RPM's. What it does effect is the guy who wants to install PHP from with PostgreSQL support enabled and cannot because of a failed dependency. Who gets blamed? PostgreSQL. Trond may correct me on this, but I don't know of a workaround for this. And any workaround has to be applied to packages that depend upon PostgreSQL, not to the PostgreSQL RPM's (which I would gladly modify) -- although I am going to try something -- I know that a symlink to the old soname works, even though it is a kludge and, IMO, stinks like a polecat. But, enough rant. That _is_ I believe what Trond was asking about. We have been bitten before with people installing the PHP from RedHat 6.2 after installing the PostgreSQL 7.0.x RPMset -- and dependency failures wreaked havoc. So, PostgreSQL 7.1 is slated to be libpq.so.2.2, then? Actually, Bruce, it would do me and Trond a great favor if a list of what so's are getting bumped and to what version were to be posted. At least we can plan for a transition at that point. I just hate to pull a threepeat on RedHat customers. (RH 5.0 shipped PG 6.2.1. RH 5.1 shipped PG 6.3.2. BONG!) (RH 6.0 shipped 6.4.2 (bong!) RH 6.1 shipped 6.5.2 (double BONG!)). RH 7 shipped 7.0.x (small bong) -- RH 7.1 ships 7.1.x (ouch bong). Whew. Trond, you ready for this? [Note: I have been ill, so this message may be more incoherent than my normal scattered self] -- Lamar Owen WGCR Internet Radio 1 Peter 4:11
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
Lamar Owen [EMAIL PROTECTED] writes: Unfortunately RPM deems a dependency upon libpq.so.2.0 to not be fulfilled by libpq.so.2.1 (how _can_ it know? A client linked to 2.0 might fail if 2.1 were to be loaded under it (hypothetically)). If so, I claim RPM is broken. The whole point of major/minor version numbering for .so's is that a minor version bump is supposed to be binary-upward-compatible. If the RPM stuff has arbitrarily decided that it won't honor that definition, why do we bother with multiple numbers at all? So, PostgreSQL 7.1 is slated to be libpq.so.2.2, then? To answer your question, there are no pending changes in libpq that would mandate a major version bump (ie, nothing binary-incompatible, AFAIK). We could ship it with the exact same version number, but then how are people to tell whether they have a 7.0 or 7.1 libpq? regards, tom lane
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
Lamar Owen [EMAIL PROTECTED] writes: Unfortunately RPM deems a dependency upon libpq.so.2.0 to not be fulfilled by libpq.so.2.1 (how _can_ it know? A client linked to 2.0 might fail if 2.1 were to be loaded under it (hypothetically)). If so, I claim RPM is broken. The whole point of major/minor version numbering for .so's is that a minor version bump is supposed to be binary-upward-compatible. If the RPM stuff has arbitrarily decided that it won't honor that definition, why do we bother with multiple numbers at all? So, PostgreSQL 7.1 is slated to be libpq.so.2.2, then? To answer your question, there are no pending changes in libpq that would mandate a major version bump (ie, nothing binary-incompatible, AFAIK). We could ship it with the exact same version number, but then how are people to tell whether they have a 7.0 or 7.1 libpq? Yes, we need to have new numbers so binaries from different releases use the proper .so files. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026
[HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
What would be good is for someone to constructively make a posting about the known problems, and come up with acceptable solutions. Asking us to fix it really isn't going to help because we don't deal with RPM's here, and we don't have enough free time to make significant changes to meet the needs of RPM's. Which is why I stepped up to the plate last year to help with RPM's. I apologize if you took my post (which I edited greatly) as 'venting' -- it was not my intention to 'vent', much less offend. I just want to know what to expect from the 7.1 release. I feel that that is germane to the Hackers list, as the knowledge necessary to answer the question is to be found on the list. (and you answered the question above). No, I was not pointing to you when I mentioned venting. There have been other RPM threads lately. I just want information on how to make things better for RPM's, not vents. Like it or not, in the eyes of many people having solid RPM's is a core issue. If there are gotchas, I want to document them so people don't get blindsided. Or work around them. Or ask why the change is necessary in the first place. Sure. I appreciate the fact that we are not here to make it easy for distributors to package our software. I also appreciate the fact that if you don't at least make an effort to work with major distributors (and RedHat, TurboLinux, Caldera, and SuSE together comprise a major userbase) that you run the risk of not being distributed in favor of an inferior product. Let them. It is their decision. Frankly, I have seen this attitude before, and I don't like it. Just the mention that "Gee, if you don't cooperate, we may yank you," is really a veiled threat. Now, I know you aren't saying that, but the "if you don't play nice, we will drop you" argument sounds a lot more like MS that a Linux vendor should be acting, especially since they are not telling us what they want or assisting in the work. The "We are big. Just fix it and let us know when it is ready" attitude does not work here, and that is what I am hearing mostly from the RPM people. I also appreciate and applaud the cross-platform mentality of the PostgreSQL developers. Linux is far from the only OS to be supported by PostgreSQL, true. But Linux is also the most popular OS for PostgreSQL deployment. True, it is the most popular, but that doesn't make the others less important. This whole statement comes across as, "You run on Linux, and look, you took the time to run on other OS's too. How quaint." In the history of this project, Linux was an after-thought. None of our platforms are inferior or superior, except to the extent that the platform does not support Unix standard functions (like NT/Cygwin). However, there are known problems that can bite people who are not using RPM's and are not running Linux. Some of those problems are such that it will take someone with more knowledge than I currently possess to solve. One is the issue of upgrading/migrating tools. This is not an RPM-specific issue. To me, that is the only big issue that I have spoken about in a way that could even remotely be construed as 'venting'. And it is not a Linux-specific issue. It is a core issue. Again, your comments where quite helpful. We need more of them. We need to hear more about the problems people are having with RPM's, and how to make them better. There must be a list of known problems. Let's hear them, so we can try to solve them as a group. However, in general, we do not make dramatic change to work around OS bugs, and do not plan to make major changes to work around the limitations of RPM's. My bet is that some middle layer can be created that will fix that for us. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
Tom Lane [EMAIL PROTECTED] writes: Lamar Owen [EMAIL PROTECTED] writes: Unfortunately RPM deems a dependency upon libpq.so.2.0 to not be fulfilled by libpq.so.2.1 (how _can_ it know? A client linked to 2.0 might fail if 2.1 were to be loaded under it (hypothetically)). You link against libpq.so.2 , not libpq.so.2.1. This isn't a problem. If the RPM stuff has arbitrarily decided that it won't honor that definition, why do we bother with multiple numbers at all? There is no such problem. So, PostgreSQL 7.1 is slated to be libpq.so.2.2, then? To answer your question, there are no pending changes in libpq that would mandate a major version bump (ie, nothing binary-incompatible, AFAIK). We could ship it with the exact same version number, but then how are people to tell whether they have a 7.0 or 7.1 libpq? If there isn't any changes, why bump it? -- Trond Eivind Glomsrød Red Hat, Inc.
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
To answer your question, there are no pending changes in libpq that would mandate a major version bump (ie, nothing binary-incompatible, AFAIK). We could ship it with the exact same version number, but then how are people to tell whether they have a 7.0 or 7.1 libpq? If there isn't any changes, why bump it? This is huge software. There are changes to every library in every major release, major for us meaning, i.e., 7.0-7.1. That is why I bump the numbers. The interesting issue is that the version number changes for .so do _not_ mean they only talk with servers of the same release. They will talk to future servers of higher release numbers. This is done because there is a backend protocol number that is passed from client to server which determines how the server should behave with that client. We can't always have new clients talking to older servers because the old servers may not know the newer protocol. We could get fancy and trade version numbers and try to get it working, but it has not been a priority, and few have asked for it. Having old clients talking to new databases has been enough for most users. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
Tom Lane wrote: Lamar Owen [EMAIL PROTECTED] writes: Unfortunately RPM deems a dependency upon libpq.so.2.0 to not be fulfilled by libpq.so.2.1 (how _can_ it know? A client linked to 2.0 might fail if 2.1 were to be loaded under it (hypothetically)). If so, I claim RPM is broken. The whole point of major/minor version numbering for .so's is that a minor version bump is supposed to be binary-upward-compatible. If the RPM stuff has arbitrarily decided that it won't honor that definition, why do we bother with multiple numbers at all? So, PostgreSQL 7.1 is slated to be libpq.so.2.2, then? To answer your question, there are no pending changes in libpq that would mandate a major version bump (ie, nothing binary-incompatible, AFAIK). We could ship it with the exact same version number, but then how are people to tell whether they have a 7.0 or 7.1 libpq? And that is a very good point. Hey, I'm caught in the middle here :-). I want to see PostgreSQL succeed and excel (which, to me, means becoming the RDBMS of choice) on RPM-based Linux distributions, which I am sure is a goal of others too. And I'm sure no one here is against that. But, there is friction between RedHat's (to use the first example of a distributor to pop into my head) needs and the needs of the PostgreSQL group. My gut feel is that RedHat may be better off shipping 7.0.x if the library version numbers are a contributory problem. The data upgrade problem is a bigger problem. To which RedHat might just want to stay at 7.0.x until either a tool is written to painlessly migrate or until the next major RedHat is released. Of course, that doesn't affect what I do as far as building 7.1 RPM's for distribution from the PostgreSQL site (or by anyone who so desires to distribute them). I have no choice for my own self but to stay on the curve. I need TOAST and OUTER JOINS too much. So, what I feel may be the best compromise is for RedHat (and myself) to continue building 7.0.x RPM's with bugfixes, etc, while I build 7.1 ad subsequent RPMset's for those who know what they're doing and not blindly upgrading their systems. Trond, do you have any comments on that? Or is the likely migration to kernel 2.4 in the next RedHat going to make a compatability compromise here moot? -- Lamar Owen WGCR Internet Radio 1 Peter 4:11
[HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
Bruce Momjian [EMAIL PROTECTED] writes: Let them. It is their decision. Frankly, I have seen this attitude before, and I don't like it. Just the mention that "Gee, if you don't cooperate, we may yank you," is really a veiled threat. Now, I know you aren't saying that, but the "if you don't play nice, we will drop you" argument sounds a lot more like MS that a Linux vendor should be acting, especially since they are not telling us what they want or assisting in the work. FWIW, I've never threatened to do so. If I wanted to, I would just do it[1] - threats are bad and never cause anything but bad feelings. That being said, my favorite wishes (in addition to as much SQL compliance and performance as possible, of course) are: * migration on upgrade * old libraries being able to speak to newer databases, so old binaries can continue working after database upgrades * good sonames on libraries - if a library hasn't changed, bumping the number to show it's part of a new version isn't necesarry. If it is backwards compatible, just bump the minor version, if it isn't, bump the major version. Or even better, use versioned symbols (I don't know how many other OSes than Linux and Solaris supports this, though). As for assisting, at least Red Hat contributes to a lot of projects, some of which are important to postgres on one or more platforms: gdb, gcc, glibc and the linux kernel. There just isn't enough resources to do everything, but I try to help out with the RPMs. When we make patches for packages, we try to cooperate with the author(s) to get them in - happily, we haven't had much of a need for that with postgresql. The "We are big. Just fix it and let us know when it is ready" attitude does not work here, and that is what I am hearing mostly from the RPM people. I haven't heard anyone say that. There must be a list of known problems. Let's hear them, so we can try to solve them as a group. However, in general, we do not make dramatic change to work around OS bugs, and do not plan to make major changes to work around the limitations of RPM's. I don't think there are any apart from the upgrade issues - if library versioning follows the standard, that certainly won't be a problem. [1] which I'm not even close to doing - I've spent a bit of time lately hunting down aliasing bugs in MySQL which causes wrong SQL query results if compiled with "-O2". Ouch. -- Trond Eivind Glomsrød Red Hat, Inc.
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
Bruce Momjian wrote: I appreciate the fact that we are not here to make it easy for distributors to package our software. I also appreciate the fact that if you don't at least make an effort to work with major distributors (and RedHat, TurboLinux, Caldera, and SuSE together comprise a major userbase) that you run the risk of not being distributed in favor of an inferior product. Let them. It is their decision. Frankly, I have seen this attitude before, and I don't like it. Just the mention that "Gee, if you don't cooperate, we may yank you," is really a veiled threat. I don't even see it as a veiled threat, Bruce. It simply _is_ a threat. There are other RDBMS choices. Currently PostgreSQL is the Officially Sanctioned RDBMS for multiple Linux distributions. As our capabilities increase, it will make us more and more attractive as the Choice, Top Shelf Open Source RDBMS. However, the upgrade gotcha has left a very bitter taste in more than one user's mouth. I'll not say more about that now, as I've said quite enough in the past. And I'm still trying to figure out enough of the internals of the storage manager to try to write the migration tools myself. But, I have other fish to fry right now, the biggest being cross-distribution RPM's. Linux is far from the only OS to be supported by PostgreSQL, true. But Linux is also the most popular OS for PostgreSQL deployment. True, it is the most popular, but that doesn't make the others less important. No, it doesn't. This whole statement comes across as, "You run on Linux, and look, you took the time to run on other OS's too. How quaint." I ran Unix before there was linux. I ran Unix years before Linus was even out of High School. Well, that is if you count Tandy Xenix V7 and System III as Unix. Or ATT 3B1 SysVR2. Or Apollo DomainOS SR10.2. Or Ultrix on a VAX 11/750 (running in tandem with VMS). And I'm considering moving my most critical public servers from Linux over to OpenBSD. A Linux bigot I'm not. However, there are known problems that can bite people who are not using RPM's and are not running Linux. Some of those problems are such that it will take someone with more knowledge than I currently possess to Again, your comments where quite helpful. We need more of them. We need to hear more about the problems people are having with RPM's, and how to make them better. Bruce, sometimes I fear my own lack of communications skills. If I can make my wife fighting mad at me with me having no clue as to what I said that made her mad, I fear I can make anyone mad, without knowing what I said to do so. So, I guess you could say I'm a little paranoid about my communications skills. So, I'm glad you considered my comments helpful -- I was beginning to get worried. There must be a list of known problems. Let's hear them, so we can try to solve them as a group. However, in general, we do not make dramatic change to work around OS bugs, and do not plan to make major changes to work around the limitations of RPM's. My bet is that some middle layer can be created that will fix that for us. Meet Mr. Middle Layer. :-) The PostgreSQL spec file that controls the RPM build is one of the most complex ones in the RedHat distribution, AFAIK. There's the middle layer. It does quite a bit of finagling already. And the work that Peter E is doing is helping my cause significantly. Bruce, when I recover fully from the illness I've had the last few days, I'll try to come up with a coherent listing of what I've had to work around in the past. My current headache won't let me think straight right now, which makes it likely that I won't effectively communicate the issues. -- Lamar Owen WGCR Internet Radio 1 Peter 4:11
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
Trond Eivind Glomsrød wrote: Lamar Owen [EMAIL PROTECTED] writes: Unfortunately RPM deems a dependency upon libpq.so.2.0 to not be fulfilled by libpq.so.2.1 (how _can_ it know? A client linked to 2.0 might fail if 2.1 were to be loaded under it (hypothetically)). There usually are no such problems, and I'm not aware of any specific to postgresql either. There have been reports to the pgsql-bugs list and to the PHP list about this very issue. -- Lamar Owen WGCR Internet Radio 1 Peter 4:11
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
However, the upgrade gotcha has left a very bitter taste in more than one user's mouth. I'll not say more about that now, as I've said quite enough in the past. And I'm still trying to figure out enough of the internals of the storage manager to try to write the migration tools myself. But, I have other fish to fry right now, the biggest being cross-distribution RPM's. Actually, I would prefer to see how we can improve what we have before making a binary conversion utility that will have to be updated for every release. Meet Mr. Middle Layer. :-) The PostgreSQL spec file that controls the RPM build is one of the most complex ones in the RedHat distribution, AFAIK. There's the middle layer. It does quite a bit of finagling already. Yes, I suspected the RPM was the middle layer. To the extent we can make that easier, let's hear it. Tell us what you need to do, and what you can't do, and see if any of us can figure out how to make things easier. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
Ok, here goes: Cool, a list. * Location-agnostic installation. Documentation (which I'll be happy to contribute) on that. Peter E is already working in this area. Getting the installation that 'make install' spits out massaged into an FHS compliant setup is the majority of the RPM's spec file. Well, we certainly don't want to make changes that make things harder or more confusing for non-RPM installs. How are they affected here? * Upgrades that don't require an ASCII database dump for migration. This can either be implemented as a program to do a pg_dump of an arbitrary version of data, or as a binary migration utility. Currently, I'm saving old executables to run under a special environment to pull a dump -- but it is far from optimal. What if the OS upgrade behind 99% of the upgrades makes it where those old executables can't run due to binary incompatibility (say I'm going from RedHat 3.0.3 to RedHat 7 -- 3.0.3, IIRC, as a.out...( and I know 3.0.3 didn't have PostgreSQL RPMs).)? What I could actually do to prevent that problem is build all of PostgreSQL's 6.1.x, 6.2.x, 6.3.x, 6.4.x, and 6.5.x and include the necessary backend executables as part of the RPM But I think you see the problem there. However, that would in my mind be better than the current situation, albeit taking up a lot of space. I really don't see the issue here. We can compress ASCII dump files, so the space need should not be too bad. Can't you just check to see if there is enough space, and error out if there is not? If the 2GIG limit is a problem, can't the split utility drop the files in 2gig chunks that can be pasted together in a pipe on reload? * A less source-centric mindset. Let's see, how to explain? The regression tests are a good example. You need make. You need the source installed, configured, and built in the usual location. You need portions of contrib. RPM's need to be installable on compiler-crippled servers for security. While the demand for regression testing on such a box may not be there, it certainly does give the user something to use to get standard output for bug reports. As a point, I run PostgreSQL in production on a compilerless machine. No compiler == more security. And Linux has enough security problems without a compiler being available :-(. Oh, and I have no make on that machine either. Well, no compiler? I can't see how we would do that without making other OS installs harder. That is really the core of the issue. We can't be making changes that make things harder for other OS's. Those have to be isolated in the RPM, or in some other middle layer. The documentation as well as many of the examples assume too much, IMHO, about the install location and the install methodology. Well, if we are not specific, things get very confusing for those other OS's. Being specific about locations makes things easier. Seems we may need to patch RPM installs to fix that. Certainly a pain, but I see no other options. I think I may have a solution for the library versioning problem. Rather than symlink libpq.so-libpq.so.2-libpq.so.2.x, I'll copy libpq.so.2.1 to libpq.so.2 and symlink libpq.so to that. A little more code for me. There is no real danger in version confusion with RPM's versioning and upgrade methodology, as long as you consistently use the RPMset. The PostgreSQL version number is readily found from an RPM database query, making the so version immaterial. Oh, that is good. The upgrade issue is the hot trigger for me at this time. It is and has been a major drain on my time and effort, as well as Trond's and others, to get the RPM upgrade working even remotely smoothly. And I am willing to code -- once I know how to go about doing it in the backend. Please give us more information about how the current upgrade is a problem. We don't hear that much from other OS's. How are RPM's specific, and maybe we can get a plan for a solution. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026
Re: [HACKERS] Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)
Bruce Momjian wrote: However, the upgrade gotcha has left a very bitter taste in more than one user's mouth. I'll not say more about that now, as I've said quite enough in the past. And I'm still trying to figure out enough of the internals of the storage manager to try to write the migration tools myself. But, I have other fish to fry right now, the biggest being cross-distribution RPM's. Actually, I would prefer to see how we can improve what we have before making a binary conversion utility that will have to be updated for every release. Meet Mr. Middle Layer. :-) The PostgreSQL spec file that controls the RPM build is one of the most complex ones in the RedHat distribution, AFAIK. There's the middle layer. It does quite a bit of finagling already. Yes, I suspected the RPM was the middle layer. To the extent we can make that easier, let's hear it. Tell us what you need to do, and what you can't do, and see if any of us can figure out how to make things easier. Ok, here goes: * Location-agnostic installation. Documentation (which I'll be happy to contribute) on that. Peter E is already working in this area. Getting the installation that 'make install' spits out massaged into an FHS compliant setup is the majority of the RPM's spec file. * Upgrades that don't require an ASCII database dump for migration. This can either be implemented as a program to do a pg_dump of an arbitrary version of data, or as a binary migration utility. Currently, I'm saving old executables to run under a special environment to pull a dump -- but it is far from optimal. What if the OS upgrade behind 99% of the upgrades makes it where those old executables can't run due to binary incompatibility (say I'm going from RedHat 3.0.3 to RedHat 7 -- 3.0.3, IIRC, as a.out...( and I know 3.0.3 didn't have PostgreSQL RPMs).)? What I could actually do to prevent that problem is build all of PostgreSQL's 6.1.x, 6.2.x, 6.3.x, 6.4.x, and 6.5.x and include the necessary backend executables as part of the RPM But I think you see the problem there. However, that would in my mind be better than the current situation, albeit taking up a lot of space. * A less source-centric mindset. Let's see, how to explain? The regression tests are a good example. You need make. You need the source installed, configured, and built in the usual location. You need portions of contrib. RPM's need to be installable on compiler-crippled servers for security. While the demand for regression testing on such a box may not be there, it certainly does give the user something to use to get standard output for bug reports. As a point, I run PostgreSQL in production on a compilerless machine. No compiler == more security. And Linux has enough security problems without a compiler being available :-(. Oh, and I have no make on that machine either. The documentation as well as many of the examples assume too much, IMHO, about the install location and the install methodology. I think I may have a solution for the library versioning problem. Rather than symlink libpq.so-libpq.so.2-libpq.so.2.x, I'll copy libpq.so.2.1 to libpq.so.2 and symlink libpq.so to that. A little more code for me. There is no real danger in version confusion with RPM's versioning and upgrade methodology, as long as you consistently use the RPMset. The PostgreSQL version number is readily found from an RPM database query, making the so version immaterial. The upgrade issue is the hot trigger for me at this time. It is and has been a major drain on my time and effort, as well as Trond's and others, to get the RPM upgrade working even remotely smoothly. And I am willing to code -- once I know how to go about doing it in the backend. -- Lamar Owen WGCR Internet Radio 1 Peter 4:11
