Re: [HACKERS] RFC: PostgreSQL Add-On Network
robertmh...@gmail.com (Robert Haas) writes:
On Fri, Jan 8, 2010 at 10:12 AM, Dave Page dp...@pgadmin.org wrote:
I have long spoken against making Windows a second class citizen. But I
don't think David is going to do that (and I'll hound him if he does). But
that doesn't mean it has to be fully supported from day one.
I'm not saying it should be supported from day 1, but I think the
initial plan will make it very difficult to add Windows support later
without a great deal of rewriting/redesign. It's lack of forward
planning I was objecting to.
I personally suspect that the client is not the most important part of
this project. I think the value of CPAN is for searching, more than
auto-installing. Personally, I never use the auto-install feature
because I always want more control than you get that way. I just use
the site to find possible modules and browse the docs, and then if I
find something I like I check with I can pull it from the Red Hat
repos with rpm, and if not I download it and look it over to see if it
DWIW, and then if so I usually make a private SRPM for it and install
from that. I'd be happy if we just had a good search-and-download
site.
If PGAN leads to us having:
a) A database containing a useful set of metadata about a large set of
extensions, and
b) A way for PostgreSQL developers and binary distribution makers (who
*do* have GCC / XCode / MingW / Visual Studio / ... available to
them) to easily:
- build
- test
- try out
- think about how to package
that large set of extensions
then we've got a Big Win of the same sort as CPAN, Ruby Gems, and PyPI.
It does NOT need to include installers for every known kind of computer;
that is a *second* problem, which actually requires a series of
solutions for:
a) Fedora
b) Debian (hence derivatives like Ubuntu)
c) BSD Ports
d) Yes, Windows
e) I think Solaris has something new for packaging...
If David gets it to the point where it's easy to build and install
extensions into a PostgreSQL installation, then turning that into
packages for specific targets should be a not-insurmountable problem
that may be treated separately.
That having been said, we should consider our filesystem layout
carefully however to make sure that if we want to provide things like
Windows installers in the future, we have a clean way to do that.
If the extensions get installed in a way that is scalable in the sense
that it's not a particularly big deal to write a script that pulls 250
extensions and installs them on a particular host for a particular PG
installation, then I'd think that the exercise has been a successful
one.
That leads, naturally enough, to an Extension BuildFarm :-).
I'd be somewhat surprised if the use of Windows was a material factor in
the matter.
--
wm(X,Y):-write(X),write('@'),write(Y). wm('cbbrowne','acm.org').
http://www3.sympatico.ca/cbbrowne/postgresql.html
Laugh-a while you can, Monkey Boy. -- Dr. Lizardo - Buckaroo Banzai
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Jan 7, 2010, at 4:07 PM, Josh Berkus wrote: Building a simple solution which doesn't initially cover all bases but can be steadily improved is a far superior strategy to trying to spec The Perfect Solution before even starting work. And if we want to keep recruiting new contributors, criticism needs to be more constructive. I think we should call it Pecan. And paint it yellow! ;P +1 from me. -- Jim C. Nasby, Database Architect j...@nasby.net 512.569.9461 (cell) http://jim.nasby.net -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Fri, Jan 8, 2010 at 05:22, Ron Mayer rm...@cheapcomplexdevices.com wrote: David E. Wheeler wrote: On Jan 7, 2010, at 1:31 PM, Dave Page wrote: No, I'm suggesting the mechanism needs to support source and binary distribution. For most *nix users, source will be fine. For Windows binaries are required. I would love to follow what Strawberry Perl has done to solve this problem. In 2.0. +1. They did a nice job. For those of us who have no idea what Strawberry Perl did (other than not shipping Microsoft compatible libraries, and is thus useless for PostgreSQL), could someone explain it? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Thu, Jan 7, 2010 at 10:07 PM, Josh Berkus j...@agliodbs.com wrote: Building them is no problem - authors can easily use EC2 for which we have an AMI pre-configured for next to no cost, can build on their own platform, on a community provided system, or get a friend to do it. So any module author, in order to submit any module, would be required to build binaries for 8-12 platforms covering up to 5 PostgreSQL versions (i.e. 30-70 different binaries)? At their own cost? Seems like a good way to not get any contributions at all. No, I'm suggesting that for the module authors that want to support Windows, there are a variety of easy ways to get binaries built. It shouldn't be a requirement of *them*. For that matter, Andrew just pointed out to me (corrected me, actually) on IRC that if a Windows user has MSVC or Mingw installed, it should be no problem supporting them. If it were likely that Windows users would have a suitable compiler installed, I wouldn't be complaining. So what you're asking has nothing to do with Windows users, but is a more general we want support for users who don't have a compiler installed. That's a different problem -- and one which the One-click installer or Stackbuilder should probably solve rather than PGAN. It already does - on Windows, Linux and Mac. I don't need this feature, and nor do the users of my distributions. I just thought I was being helpful by pointing out some problems in David's design which I assumed was the point of the RFC. No. The essence is, 'If you're going to do it in a way that will never work for maybe 50% or more of PostgreSQL installations, then you have fundamental design issues to overcome'. Again, that's the wrong attitude. You're saying If it doesn't work for 100% of Windows users from Day 1, it won't ever work for them. By that logic, we should have held back version 7.4 until the windows port was done, dammit! And we shouldn't have released until it worked with Visual C++. Even if it meant releasing in 2007. And we shouldn't have released PITR until it supported HS. And we shouldn't be releasing 8.5 without Win64 support. Please re-read what I said before churning out ridiculous arguments that in no way relate to what I wrote. I am saying that if the design won't ever work without requiring painful dependency installation that users will likely not want to bother with, then it is fundamentally broken. Better to write one system that can _eventually_ work everywhere, than one that works for some, and then another that works elsewhere. This list *also* has a real tendency to have an incredible negative attitude, which *you* are currently expressing. The constructive way for you to approach this would have been to say I think that the general idea has merit, but that putting off Windows support is a mistake. What about supporting binary distribution at the outset? Again, please read what I wrote. That is almost exactly what I did write. I said it was a good idea, I explained why I thought ignoring Windows was an issue, and I noted that we had previously discussed binary distribution. coding the client in C? Instead, you said this doesn't solve problems A, B, and C, so it's stupid. No - I said it won't work for most windows users, use C instead as that'll work everywhere. At no point did I say or imply it was stupid, just that it wasn't a universal solution. Building a simple solution which doesn't initially cover all bases but can be steadily improved is a far superior strategy to trying to spec The Perfect Solution before even starting work. And if we want to keep recruiting new contributors, criticism needs to be more constructive. For every criticism I made, I gave my reasoning, and offered a solution or an idea that would resolve my concern. In what bloody way is that not constructive? Remember, David posted an RFC. I'm not going to fill this list with grovelling bullshit about how every line he wrote is genius. I commented on the parts I considered a problem, and gave an overall nod to the rest. If all you want on this list is positive comments, then I won't bother any more as that won't help anyone. /D -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
Hi, Josh Berkus wrote: Dave wrote: and frankly, isn't the way this project generally works. Isn't it? We didn't even support Windows for quite a long time. We still have lots more active Unix developers and knowledge that Windows ones. And isn't there some scratch your own itch philosophy in every OSS project? But why is that *this* project's job to solve? It's already the case that contrib modules (or PostgreSQL) are not buildable on Windows in an automated fashion. And that Windows does not ship with developer tools. That's not PGAN's fault. I'm with Josh here, get a PGAN up and running. Make it accessible via web to promote extensibility of Postgres and availability of extensions. Whether the first incarnation of the PGAN client works on Windows or Linux doesn't matter for now. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Fri, Jan 8, 2010 at 10:40 AM, Markus Wanner mar...@bluegap.ch wrote: Hi, Josh Berkus wrote: Dave wrote: and frankly, isn't the way this project generally works. Isn't it? We didn't even support Windows for quite a long time. No, it's quite different for the PostgreSQL not to support a platform at all, than for it to say it's supported on FOO/OS, but you can't use Window Functions or on BAR/OS, you can't use PITR. The only reason we ever offer different functionality on different platforms is when there are external reasons forcing us to - for example, lack of reparse points in NTFS on Windows NT 4.0 prevented us offering table space support, and for some time we had no Win32 port of libuuid, so couldn't offer the UUID contrib module. There are no such problems with this project that I can see - at least that cannot be overcome with a little thought and appropriate tool selection. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
Dave Page wrote: The only reason we ever offer different functionality on different platforms is when there are external reasons forcing us to - for example, lack of reparse points in NTFS on Windows NT 4.0 prevented us offering table space support, and for some time we had no Win32 port of libuuid, so couldn't offer the UUID contrib module. There are no such problems with this project that I can see - at least that cannot be overcome with a little thought and appropriate tool selection. Dave, Windows came late to the buildfarm. According to the CVS log, the buildfarm client was first checked in in Sept 2004, got initial Mingw support in Jan 2005 and MSVC support in March 2007, when we finally got some of the tools sorted out. I have long spoken against making Windows a second class citizen. But I don't think David is going to do that (and I'll hound him if he does). But that doesn't mean it has to be fully supported from day one. Personally, I'd like to see us with a build service for such modules (c.f. OpenSuse's build service). cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
Magnus Hagander wrote: On Fri, Jan 8, 2010 at 05:22, Ron Mayer rm...@cheapcomplexdevices.com wrote: David E. Wheeler wrote: On Jan 7, 2010, at 1:31 PM, Dave Page wrote: No, I'm suggesting the mechanism needs to support source and binary distribution. For most *nix users, source will be fine. For Windows binaries are required. I would love to follow what Strawberry Perl has done to solve this problem. In 2.0. +1. They did a nice job. For those of us who have no idea what Strawberry Perl did (other than not shipping Microsoft compatible libraries, and is thus useless for PostgreSQL), could someone explain it? As far as I can tell they shipped the minimal set of tools that they needed to build extensions rather than distribute binaries. http://en.wikipedia.org/wiki/Strawberry_Perl I don't know the details, but it works smoothly for them. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Fri, Jan 8, 2010 at 15:14, Ron Mayer rm...@cheapcomplexdevices.com wrote: Magnus Hagander wrote: On Fri, Jan 8, 2010 at 05:22, Ron Mayer rm...@cheapcomplexdevices.com wrote: David E. Wheeler wrote: On Jan 7, 2010, at 1:31 PM, Dave Page wrote: No, I'm suggesting the mechanism needs to support source and binary distribution. For most *nix users, source will be fine. For Windows binaries are required. I would love to follow what Strawberry Perl has done to solve this problem. In 2.0. +1. They did a nice job. For those of us who have no idea what Strawberry Perl did (other than not shipping Microsoft compatible libraries, and is thus useless for PostgreSQL), could someone explain it? As far as I can tell they shipped the minimal set of tools that they needed to build extensions rather than distribute binaries. http://en.wikipedia.org/wiki/Strawberry_Perl I don't know the details, but it works smoothly for them. Right. They are msys compatible, not Microsoft compatible. And msys uses non-Windows-standard format of the import libraries. and the import libraries are needed for people to link to the interpreter, like pl/perl. Also a quick re-check shows that there appears to be no 64-bit version available. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
Hey Andrew On Fri, Jan 8, 2010 at 12:57 PM, Andrew Dunstan and...@dunslane.net wrote: Windows came late to the buildfarm. According to the CVS log, the buildfarm client was first checked in in Sept 2004, got initial Mingw support in Jan 2005 and MSVC support in March 2007, when we finally got some of the tools sorted out. Right - but the buildfarm isn't a feature being offered to end users. I have long spoken against making Windows a second class citizen. But I don't think David is going to do that (and I'll hound him if he does). But that doesn't mean it has to be fully supported from day one. I'm not saying it should be supported from day 1, but I think the initial plan will make it very difficult to add Windows support later without a great deal of rewriting/redesign. It's lack of forward planning I was objecting to. Personally, I'd like to see us with a build service for such modules (c.f. OpenSuse's build service). +1 -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Fri, Jan 8, 2010 at 10:12 AM, Dave Page dp...@pgadmin.org wrote: I have long spoken against making Windows a second class citizen. But I don't think David is going to do that (and I'll hound him if he does). But that doesn't mean it has to be fully supported from day one. I'm not saying it should be supported from day 1, but I think the initial plan will make it very difficult to add Windows support later without a great deal of rewriting/redesign. It's lack of forward planning I was objecting to. I personally suspect that the client is not the most important part of this project. I think the value of CPAN is for searching, more than auto-installing. Personally, I never use the auto-install feature because I always want more control than you get that way. I just use the site to find possible modules and browse the docs, and then if I find something I like I check with I can pull it from the Red Hat repos with rpm, and if not I download it and look it over to see if it DWIW, and then if so I usually make a private SRPM for it and install from that. I'd be happy if we just had a good search-and-download site. That having been said, we should consider our filesystem layout carefully however to make sure that if we want to provide things like Windows installers in the future, we have a clean way to do that. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Fri, 2010-01-08 at 15:12 +, Dave Page wrote: Hey Andrew On Fri, Jan 8, 2010 at 12:57 PM, Andrew Dunstan and...@dunslane.net wrote: Windows came late to the buildfarm. According to the CVS log, the buildfarm client was first checked in in Sept 2004, got initial Mingw support in Jan 2005 and MSVC support in March 2007, when we finally got some of the tools sorted out. Right - but the buildfarm isn't a feature being offered to end users. Neither is this. These people are developers or DBAs after all. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Fri, Jan 8, 2010 at 3:56 PM, Joshua D. Drake j...@commandprompt.com wrote: On Fri, 2010-01-08 at 15:12 +, Dave Page wrote: Hey Andrew On Fri, Jan 8, 2010 at 12:57 PM, Andrew Dunstan and...@dunslane.net wrote: Windows came late to the buildfarm. According to the CVS log, the buildfarm client was first checked in in Sept 2004, got initial Mingw support in Jan 2005 and MSVC support in March 2007, when we finally got some of the tools sorted out. Right - but the buildfarm isn't a feature being offered to end users. Neither is this. These people are developers or DBAs after all. I know I'm a little slower than normal this week as I've had some sort of virus, but say what? I was kinda under the impression that developers/DBAs are end users of PostgreSQL. They make up a significant percentage of people that I know than install, configure and use it. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Fri, 2010-01-08 at 16:33 +, Dave Page wrote: On Fri, Jan 8, 2010 at 3:56 PM, Joshua D. Drake j...@commandprompt.com wrote: On Fri, 2010-01-08 at 15:12 +, Dave Page wrote: Hey Andrew On Fri, Jan 8, 2010 at 12:57 PM, Andrew Dunstan and...@dunslane.net wrote: Windows came late to the buildfarm. According to the CVS log, the buildfarm client was first checked in in Sept 2004, got initial Mingw support in Jan 2005 and MSVC support in March 2007, when we finally got some of the tools sorted out. Right - but the buildfarm isn't a feature being offered to end users. Neither is this. These people are developers or DBAs after all. I know I'm a little slower than normal this week as I've had some sort of virus, but say what? I was kinda under the impression that developers/DBAs are end users of PostgreSQL. They make up a significant percentage of people that I know than install, configure and use it. My point is... if they are a developer or DBA, they aren't going to have any trouble dealing with these issues. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Fri, Jan 8, 2010 at 4:34 PM, Joshua D. Drake j...@commandprompt.com wrote: On Fri, 2010-01-08 at 16:33 +, Dave Page wrote: On Fri, Jan 8, 2010 at 3:56 PM, Joshua D. Drake j...@commandprompt.com wrote: On Fri, 2010-01-08 at 15:12 +, Dave Page wrote: Hey Andrew On Fri, Jan 8, 2010 at 12:57 PM, Andrew Dunstan and...@dunslane.net wrote: Windows came late to the buildfarm. According to the CVS log, the buildfarm client was first checked in in Sept 2004, got initial Mingw support in Jan 2005 and MSVC support in March 2007, when we finally got some of the tools sorted out. Right - but the buildfarm isn't a feature being offered to end users. Neither is this. These people are developers or DBAs after all. I know I'm a little slower than normal this week as I've had some sort of virus, but say what? I was kinda under the impression that developers/DBAs are end users of PostgreSQL. They make up a significant percentage of people that I know than install, configure and use it. My point is... if they are a developer or DBA, they aren't going to have any trouble dealing with these issues. What issues? Lack of support from PGAN, or having to download and install a complex build environment weighing in at over a gigabyte of downloads, just to install an add-on? -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
Dave, * Dave Page (dp...@pgadmin.org) wrote: Right - but the buildfarm isn't a feature being offered to end users. And this network isn't a feature of the core code either, nor, do I believe, is it being designed in a way that would require an overhaul down the road to support Windows. To be honest, I really don't feel this compares to features in the core code to begin with- I feel like generally we don't like things which only work on a subset of platforms because it's a small incremental change, and typically just requires good programming practice, to make it work for all supported platforms. That isn't true for this. I'm not saying it should be supported from day 1, but I think the initial plan will make it very difficult to add Windows support later without a great deal of rewriting/redesign. It's lack of forward planning I was objecting to. I disagree with this, but in the end I don't think that it really matters. If all we have resources for to work on this is a Perl expect, then by golly let that be a dependency for the initial work. If there are other resources who are willing to write the initial version in C, or what have you, let them step up and offer. This isn't a zero-sum game here and I'm strongly against telling someone please don't work on this when it's something which will be of benefit to many users. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Jan 8, 2010, at 1:35 AM, Dave Page wrote: I am saying that if the design won't ever work without requiring painful dependency installation that users will likely not want to bother with, then it is fundamentally broken. Better to write one system that can _eventually_ work everywhere, than one that works for some, and then another that works elsewhere. This whole bit about Windows is a red herring. Perhaps I should not have phrased it the way I did WRT Windows. So I'm going to change it to: The PGAN client will make no other assumptions about how to build and install extensions, leaving such to the PostgreSQL core. To the extent that PGXS-powered `make` works on a given platform, the client will support it. Discussing it with Andrew, that means it should work if you have mingw, and we might have to tweak it a bit to work with `src/tools/msvc`. So the point is: the PGAN client (which is just one part of this project, after all), will *not* include a build system. It will use whatever build system is supported by the community. Right now that's PGXS. If core switches to something later, or provides binary builds for Windows, the client will be easily adapted to take advantage of it. No sweat. The upshot is this: PGAN does *not* ignore Windows or any other platform. Rather, it relies on others to create the appropriate community-supported installers for all platforms. The issue of build systems and installers is not within its domain. Thus, I've also changed the FAQ to: * '''What about Windows?''' The PGAN client will always follow the lead of the PostgreSQL core on installing extensions. If support for installing extensions on Windows improves such that a compiler is no longer required, the PGAN client will be modified as appropriate to take advantage of it. This applies not specifically to Windows, but to the ability of the core intaller (or any future community-supported installer) to work on ''any'' platform. Please let the Windows thread die now. PGAN doesn't ignore Windows; it ignores installer development. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Fri, Jan 8, 2010 at 5:13 PM, David E. Wheeler da...@kineticode.com wrote: This whole bit about Windows is a red herring. Perhaps I should not have phrased it the way I did WRT Windows. So I'm going to change it to: The PGAN client will make no other assumptions about how to build and install extensions, leaving such to the PostgreSQL core. To the extent that PGXS-powered `make` works on a given platform, the client will support it. Discussing it with Andrew, that means it should work if you have mingw, and we might have to tweak it a bit to work with `src/tools/msvc`. So the point is: the PGAN client (which is just one part of this project, after all), will *not* include a build system. It will use whatever build system is supported by the community. Right now that's PGXS. If core switches to something later, or provides binary builds for Windows, the client will be easily adapted to take advantage of it. No sweat. The upshot is this: PGAN does *not* ignore Windows or any other platform. Rather, it relies on others to create the appropriate community-supported installers for all platforms. The issue of build systems and installers is not within its domain. Thus, I've also changed the FAQ to: If that is the goal of your project then I withdraw my previous comments, which were written on the belief that you were proposing a generic distribution/build/installation system for PostgreSQL users. Sorry for the noise! -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Fri, Jan 8, 2010 at 5:13 PM, David E. Wheeler da...@kineticode.com wrote: Please let the Windows thread die now. PGAN doesn't ignore Windows; it ignores installer development. yeah, I think there are two quite separable projects here. It's quite possible that once the binary installer people have a source package with all the meta data and computer-readable build instructions that PGAN will need that they'll be able to implement support for binary pre-built modules as well. But just tackling the source package problem solves a lot of the problems already. The binary installer people will have their own set of problems to solve as well, they'll have to deal with environment dependencies such as OS versions, external binary versions, privileges, etc. These are all additional problems on top of having a source package already. To make an analogy configure and make are still useful even after you have dpkg -- you still need to be able to build the individual source package first before you get to put it into a binary package. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Thu, Jan 7, 2010 at 21:07, David E. Wheeler da...@kineticode.com wrote: Hackers, I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][] to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL tools and the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedback and suggestions. [plan]: http://wiki.postgresql.org/wiki/PGAN [CPAN philosophy]: http://use.perl.org/article.pl?sid=02/11/12/1616209 [CPAN]: http://cpan.org/ [JSAN]: http://www.openjsan.org [JFDI]: http://acronyms.thefreedictionary.com/JFDI One note on a completely different topic from where this thread appears to have gone.. :-) Is there a particular reason not to use the existing mirroring network to distribute the files? If not, then I suggest using them should be part of the design. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Fri, Jan 8, 2010 at 5:38 PM, Magnus Hagander mag...@hagander.net wrote: On Thu, Jan 7, 2010 at 21:07, David E. Wheeler da...@kineticode.com wrote: Hackers, I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][] to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL tools and the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedback and suggestions. [plan]: http://wiki.postgresql.org/wiki/PGAN [CPAN philosophy]: http://use.perl.org/article.pl?sid=02/11/12/1616209 [CPAN]: http://cpan.org/ [JSAN]: http://www.openjsan.org [JFDI]: http://acronyms.thefreedictionary.com/JFDI One note on a completely different topic from where this thread appears to have gone.. :-) Is there a particular reason not to use the existing mirroring network to distribute the files? If not, then I suggest using them should be part of the design. With StackBuilder, a package can either be listed as a path, relative to the FTP root on the mirror network, or using an alternate URL for something stored elsewhere. The current set of active mirrors can always be found at http://www.postgresql.org/mirrors.xml, so you can build URLs on the mirror network using the protocol, host, port and path from the mirror list, and then the relative path for the file. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Fri, Jan 8, 2010 at 18:44, Dave Page dp...@pgadmin.org wrote: On Fri, Jan 8, 2010 at 5:38 PM, Magnus Hagander mag...@hagander.net wrote: On Thu, Jan 7, 2010 at 21:07, David E. Wheeler da...@kineticode.com wrote: Hackers, I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][] to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL tools and the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedback and suggestions. [plan]: http://wiki.postgresql.org/wiki/PGAN [CPAN philosophy]: http://use.perl.org/article.pl?sid=02/11/12/1616209 [CPAN]: http://cpan.org/ [JSAN]: http://www.openjsan.org [JFDI]: http://acronyms.thefreedictionary.com/JFDI One note on a completely different topic from where this thread appears to have gone.. :-) Is there a particular reason not to use the existing mirroring network to distribute the files? If not, then I suggest using them should be part of the design. With StackBuilder, a package can either be listed as a path, relative to the FTP root on the mirror network, or using an alternate URL for something stored elsewhere. The current set of active mirrors can always be found at http://www.postgresql.org/mirrors.xml, so you can build URLs on the mirror network using the protocol, host, port and path from the mirror list, and then the relative path for the file. Or you can hit it off the mirror redirector on the website, which will then give you automatic download tracking. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
2010/1/8 Magnus Hagander mag...@hagander.net: On Fri, Jan 8, 2010 at 18:44, Dave Page dp...@pgadmin.org wrote: The current set of active mirrors can always be found at http://www.postgresql.org/mirrors.xml, so you can build URLs on the mirror network using the protocol, host, port and path from the mirror list, and then the relative path for the file. Or you can hit it off the mirror redirector on the website, which will then give you automatic download tracking. Right, but you need the mirror ID for that iirc. Maybe we should add that to the XML... -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Jan 8, 2010, at 9:24 AM, Dave Page wrote: If that is the goal of your project then I withdraw my previous comments, which were written on the belief that you were proposing a generic distribution/build/installation system for PostgreSQL users. It is a generic distribution and installation system, but it just uses installer approaches provided by others. This is not unlike CPAN.pm, which doesn't include an installer itself, but things through Module::Build or ExtUtils::MakeMaker as appropriate. Completely separate domain problem, as Greg notes. Sorry for the noise! Glad to have it cleared up! :-) Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Jan 8, 2010, at 9:38 AM, Magnus Hagander wrote: Is there a particular reason not to use the existing mirroring network to distribute the files? If not, then I suggest using them should be part of the design. No, as long as PAUS can drop uploaded distributions onto the master FTP server, or else the existing mirror system can rsync from PGAN's own master (I'll build all this on my own box to start with). It'll just be rsync, really, it won't where it's mirrored or where the master index lives. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Fri, Jan 8, 2010 at 5:55 PM, David E. Wheeler da...@kineticode.com wrote: On Jan 8, 2010, at 9:38 AM, Magnus Hagander wrote: Is there a particular reason not to use the existing mirroring network to distribute the files? If not, then I suggest using them should be part of the design. No, as long as PAUS can drop uploaded distributions onto the master FTP server, or else the existing mirror system can rsync from PGAN's own master (I'll build all this on my own box to start with). It'll just be rsync, really, it won't where it's mirrored or where the master index lives. Either can be arranged. For StackBuilder, we created a pgFoundry project, so files can just be uploaded there by authorised users, from where they get replicated back onto the mirror network. Which leads us neatly back to the GForge URL thread :-) BTW, what's a PAUS? I missed that one... -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Jan 8, 2010, at 9:59 AM, Dave Page wrote: Either can be arranged. For StackBuilder, we created a pgFoundry project, so files can just be uploaded there by authorised users, from where they get replicated back onto the mirror network. Which leads us neatly back to the GForge URL thread :-) BTW, what's a PAUS? I missed that one... That's the very first part of PGAN: • PAUS: The PostgreSQL Authors Upload Server. Users will be able to create logins and upload extension distributions. Details in the wiki. http://wiki.postgresql.org/wiki/PGAN Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Fri, Jan 8, 2010 at 18:55, David E. Wheeler da...@kineticode.com wrote: On Jan 8, 2010, at 9:38 AM, Magnus Hagander wrote: Is there a particular reason not to use the existing mirroring network to distribute the files? If not, then I suggest using them should be part of the design. No, as long as PAUS can drop uploaded distributions onto the master FTP server, or else the existing mirror system can rsync from PGAN's own master (I'll build all this on my own box to start with). It'll just be rsync, really, it won't where it's mirrored or where the master index lives. Either can be easily arranged. Then you just need some logic in the client thingy to actually grab stuff from the mirrors. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Fri, Jan 8, 2010 at 18:48, Dave Page dp...@pgadmin.org wrote: 2010/1/8 Magnus Hagander mag...@hagander.net: On Fri, Jan 8, 2010 at 18:44, Dave Page dp...@pgadmin.org wrote: The current set of active mirrors can always be found at http://www.postgresql.org/mirrors.xml, so you can build URLs on the mirror network using the protocol, host, port and path from the mirror list, and then the relative path for the file. Or you can hit it off the mirror redirector on the website, which will then give you automatic download tracking. Right, but you need the mirror ID for that iirc. Maybe we should add that to the XML... That, or implement that send me to a random mirror feature. Or maybe the send me to a random close mirror if available, or a random global if not feature. :-) Either way, there's definitely room for some improvement there, but let's figure out what the exact needs are first :-) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Jan 8, 2010, at 10:08 AM, Magnus Hagander wrote: That, or implement that send me to a random mirror feature. Or maybe the send me to a random close mirror if available, or a random global if not feature. :-) Either way, there's definitely room for some improvement there, but let's figure out what the exact needs are first :- I'll come back to this when I'm actually implementing it, for sure. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
Magnus Hagander wrote: On Fri, Jan 8, 2010 at 18:48, Dave Page dp...@pgadmin.org wrote: 2010/1/8 Magnus Hagander mag...@hagander.net: On Fri, Jan 8, 2010 at 18:44, Dave Page dp...@pgadmin.org wrote: The current set of active mirrors can always be found at http://www.postgresql.org/mirrors.xml, so you can build URLs on the mirror network using the protocol, host, port and path from the mirror list, and then the relative path for the file. Or you can hit it off the mirror redirector on the website, which will then give you automatic download tracking. Right, but you need the mirror ID for that iirc. Maybe we should add that to the XML... That, or implement that send me to a random mirror feature. Or maybe the send me to a random close mirror if available, or a random global if not feature. :-) Either way, there's definitely room for some improvement there, but let's figure out what the exact needs are first :-) well we could fairly trivially do that using DNS as well as in providing sets of mirrors geographically spread out (like us.mirrors.postgresql.org, eu.mirrors.postgresql.org) or whatever. Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] RFC: PostgreSQL Add-On Network
Hackers, I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][] to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL tools and the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedback and suggestions. [plan]: http://wiki.postgresql.org/wiki/PGAN [CPAN philosophy]: http://use.perl.org/article.pl?sid=02/11/12/1616209 [CPAN]: http://cpan.org/ [JSAN]: http://www.openjsan.org [JFDI]: http://acronyms.thefreedictionary.com/JFDI Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Thu, Jan 7, 2010 at 8:07 PM, David E. Wheeler da...@kineticode.com wrote: Hackers, I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][] to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL tools and the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedback and suggestions. Whilst the aim is a noble one, glossing over 'it won't work on Windows' which is by far our most popular platform these days seems incredibly short sighted, and liable to lead to an endless stream of 'why doesn't this work' questions. It also does the module authors no favours, by excluding 50%+ of their potential audience, and frankly, isn't the way this project generally works. We have discussed this sort of facility at previous developer meetings, and as I recall came to the conclusion that we need to have the ability to distribute pre-built binaries, not just source code as virtually no Windows users are ever going to have a build environment setup. Similarly, neither are Mac users, if they didn't install XCode. We also discussed extension management at the DBMS level, which I believe Dimitri was working on in his spare time. You should look at what he's been doing. Finally, don't write the client in Perl. Again, that's of no use to most Windows users. C will work on all platforms from the outset, with no dependencies required. Of course, the server side doesn't matter. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Thu, 2010-01-07 at 20:36 +, Dave Page wrote: On Thu, Jan 7, 2010 at 8:07 PM, David E. Wheeler da...@kineticode.com wrote: Hackers, I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][] to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL tools and the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedback and suggestions. Whilst the aim is a noble one, glossing over 'it won't work on Windows' which is by far our most popular platform these days Although I understand your point, this is debatable to the point of a pointless point :P. The reason it seems like the most popular is that the only way to get the windows version is to download it from .Org/EDB. Linux/FreeBSD already has it in their own repositories. I do believe that we need to support windows, but not everything will. That is just the nature of it. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Thu, Jan 7, 2010 at 21:42, Joshua D. Drake j...@commandprompt.com wrote: On Thu, 2010-01-07 at 20:36 +, Dave Page wrote: On Thu, Jan 7, 2010 at 8:07 PM, David E. Wheeler da...@kineticode.com wrote: Hackers, I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][] to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL tools and the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedback and suggestions. Whilst the aim is a noble one, glossing over 'it won't work on Windows' which is by far our most popular platform these days Although I understand your point, this is debatable to the point of a pointless point :P. The reason it seems like the most popular is that the only way to get the windows version is to download it from .Org/EDB. Linux/FreeBSD already has it in their own repositories. If you count production servers, it's certainly going to be more Linux/BSD. But if you start counting developer machines, Windows gets ahead quickly. And then start looking at all the point-of-sales systems etc that come with an embedded PostgreSQL - the vast majority of those (unfortunately) run on Windows. It's not the biggest databases. But it's the most. And particularly something like this needs to work well on the developers machines, or things won't get through the door. I do believe that we need to support windows, but not everything will. That is just the nature of it. True. But if it's something that's infrastructure, it should work on all major platforms. This certainly includes Windows. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
David E. Wheeler wrote: Hackers, I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][] to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL tools and the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedback and suggestions. [plan]: http://wiki.postgresql.org/wiki/PGAN [CPAN philosophy]: http://use.perl.org/article.pl?sid=02/11/12/1616209 [CPAN]: http://cpan.org/ [JSAN]: http://www.openjsan.org [JFDI]: http://acronyms.thefreedictionary.com/JFDI Not enough brackets. ;-) -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
Dave, Whilst the aim is a noble one, glossing over 'it won't work on Windows' which is by far our most popular platform these days seems incredibly short sighted, and liable to lead to an endless stream of 'why doesn't this work' questions. It also does the module authors no favours, by excluding 50%+ of their potential audience, and frankly, isn't the way this project generally works. But why is that *this* project's job to solve? It's already the case that contrib modules (or PostgreSQL) are not buildable on Windows in an automated fashion. And that Windows does not ship with developer tools. That's not PGAN's fault. We have discussed this sort of facility at previous developer meetings, and as I recall came to the conclusion that we need to have the ability to distribute pre-built binaries, not just source code as virtually no Windows users are ever going to have a build environment setup. Similarly, neither are Mac users, if they didn't install XCode. Binary distributions are a good idea, for v2+. The problem with requiring that is then you're effectively requiring every module author, or a well-funded 3rd party, to have a huge build farm capable of building binaries for a wide variety of platforms and PostgresQL versions. Requiring that from the outset is a good way to ensure that nobody *ever* builds an extension management platform. If someone is available to provide that build platform, then it's a different story, but I don't know of anyone with those resources right now. It's *already* the case that Windows users have no access to the temporal, prefix or pgmemcache projects without Visual C++ and some jiggery-pokery. How does PGAN make them worse off? We also discussed extension management at the DBMS level, which I believe Dimitri was working on in his spare time. You should look at what he's been doing. This is designed to be complimentary to Dimitri's project, if you read the wiki page. Finally, don't write the client in Perl. Again, that's of no use to most Windows users. C will work on all platforms from the outset, with no dependencies required. Of course, the server side doesn't matter. It would make sense to build a C version when the other issues with supporting Windows are solved. At that point, the specification for the client should be well-worn enough to make doing the C version not a halt to further development. Until then, it doesn't matter. What I'm getting from your e-mail, Dave, is If it doesn't solve all problems for everyone in the world from Day 1, it's not worth doing. It's my experience that the way to get OSS software that works is to build a little bit at a time, each delivery of which is useful to some people and solves some problems. Projects which attempt to paint the whole world never get anywhere. David's proposal is designed to be something which he can get done *this year*, possibly before 8.5 is released, and be built on later. It'll be useful to a substantial number of our users, and will be an improvement on what we have now. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Jan 7, 2010, at 12:36 PM, Dave Page wrote: Whilst the aim is a noble one, glossing over 'it won't work on Windows' which is by far our most popular platform these days seems incredibly short sighted, and liable to lead to an endless stream of 'why doesn't this work' questions. It also does the module authors no favours, by excluding 50%+ of their potential audience, and frankly, isn't the way this project generally works. That's orthogonal to the development of PGAN. The PGAN client will depend on PGXS to do the installing. If the core implements something that works better on Windows, the PGAN client will use it. We have discussed this sort of facility at previous developer meetings, and as I recall came to the conclusion that we need to have the ability to distribute pre-built binaries, not just source code as virtually no Windows users are ever going to have a build environment setup. Similarly, neither are Mac users, if they didn't install XCode. It's no hardship to install XCode. People do it for Perl/CPAN installs all the time. We also discussed extension management at the DBMS level, which I believe Dimitri was working on in his spare time. You should look at what he's been doing. He and I have discussed it. As I wrote in the proposal, when extensions get in, the PGAN client will be updated as appropriate to support them. Finally, don't write the client in Perl. Again, that's of no use to most Windows users. C will work on all platforms from the outset, with no dependencies required. Of course, the server side doesn't matter. I'll do it in Perl because: 1. It's what I know. 2. There's a lot to draw on already (CPAN.pm, JSAN::Client, CPANPLUS) 3. JFDI. The nice thing is that anyone will be able to write a client if they want it in some other language. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Thu, Jan 7, 2010 at 9:22 PM, Josh Berkus j...@agliodbs.com wrote: Dave, Whilst the aim is a noble one, glossing over 'it won't work on Windows' which is by far our most popular platform these days seems incredibly short sighted, and liable to lead to an endless stream of 'why doesn't this work' questions. It also does the module authors no favours, by excluding 50%+ of their potential audience, and frankly, isn't the way this project generally works. But why is that *this* project's job to solve? Because if we (PostgreSQL) are going to support this effort, then it should not ignore such a huge percentage of our installation base. It's already the case that contrib modules (or PostgreSQL) are not buildable on Windows in an automated fashion. And that Windows does not ship with developer tools. That's not PGAN's fault. No. But that's why we built binaries for people to use. That's pretty much my point. Binary distributions are a good idea, for v2+. The problem with requiring that is then you're effectively requiring every module author, or a well-funded 3rd party, to have a huge build farm capable of building binaries for a wide variety of platforms and PostgresQL versions. Requiring that from the outset is a good way to ensure that nobody *ever* builds an extension management platform. No, I'm suggesting the mechanism needs to support source and binary distribution. For most *nix users, source will be fine. For Windows binaries are required. Building them is no problem - authors can easily use EC2 for which we have an AMI pre-configured for next to no cost, can build on their own platform, on a community provided system, or get a friend to do it. We also discussed extension management at the DBMS level, which I believe Dimitri was working on in his spare time. You should look at what he's been doing. This is designed to be complimentary to Dimitri's project, if you read the wiki page. Finally, don't write the client in Perl. Again, that's of no use to most Windows users. C will work on all platforms from the outset, with no dependencies required. Of course, the server side doesn't matter. It would make sense to build a C version when the other issues with supporting Windows are solved. At that point, the specification for the client should be well-worn enough to make doing the C version not a halt to further development. Until then, it doesn't matter. So you have to rewrite the code. Seems like a waste of effort. What I'm getting from your e-mail, Dave, is If it doesn't solve all problems for everyone in the world from Day 1, it's not worth doing. No. The essence is, 'If you're going to do it in a way that will never work for maybe 50% or more of PostgreSQL installations, then you have fundamental design issues to overcome'. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Jan 7, 2010, at 1:31 PM, Dave Page wrote: No, I'm suggesting the mechanism needs to support source and binary distribution. For most *nix users, source will be fine. For Windows binaries are required. I would love to follow what Strawberry Perl has done to solve this problem. In 2.0. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
Dave Page dp...@pgadmin.org writes:
We have discussed this sort of facility at previous developer
meetings, and as I recall came to the conclusion that we need to have
the ability to distribute pre-built binaries, not just source code as
virtually no Windows users are ever going to have a build environment
setup. Similarly, neither are Mac users, if they didn't install XCode.
And I think that's exactly why David and you are both right. The way I
see it, we need *both* a source level distribution infrastructure, which
David is proposing to implement under the PGAN name, and a binary one
too, which is implemented by distributions like debian, Fedora and the
like, and by Devrim when it comes to PGDG RPMs repos and… by you when we
talk about MacOSX and Windows binary distribution.
I think the aim of the PGAN should be to make it damn easy for binary
distributions to include whatever PGXS extension they whish, with about
no effort.
We also discussed extension management at the DBMS level, which I
believe Dimitri was working on in his spare time. You should look at
what he's been doing.
We've been talking about that together on IRC, and my only reserve here
ain't technical, as we already have a somewhat advanced notion of an
extension. What we miss is support for dump and restore, so my reserve
is that pushing the current system to point-and-clic users when it's not
ready to allow you to just pg_restore a dump when you installed any PGXS
extension (contribs included) is going to have us *lose* users, not gain
any.
Unfortunately I was overly optimistic last May when I though I could
reach a patch submission in the 8.5 cycle. I'd put it on too much work
on the day and no more free time with the required energy when my son is
so young (yes, that's a good problem to have). But still.
What I have been able to do is to determine what features to include in
a first extensions implementation, and proposing a syntax that no one
refused (after less successful attempts). Also the work remaining to be
done would represent easily a month of my evening and week-end time, and
maybe a Tom's morning. But he wouldn't have the fun of learning anything
new, and to paraphrase him, source contributors don't grow on trees :)
Anyway I've tried to summary it all (with some code perspective) in the
following wiki page:
http://wiki.postgresql.org/wiki/ExtensionPackaging
Essentially, it's all about coding a new backend function to execute
commands from a file on the server, in the spirit of \i for psql, say
e.g. pg_execute_commands_from_file('path/ to/file.sql'). It would not
send any result back to the client because you can only describe the
portal once for any given single query.
Then you need to add a catalog for holding the extensions metadata, like
the path of the install file, uninstall file, current version, upgrade
function, custom class if any, path of the setup file (GUCs).
And dependancies between extensions, and between a given extension and
what core provides too (plpgsql e.g., or major postgresql version too).
Now you can hack a CREATE EXTENSION command to fill-in the catalog, and
the commands INSTALL EXTENSION and DROP EXTENSION to execute the given
files. You have to invent a trick to be able to assign any SQL object
created while executing the install file a dependency entry targeting
the extension's catalog oid. As this is all done in a single given
backend, my guess is that a static variable holding the oid would be
enough, but you still have to touch all object creation code path.
You do *not* have to care about schema in any way *BUT* to prepend the
search_path with pg_extension just before executing the install
file. It's a new schema to provide in template databases, so that
extensions default to creating their objects there. This new schema
should be installed just before pg_catalog in the search_path in places
that search for objects from their non qualified name. Full stop on this
topic.
Oh and special bonus if CREATE EXTENSION, given a newer version of an
already installed extension, will register things so as to run the
upgrade function from the newer version at INSTALL EXTENSION time. Or
invent yet another syntax for upgrading.
And. Wait. That's it.
Oh, you could also try to turn plpgsql, plperl, plpython and their
unstrusted variants into built-in extensions too, and contribs too. It
would be nice if contribs where all CREATEd into template0 and template1
and you only had to INSTALL EXTENSION hstore; for enjoying it. That'd be
a Tom's easy evening I guess :)
Anyone having the hours and skill required to do it? It's great fun,
rewarding, everybody and his neighbour *want* it to be done, and I'll be
as available as possible to answer mails, be there on IRC, and do the
necessary political work if there's still some more to be done.
Consider it as a gift I'm giving you: I'm refusing easily earned
reputation by allowing you to code that for me ;)
Regards,
--
dim
PS: that ought to do
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Thu, 2010-01-07 at 13:22 -0800, Josh Berkus wrote: Dave, What I'm getting from your e-mail, Dave, is If it doesn't solve all problems for everyone in the world from Day 1, it's not worth doing. I doubt that is Dave's intent because then we might as well stop work on PostgreSQL too. David's proposal is designed to be something which he can get done *this year*, possibly before 8.5 is released, and be built on later. It'll be useful to a substantial number of our users, and will be an improvement on what we have now. I say if Wheeler wants to do it, he should do it and stop quibbling with the rudiments of the Org opinion. Guess how many people have opinions? Buildfarm succeeded because someone stepped up, evaluated, took a little feedback and ran with it. He didn't screw around waiting for everyone to be happy. So +1 on Wheeler's idea. Joshua D. Drake --Josh Berkus -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
Building them is no problem - authors can easily use EC2 for which we have an AMI pre-configured for next to no cost, can build on their own platform, on a community provided system, or get a friend to do it. So any module author, in order to submit any module, would be required to build binaries for 8-12 platforms covering up to 5 PostgreSQL versions (i.e. 30-70 different binaries)? At their own cost? Seems like a good way to not get any contributions at all. For that matter, Andrew just pointed out to me (corrected me, actually) on IRC that if a Windows user has MSVC or Mingw installed, it should be no problem supporting them. So what you're asking has nothing to do with Windows users, but is a more general we want support for users who don't have a compiler installed. That's a different problem -- and one which the One-click installer or Stackbuilder should probably solve rather than PGAN. No. The essence is, 'If you're going to do it in a way that will never work for maybe 50% or more of PostgreSQL installations, then you have fundamental design issues to overcome'. Again, that's the wrong attitude. You're saying If it doesn't work for 100% of Windows users from Day 1, it won't ever work for them. By that logic, we should have held back version 7.4 until the windows port was done, dammit! And we shouldn't have released until it worked with Visual C++. Even if it meant releasing in 2007. And we shouldn't have released PITR until it supported HS. And we shouldn't be releasing 8.5 without Win64 support. The reason I'm harping on this is this list has a real tendency to reject simple solutions which allow room for growth in favor of overcomplicated world-spanning specifications which will never be implemented. Despite the fact that we only have some notable features because we took the simple approach: partitioning, PITR, the Buildfarm, CommitFests. And that features which there is no obvious way to implement in small steps incrementally (Windows Port, HS, HOT) are huge development problems for us. This list *also* has a real tendency to have an incredible negative attitude, which *you* are currently expressing. The constructive way for you to approach this would have been to say I think that the general idea has merit, but that putting off Windows support is a mistake. What about supporting binary distribution at the outset? And coding the client in C? Instead, you said this doesn't solve problems A, B, and C, so it's stupid. Building a simple solution which doesn't initially cover all bases but can be steadily improved is a far superior strategy to trying to spec The Perfect Solution before even starting work. And if we want to keep recruiting new contributors, criticism needs to be more constructive. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Jan 7, 2010, at 1:59 PM, Joshua D. Drake wrote: So +1 on Wheeler's idea. Thanks! David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On tor, 2010-01-07 at 12:07 -0800, David E. Wheeler wrote: I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][] to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL tools and the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedback and suggestions. Sounds good. You might want to clarify in your prose what an extension is. I suspect I know what you mean, but perhaps not everyone does. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Jan 7, 2010, at 2:11 PM, Peter Eisentraut wrote: You might want to clarify in your prose what an extension is. I suspect I know what you mean, but perhaps not everyone does. Good suggestion, thanks. How about this in the FAQ? * WTF is an extension? An extension is a piece of software that adds functionality to PostgreSQL itself. Examples are data types (CITEXT, PERIOD), utilities (newsysviews, pgTAP), and procedural languages (PL/Ruby, PL/R), among others. An extension is *not* a piece of software designed to run on top of PostgreSQL (Bricolage, Drupal). Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
David E. Wheeler da...@kineticode.com writes: On Jan 7, 2010, at 2:11 PM, Peter Eisentraut wrote: You might want to clarify in your prose what an extension is. I suspect I know what you mean, but perhaps not everyone does. Good suggestion, thanks. How about this in the FAQ? * WTF is an extension? An extension is a piece of software that adds functionality to PostgreSQL itself. Examples are data types (CITEXT, PERIOD), utilities (newsysviews, pgTAP), and procedural languages (PL/Ruby, PL/R), among others. An extension is *not* a piece of software designed to run on top of PostgreSQL (Bricolage, Drupal). Maybe with a link to: http://www.postgresql.org/docs/8.4/static/extend.html In the sections that follow, we will discuss how you can extend the PostgreSQL SQL query language by adding: - functions (starting in Section 34.3) - aggregates (starting in Section 34.10) - data types (starting in Section 34.11) - operators (starting in Section 34.12) - operator classes for indexes (starting in Section 34.14) The PostgreSQL server can moreover incorporate user-written code into itself through dynamic loading. That is, the user can specify an object code file (e.g., a shared library) that implements a new type or function, and PostgreSQL will load it as required. Code written in SQL is even more trivial to add to the server. This ability to modify its operation on the fly makes PostgreSQL uniquely suited for rapid prototyping of new applications and storage structures. Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Jan 7, 2010, at 2:23 PM, Dimitri Fontaine wrote: Maybe with a link to: http://www.postgresql.org/docs/8.4/static/extend.html Good call, thanks. David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
Dimitri Fontaine dfonta...@hi-media.com writes:
e.g. pg_execute_commands_from_file('path/ to/file.sql'). It would not
[...]
Then you need to add a catalog for holding the extensions metadata, like
[...]
Now you can hack a CREATE EXTENSION command to fill-in the catalog, and
the commands INSTALL EXTENSION and DROP EXTENSION to execute the given
files. You have to invent a trick to be able to assign any SQL object
created while executing the install file a dependency entry targeting
the extension's catalog oid. As this is all done in a single given
backend, my guess is that a static variable holding the oid would be
enough, but you still have to touch all object creation code path.
Now that you have dependency information tracked, don't forget to add
pg_dump support by skipping any objects installed as part of an
extension and only issuing INSTALL EXTENSION foo; instead.
Then let pg_restore run this and complain loudly when the needed
extension entry does not exists yet, you have to CREATE and INSTALL the
extension on the new database before getting to restore your dump.
You do *not* have to care about schema in any way *BUT* to prepend the
search_path with pg_extension just before executing the install
file. It's a new schema to provide in template databases, so that
extensions default to creating their objects there. This new schema
should be installed just before pg_catalog in the search_path in places
that search for objects from their non qualified name. Full stop on this
topic.
Oh and special bonus if CREATE EXTENSION, given a newer version of an
already installed extension, will register things so as to run the
upgrade function from the newer version at INSTALL EXTENSION time. Or
invent yet another syntax for upgrading.
And. Wait. That's it.
Oh, you could also try to turn plpgsql, plperl, plpython and their
unstrusted variants into built-in extensions too, and contribs too. It
would be nice if contribs where all CREATEd into template0 and template1
and you only had to INSTALL EXTENSION hstore; for enjoying it. That'd be
a Tom's easy evening I guess :)
Anyone having the hours and skill required to do it? It's great fun,
rewarding, everybody and his neighbour *want* it to be done, and I'll be
as available as possible to answer mails, be there on IRC, and do the
necessary political work if there's still some more to be done.
Consider it as a gift I'm giving you: I'm refusing easily earned
reputation by allowing you to code that for me ;)
Regards,
--
dim
PS: that ought to do it.
Hey, there's about a week left. :)
--
dim
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
On Thu, Jan 07, 2010 at 12:07:19PM -0800, David Wheeler wrote: Hackers, I've posted a [plan] to implement PGAN][], a CPAN for PostgreSQL extensions. I've tried to closely follow the [CPAN philosophy][] to come up with a plan that requires a minimum-work implementation that builds on the existing PostgreSQL tools and the examples of the [CPAN][] and [JSAN][]. My hope is that it's full of [JFDI][]! I would be very grateful for feedback and suggestions. +1 for a modest PGAN to start. :) Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RFC: PostgreSQL Add-On Network
* Josh Berkus (j...@agliodbs.com) wrote: What I'm getting from your e-mail, Dave, is If it doesn't solve all problems for everyone in the world from Day 1, it's not worth doing. It's my experience that the way to get OSS software that works is to build a little bit at a time, each delivery of which is useful to some people and solves some problems. Projects which attempt to paint the whole world never get anywhere. +1 from me. I was just discussing this with some other folks at my company and had to make the point that if all you do is show the CEO a big fat $ figure, he's gonna say no. If you actually want to make it happen, you have to show a progression to get there from where we are today, with the incremental costs required along the way and the functionality that you get for that. David's proposal is designed to be something which he can get done *this year*, possibly before 8.5 is released, and be built on later. It'll be useful to a substantial number of our users, and will be an improvement on what we have now. Sounds excellent to me. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] RFC: PostgreSQL Add-On Network
* Dave Page (dp...@pgadmin.org) wrote: Because if we (PostgreSQL) are going to support this effort, then it should not ignore such a huge percentage of our installation base. Not doing it day 1 is not ignoring. It's using what resources *are* being made available to the best extent we can. If you're offering to do the work for Windows, great! It would make sense to build a C version when the other issues with supporting Windows are solved. At that point, the specification for the client should be well-worn enough to make doing the C version not a halt to further development. Until then, it doesn't matter. So you have to rewrite the code. Seems like a waste of effort. So the options are some Perl code that works for quite a few users or.. nothing because he's not a C hacker or doesn't want to write it in C? Sounds like #1 is a win to me. If David's happy to write it in C to begin with (presuming he has to write anything- if there's existing Perl code that does 90% of what he needs, you're asking for alot more), great. I'm even happy to encourage him to do that if it's anywhere close to the same level of effort. No. The essence is, 'If you're going to do it in a way that will never work for maybe 50% or more of PostgreSQL installations, then you have fundamental design issues to overcome'. And my vote is that you have to start somewhere and I strongly disagree that what you're concerned with are serious *design* issues. What David has described includes alot of implementation details, let's not confuse the two. If the server-side had to be scrapped entirely and rewritten to support Windows, you might have a leg to stand on. If adding Windows support is an incremental change to the existing system (as a whole, which, yes, I'd consider the port of a perl client app to C to be an incremental change), then it's not a design issue. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] RFC: PostgreSQL Add-On Network
David E. Wheeler wrote: On Jan 7, 2010, at 1:31 PM, Dave Page wrote: No, I'm suggesting the mechanism needs to support source and binary distribution. For most *nix users, source will be fine. For Windows binaries are required. I would love to follow what Strawberry Perl has done to solve this problem. In 2.0. +1. They did a nice job. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
