[HACKERS] Postgres and 'directio' on Solaris/UFS
So I'm reading Sun's _Configuring Tuning Databases on the Solaris
Platform_ book (ISBN: 0-13-083417-3). One of the first things mentioned
in the book is that UFS is, well, slow. Especially because of how
Solaris utilizes it. The authors contend that enabling directio(3C) can
increase performance substantially. It can lead to problems with
filesystems where strange things are done with files. Since databases
tend to manage their files very well to begin with, the authors say
this isn't normally a problem.
I searched the web for it, but all I found was some references to Zend.
Additionally, I grepped through source only to realize that direction
is used a lot. So I used the following find :
[goro:~/postgresql-7.3.1] alex% find . -type f -exec egrep -il
'directio[^Nn]' {} \;
And also didn't find anything. The manpage for directio (Solaris 9) is
here:
http://docs.sun.com/db/doc/816-0213/6m6ne37so?a=view
I haven't gotten my ultrasparc database server up yet, so I can't run
any benchmarks against this. Is it possible somebody with some spare
time could mount a filesystem with directio forced (forcedirectio) and
run some benchmarks on a Solaris/UFS database?
Thanks
alex
--
alex avriette, unix geek for hire
http://envy.posixnap.net/
[EMAIL PROTECTED]
---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Irix 6.2, Postgres 7.3.1, some brokenness
Disregard previous. Using /bin/ld (with LDREL = -r) works fine as a linker. Call it force of habit. Is it worth warning the user that you cannot use gcc as ld on Irix? I used it because I figured I would need gnu ld (which I of course didn't have). Anyhow, 7.3.1 is successfully built. Alex -- alex avriette $^X is my programming language of choice. [EMAIL PROTECTED] ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
[HACKERS] Irix 6.2, Postgres 7.3.1, some brokenness
I've got a 6-way SGI Challenge L with R4400/200 cpu's in it, and a gig of interleaved ram. I am running postgres 7.3.1, which I compiled 64-bit with MIPSPro. The database is living on an ultrawide scsi raid. I am also running postgres on a 2-cpu (85mhz) Sparc 20 with 384mb of ram, and the database lives on a 10krpm single narrow disk. I'm finding that the Sparc 20 is about twice as fast as the Challenge. Postgres on the '20 was compiled 32-bit (of course) with gcc 3.2. When I am testing the machines, I have run tests with 1 db handle, with 2 db handles, and with 2 vs 6 handles. Consistently, the 20 is faster than the Challenge. I suspect something is going wrong here on mips. I have tried, without any luck, to get postgres compiled with gcc 3.2 (from freeware.sgi.com). It keeps adding '-r' when its linking, even when LDFLAGS is unset, LD is 'gcc' and CC is 'gcc'. I could do the linking manually, but this is not really a viable option for a release. Additionally, postgres does *not* trap divide-by-zero errors, as Solaris and Linux do. 'select 1/0' causes the backend to crash. elvis=# select 1/0; server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. The connection to the server was lost. Attempting reset: WARNING: Message from PostgreSQL backend: The Postmaster has informed me that some other backend died abnormally and possibly corrupted shared memory. I have rolled back the current transaction and am going to terminate your database system connection and exit. Please reconnect to the database system and repeat your query. Failed. !# Justin tells me there is a release imminent, and that I should mention this ASAP. Neil suggested I try with gcc, but as I said, I've been unable to get it to compile with gcc. Also, I understand (but can confirm on monday) that Oracle has dropped support for Irix entirely. This would make postgres the *only* database for SGI MIPS, and I'd really like to see it better supported. SGI machines are awesome for database servers. Anyone needing a shell on the machine can feel free to ask. Alex -- alex avriette $^X is my programming language of choice. [EMAIL PROTECTED] ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Irix 6.2, Postgres 7.3.1, some brokenness
I have tried, without any luck, to get postgres compiled with gcc 3.2 (from freeware.sgi.com). It keeps adding '-r' when its linking, even when LDFLAGS is unset, LD is 'gcc' and CC is 'gcc'. I could do the linking manually, but this is not really a viable option for a release. Looking at this more, I see this: [goro:~/postgresql-7.3.1/src] alex% grep LD Makefile* | grep -- '-r' Makefile.global:LDREL = -r Can we just not append LDREL to LD if we're on Irix? alex -- alex avriette $^X is my programming language of choice. [EMAIL PROTECTED] ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Irix 6.2, Postgres 7.3.1, some brokenness
When compiled with gcc, it does trap divide by zero. Alex ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
[HACKERS] problem building pg 7.3 beta 3 on solaris 8 -m64
Dave Miller ([EMAIL PROTECTED]) tells me this bug is fixed in gcc 3.2.1 (which does not appear to be distributed yet). FYI. alex ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] problem building pg 7.3 beta 3 on solaris 8 -m64
make[2]: Entering directory `/data/aja96/postgresql-7.3b3/src/backend/utils/mb/conversion_procs' make[3]: Entering directory `/data/aja96/postgresql-7.3b3/src/backend/utils/mb/conversion_procs/ ascii_and_mic' gcc -shared -h libascii_and_mic.so.0 ascii_and_mic.o -L../../../../../../src/port -o libascii_and_mic.so.0.0 ld: fatal: library -lgcc_s: not found ld: fatal: library -lgcc_s: not found ld: fatal: File processing errors. No output written to libascii_and_mic.so.0.0 collect2: ld returned 1 exit status make[3]: *** [libascii_and_mic.so.0.0] Error 1 make[3]: Leaving directory `/data/aja96/postgresql-7.3b3/src/backend/utils/mb/conversion_procs/ ascii_and_mic' make[2]: *** [all] Error 2 make[2]: Leaving directory `/data/aja96/postgresql-7.3b3/src/backend/utils/mb/conversion_procs' make[1]: *** [all] Error 2 make[1]: Leaving directory `/data/aja96/postgresql-7.3b3/src' make: *** [all] Error 2 Darcy explained that it might be a problem with gcc 3.1, and that i should upgrade to 3.2. The compile line from the makefile is actually: gcc -shared -h libascii_and_mic.so.0 ascii_and_mic.o -L../../../../../../src/port -o libascii_and_mic.so.0.0 While attempting to compile gcc 3.2, I found a bug in the build process (which i have corrected, bug report to gcc folks on its way). I'll report back after that. I think however, this is a gcc bug (that will because of the bug I just found) be present in 3.2. The problem is that when gcc is built, the directory $build_dir/gcc/sparcv9 does not include the following two symlinks: lrwxrwxrwx 1 aja96atd 13 Nov 6 16:23 libgcc_s.so - libgcc_s.so.1 lrwxrwxrwx 1 aja96atd 13 Nov 6 16:25 libgcc_s_sparcv9.so - libgcc_s.so.1 (note: -rwxr-xr-x 1 aja96atd 512136 Nov 6 14:12 libgcc_s.so.1) creating those symlinks (and adding the dir to LD_LIBRARY_PATH, but in retrospect i think it was unnecessary) allows it to continue compiling (gcc). I havent figured out where those are fubar on the system yet, I'm sure somebody here does; I'm going to get gcc 3.2 built and see if this fix fixes the pg compile. Some sort of logic understanding this buggery should probably incorporated into the release. i've included my home address on CC, please maintain that CC. my contract here is up in a couple weeks, and i will have limited access to the address currently subscribed to hackers. thanks alex ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] IRIX and large SMP: donations of shells c
I'm going to be overseeing a move from a Mac-based postgres database (100k transactions/day, roughly 5M rows) to an SGI Octane in the near-ish term.The machine will only be two-way SMP. I'd like to see it working 64-bit and compiled with MIPSpro. I have a friend who has mostly succeeded in getting it compiiled with MIPSpro, but Neil told me today there might be concerns with SMP systems 4cpu's. I offered access on a system with 6 cpus (SGI Challenge L, R4400's). I may have access to other machines, including a 36-cpu Octane with R10k's. If this is useful to somebody on the core group, please let me know. I'd really like to see Postgres understand MIPSpro and irix out of the box. I understand there is some difficulty at present. I'd appreciate a Cc on the thread, if possible. Thanks, alex ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] Darwin 1.4 (OS X 10.1) Broken Compile, Snapshot and 7.1.3
I had a catastrophic crash on one of our webservers here and took the
opportunity to upgrade it. Unfortunately, after the upgrade, I am unable to
compile either 7.1.3 or the current snapshot of postgres.
The error I get is rather opaque:
/usr/bin/ld: -undefined error must be used when -twolevel_namespace is in
effect
This is in the final stretch of compiling, on libpq. I checked google for
it, and came up with three header files in an MIT Apple sourcetree. So this
strikes me as a particularly Darwinish failing. I also ran a recursive grep
on the postgres source tree and wasnt able to find -twolevel_namespace in
any of the Makefiles. This makes me think it is something external.
Anyone have an idea as to what is causing this? This box is down until
postgres comes back up. :-7
alex
--
alex j. avriette
perl hacker.
[EMAIL PROTECTED]
$dbh - do('unhose');
---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster
drop-in-ability (was: RE: [HACKERS] Re: [PATCHES] Select parser at runtime )
I sent the list a message a little while ago about what I do with postgres. I thought, after all this discussion, that it might be important to send a further message to the list indicating why I chose not to use Oracle. I'm going to go out on a limb and say that I have one of the largest postgres databases in the world. At our site, we have quite a few oracle instances and something like 27 schemas (and, I'm afraid to report, a few Access as well). Our PG database dwarfs all the other databases combined. I chose postgres not because of any cost issue at all. The project that I am working on here is accustomed to spending $25,000 in a given week for new hardware, we bring on consultants as we need them et cetera. To me, what made postgres attractive was how simple it was to install and configure. Furthermore, I have had discussions with our oracle DBA's. When I mentioned I thought our number of rows could reach into the tens of millions, and the eventual storage estimate is something on the order of several terabytes, he told me I would need to hire on 2-3 fulltime DBA's if I wanted to use Oracle. My feeling on whether postgres is a drop-in replacement for Oracle is similarly simple. Any sufficiently competent programmer can make database-independant code. Up to this point, I have been working hard to make sure that, should higher management decide to use Oracle, I can use pg_dump and actually just move on over to Oracle. I would have chosen a similar approach if I had started with Oracle. Perhaps it is my background as a Perl programmer (it is the DataBase Independant driver, afterall), or perhaps it is my fear of change. But if you guys are concerned that one database is not a drop-in replacement for another, you are paying your programmers too much. I think perhaps it is the Oracle Mentality (or dare I say, Microsoft Mentality) to not worry about portability and compatibility that breeds this kind of programming and production. It is very much against, however, the opensource mentality. Dont beat yourself up, guys, over making postgres a drop-in replacement for Oracle. The people that would benefit from actually dropping in postgres into an Oracle install will have already eased the burden on themselves by being responsible in their database construction and programming. I haven't even been able to convince our Oracle guys that Postgres is actually a real database (its free?!! how can it be free?!). They would never, (ever!) consider dropping in postgres. If you're intent on taking customers from Oracle, catch them where youll be able to convert them -- before Oracle is even installed. alex ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
[HACKERS] What I do with PostgreSQL
This might not be the correct list to send this to, but none of the other lists seemed appropriate. A friend of mine who uses postgres extensively at his job suggested I might send y'all a note outlining what we do with it here. In general, I am discouraged from providing specific data to non-employees about what we do. But Dan (the aforementioned friend) said that you guys would be interested in knowing what I am currently doing with postgres, so that you know that its up to the challenges we don¹t often get to put hardware and software to. I am working in the publications division of the American Chemical Society. We are in the process of taking all of our 30+ journals from the last 150 or so years and digitizing them. This process entails scanning over 2.5 million pages (though this is really only a rough estimate. It could be much higher) and digitizing them. Our output is in several formats. First, we have the input TIFF (from the scans), we have PDF's which we render using Adobe Capture, XML (which we pay a vendor for), and a proprietary format called DjVu which is kind of Well, its like metadata. Initially, we were using perl scripts and shell scripts to traverse the entire filesystem looking for files. This got rather difficult and was time consuming. My suggestion was to just use a database for keeping track of stuff. We have something like 27 different instances of oracle running here on 4 or 5 different machines. I don't know much about our oracle stuff. My solution was to just go download and install postgres. Our hardware is a cluster of 3 ultra 10's, a pair of 700-dvd jukeboxes (with burners), a 2.5tb SAN, 10 DAT tape readers, a pair of dvd-roms, and 2 200gb disk packs (one for each of our tape-reading suns -- the other one manages the DVD jukes). We also run capture on four dell poweredge servers running NT. We run the DjVu software on an additional 3 poweredge servers. That stuff is NT. The SAN is run on a cluster of 4 sun e 3500's. I am pumping about 200gb a week through the pg database, and our estimated database size is something like 4tb by the end of the year. We populate the database with perl scripts. The sun that runs the dvd jukes is also our database server. We have shell scripts that look over our data on the disk, and we use sun's NFS to keep disks between the suns and some funky Sun smb-esque software to keep disks mounted on the nt boxes. And that's just the large database. I have an additional database that I am using to store the textual data we receive in the form of crystallography information files (http://www.iucr.org/) which are roughly 6,000 lines long. I have 10,000 of them stored at the moment in the database, going back to about 1996. As you can tell, this database is going to get much bigger. At the moment it's living on an Ultra 2 in a 2gb partition. In some ways, I am amazed that postgres has stood up to the challenge. In others, however, I am not in the least surprised. Its a fantastic piece of software that requires almost no intervention on my part. I talked to one of our oracle dba's about it. He actually (im not kidding here) did not believe it could be a database if it did not require maintenance. I am very happy with postgres and I am glad to provide information about our setup if you'd like to know anything else. If you'd like to quote me on the environment if youre interested in putting something in a FAQ (i.e., can postgres scale up to tb scale?), that¹s fine as well, but I would like to make sure that it doesn¹t point to ACS and is not too specific. Anyhow, thanks for your hard work guys/gals. alex ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
