[HACKERS] Relation hash table
I am trying to divide the buffer pool into 5 parts and i call these clusters (different from postgres db clusters). What i want to do is to make a mapping from the relations to these clusters. One relation belongs to one cluster and a cluster can obviously have more than one relations. I use a hash table to do this similar to the buf_table.c file. Right now i don't know how many relations there are in total. I have just initialized it to 100. The key is relation and the data is the cluster information. Can anyone see any problems in using this approach. Feed back will be greatly appreciated. PS i am aware that because of OS caching this may not cause any improvements. thanks nailah ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] [GENERAL] dbmirror revisions
Are any of these changes ready for CVS for 7.4? --- Ed L. wrote: I've been modifying dbmirror and wanted to offer my changes to anyone that cared to experiment, FWIW. My effort is ongoing, the docs aren't perfect, I make no claims of production readiness, and testing of this latest version has been minimal, so I strongly advise you to conduct your own thorough testing before considering a production deployment. That said, it's a significantly improved solution for our async master-slave needs, with a few caveats below, and shouldn't be too hard to setup. There are enough changes that I would hardly consider this a patch, closer to an overhaul, since I've removed files, renamed others, and added new files. Among the changes I've made so far: * Added script for easier setup of many tables/dbs/slaves; * Added initial support for multiple master replicating distinct data to a single slave; * Added batching to minimize load on master and net traffic. You can grab a configurable number of updates to replicate before hitting the master again. * Added port specification; * Wrapped all replication in transactions; * Bulletproofed against downed master or slave; * Started modularization of DB access layer, added some error handling; * Added a number of config vars for sync delays, etc; * Eliminated bug in transaction ordering for replay. Updates cannot be replicated in the order of the transactions (see archives for discussion of why). * Eliminated need for clear_pending.pl by making dbmirror.pl self-clearing; * Collasped schema into 1 queue table for performance; * Changed sequence ID column types to BIGINT for 64-bit sequence; * Added reconnection handling for robustness; * Added local tracking of last seq_id to help with recovery robustness; * Added master/slave compatibility checking; * Enabled slave setup during production service so master does not have to stop serving. * Renamed tables to minimize namespace conflicts; * Added lots of logging/debug messages; * Maybe a few other things I've forgotten... AFAICS, there are still at least a few major drawbacks to this approach: * DML statements are not replicated (same for eRServer, AFAIK). * SEQUENCE objects are not handled; nextval() will not be replicated, so sequence objects (and serial columns) between master and slave can easily get out of sync. I wonder if eRServer has this same issue? * Mass updates/deletes/inserts of 5000 rows with a single SQL command on the master will result in 5000 individual trigger-firings, and 5000 individual replication inserts on the slave. Rumor has it eRServer's snapshot gets around this problem. The code is here: http://bluepolka.net/dbmirror/dbmirror-20030403-1605.tar.gz Ed ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] RBLs ... I'm tired of spam ...
*Way* off topic ... but I'm tired of processing through 300 messages nightly of which 10 are stuff that need to be approved for the lists, and 290 are trash ... What are ppl using / trusting out there as far as Free RBLs are concerned? ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] RBLs ... I'm tired of spam ...
FEATURE(dnsbl,`korea.services.net',``Mail from ${client_addr} rejected by
korea.services.net'')dnl
FEATURE(dnsbl,`brazil.blackholes.us',``Mail from ${client_addr} rejected
by brazil.blackholes.us'')dnl
FEATURE(dnsbl,`opm.blitzed.org',``Mail from ${client_addr} rejected by
opm.blitzed.org'')dnl
plus I have a contract for rbl-plus.mail-abuse.org
opm.blitzed.org is open proxies, and the others are obvious.
LER
--On Tuesday, May 27, 2003 17:41:37 -0300 Marc G. Fournier
[EMAIL PROTECTED] wrote:
*Way* off topic ... but I'm tired of processing through 300 messages
nightly of which 10 are stuff that need to be approved for the lists, and
290 are trash ...
What are ppl using / trusting out there as far as Free RBLs are concerned?
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED]
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [HACKERS] RBLs ... I'm tired of spam ...
On Tue, 27 May 2003, Marc G. Fournier wrote: *Way* off topic ... but I'm tired of processing through 300 messages nightly of which 10 are stuff that need to be approved for the lists, and 290 are trash ... What are ppl using / trusting out there as far as Free RBLs are concerned? Avoid SPEWS. Vince. -- Fast, inexpensive internet service 56k and beyond! http://www.pop4.net/ http://www.meanstreamradio.com http://www.unknown-artists.com Internet radio: It's not file sharing, it's just radio. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] RBLs ... I'm tired of spam ...
Install SpamAssassin, and let it figure it out for you. It uses a whole list of RBLs and uses them to score a message as spam, instead of just blanket-denying messages from those SMTP servers. It works quite well. On Tuesday 27 May 2003 01:41 pm, Marc G. Fournier wrote: *Way* off topic ... but I'm tired of processing through 300 messages nightly of which 10 are stuff that need to be approved for the lists, and 290 are trash ... What are ppl using / trusting out there as far as Free RBLs are concerned? ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly -- Michael A Nachbaur [EMAIL PROTECTED] ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] RBLs ... I'm tired of spam ...
Larry Rosenman wrote:
FEATURE(dnsbl,`korea.services.net',``Mail from ${client_addr}
rejected by korea.services.net'')dnl
FEATURE(dnsbl,`brazil.blackholes.us',``Mail from ${client_addr}
rejected by brazil.blackholes.us'')dnl
Yes, SPAM is really a problem here. Unfortunatelly, telco companies
won't do anything to keep spammers out of service. It's very radical to
simply drop all the brazilian IPs, and I just cannot do this otherwise
I'd be unable to talk to anyone. I've added about 20 B classes to the
mta's access list and 99% of the spam went away. This is because ALL the
spam that comes from Brazil are originated from ADSLs/Cablemodens.
Yet, I used to receive many spams from US/China until I started
using spews.
I think that people with .org/.net/.com domains are really in a bad
situation. People say that when they receive 300 mails a day, 250 or so
are spam. This sounds really bad. In the worst times, I used to receive
at most 5-10 spams a day, from 400 other e-mails.
FEATURE(dnsbl,`opm.blitzed.org',``Mail from ${client_addr} rejected
by opm.blitzed.org'')dnl
plus I have a contract for rbl-plus.mail-abuse.org
opm.blitzed.org is open proxies, and the others are obvious.
I found mail-abuse.org to be very burocratic to add a single IP to
it's database. I think this makes them much less effective.
[]s
Ricardo.
---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] RBLs ... I'm tired of spam ...
Another vote for SpamAssassin. We use it at work here and it's quite nice. It puts all the borderline spam in a holding area and sends you a daily email with all the topics / names listed and you can request those out of the spam bucket. It's configurable to the extreme. On Tue, 27 May 2003, Michael A Nachbaur wrote: Install SpamAssassin, and let it figure it out for you. It uses a whole list of RBLs and uses them to score a message as spam, instead of just blanket-denying messages from those SMTP servers. It works quite well. On Tuesday 27 May 2003 01:41 pm, Marc G. Fournier wrote: *Way* off topic ... but I'm tired of processing through 300 messages nightly of which 10 are stuff that need to be approved for the lists, and 290 are trash ... What are ppl using / trusting out there as far as Free RBLs are concerned? ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] RBLs ... I'm tired of spam ...
On Tue, 27 May 2003, Vince Vielhaber wrote: On Tue, 27 May 2003, Marc G. Fournier wrote: *Way* off topic ... but I'm tired of processing through 300 messages nightly of which 10 are stuff that need to be approved for the lists, and 290 are trash ... What are ppl using / trusting out there as far as Free RBLs are concerned? Avoid SPEWS. Ya, Spews is one of the 'evil ones' ... one of the blocks we were on in panama was in spews, and they don't seem to provide any way (that we could find) of removing the IPs when the offending server(s) were taken off the network ... ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] RBLs ... I'm tired of spam ...
On Tue, 27 May 2003, scott.marlowe wrote: Another vote for SpamAssassin. We use it at work here and it's quite nice. It puts all the borderline spam in a holding area and sends you a daily email with all the topics / names listed and you can request those out of the spam bucket. It's configurable to the extreme. 'K, I haven't found *that* feature yet ... can you do this on a per-user basis as well, or is this a 'blanket, site wide' configuration ... ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] [PATCHES] Sequence usage patch
(Moved to -hackers) Rod Taylor [EMAIL PROTECTED] writes: Are you ok with the DB2 and draft-spec syntax of NEXT VALUE FOR (where value is not a reserved word)? Or should I hold onto that until the spec has gone through the final draft / release? By that time we'll have done the Oracle-style foo.nextval, and it'll become kind of a moot point ;-) I actually like the NEXT VALUE FOR a lot more. The reason is that the Oracle syntax is very much an 'object.property' lookup, which we do nowhere else in PostgreSQL. In fact, it's actually a bit bizarre when you start going database.schema.sequence.nextval, etc. The NEXT VALUE FOR syntax would be more in keeping with our current sytacies methinks... Chris ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] SIGSEGV on cvs tip/7.3.2
There's been some past speculation about putting in a function call nesting depth limit, but I haven't been able to think of any reasonable way to estimate a safe limit. The stack size limit varies a lot across different platforms, and the amount of stack space consumed per PL function call level seems hard to estimate too. We do have a nesting depth limit for expressions, which is intended specifically to avoid stack overflow during expression eval --- but the amount of stack chewed per expression level is relatively small and predictable. GUC variable? Hmm...but that would mean that a normal user could still just crash the machine...? Chris ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] RBLs ... I'm tired of spam ...
Per user or site I use Spamassassin AFTER the 4 RBL's and a personal fecal Roster. --On Tuesday, May 27, 2003 21:31:45 -0300 The Hermit Hacker [EMAIL PROTECTED] wrote: On Tue, 27 May 2003, scott.marlowe wrote: Another vote for SpamAssassin. We use it at work here and it's quite nice. It puts all the borderline spam in a holding area and sends you a daily email with all the topics / names listed and you can request those out of the spam bucket. It's configurable to the extreme. 'K, I haven't found *that* feature yet ... can you do this on a per-user basis as well, or is this a 'blanket, site wide' configuration ... ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED] -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED] US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Expect problems with PL/Python and Python version 2.2.3+ 2.3+
Tilo Schwarz [EMAIL PROTECTED] writes: Tom Lane writes: Looks like we either change plpython to untrusted status or remove it entirely :-(. Sean, do you have time to prepare a patch for the former? Please, don't remove it. We (a group of trusted people using Postgresql) are actually waiting for plpython to become untrusted, so we can use the full power of python (e.g. file access) from Postgresql. Well, Sean disclaimed the project, so you seem to be next in line ;-) Go to it ... regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] vacuum analyze corrupts database
Hmm. Are you running the database in a non-C locale? (pg_controldata would tell you.) --- Here's the output of pg_controldata: pg_control version number:72 Catalog version number: 200211021 Database cluster state: in production pg_control last modified: Sun May 25 18:38:06 2003 Current log file ID: 0 Next log file segment:2 Latest checkpoint location: 0/15EF7A8 Prior checkpoint location:0/15ED2D8 Latest checkpoint's REDO location:0/15EF7A8 Latest checkpoint's UNDO location:0/0 Latest checkpoint's StartUpID:47 Latest checkpoint's NextXID: 3563 Latest checkpoint's NextOID: 118086 Time of latest checkpoint:Sun May 25 18:38:02 2003 Database block size: 8192 Blocks per segment of large relation: 131072 Maximum length of identifiers:64 Maximum number of function arguments: 32 Date/time type storage: Floating point Maximum length of locale name:128 LC_COLLATE: en_US.ISO8859-1 LC_CTYPE: en_US.ISO8859-1 - locale settings on the host: tomkins% locale LANG= LC_CTYPE=C LC_NUMERIC=C LC_TIME=C LC_COLLATE=C LC_MONETARY=C LC_MESSAGES=C LC_ALL= - I am wondering if strxfrm() on your platform sometimes writes more bytes than it is supposed to. I have seen vsnprintf() overrun its output buffer on some flavors of Solaris (according to FAQ_Solaris, the 64-bit libc in Solaris 7 had such a problem). Could there be a similar bug in their strxfrm? We're on Solaris 8. I'll try to find information on strxfrm bugs, but do you rule out any problems in Postgres code? Is there a good explanation to why the same table loaded into another Postgres installation on the same host can be analyzed without any problems? Also in my database I can drop/create database, load table and reproduce the error. Not sure what to make out of this. Mike. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Tom Lane Sent: Sunday, May 25, 2003 9:03 PM To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: [HACKERS] vacuum analyze corrupts database Michael Brusser [EMAIL PROTECTED] writes: (dbx) where 30 =[1] pfree(0x489420, 0xffbee890, 0x489420, 0xffbee880, 0x489628, 0xffbee888), at 0x2535e4 [2] convert_to_scalar(0x489078, 0x19, 0xffbee890, 0x489008, 0x488fc0, 0x413), at 0x1fc6b4 [3] scalarineqsel(0x484608, 0x42a, 0x0, 0x488a88, 0x489078, 0x19), at 0x1f94e4 Hmm. Are you running the database in a non-C locale? (pg_controldata would tell you.) If so, this pfree is trying to pfree one of three strings that were filled with strxfrm(). I am wondering if strxfrm() on your platform sometimes writes more bytes than it is supposed to. I have seen vsnprintf() overrun its output buffer on some flavors of Solaris (according to FAQ_Solaris, the 64-bit libc in Solaris 7 had such a problem). Could there be a similar bug in their strxfrm? regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] vacuum analyze corrupts database
Michael Brusser [EMAIL PROTECTED] writes: but do you rule out any problems in Postgres code? Never ... but when you're the only one reporting a problem, a local issue is something to consider. Is there a good explanation to why the same table loaded into another Postgres installation on the same host can be analyzed without any problems? Well, first thing I'd ask is whether the other installation is using the same locale settings. regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] RBLs ... I'm tired of spam ...
On 2003-05-27 21:31:45, The Hermit Hacker wrote: 'K, I haven't found *that* feature yet ... can you do this on a per-user basis as well, or is this a 'blanket, site wide' configuration ... # When using the 'local:' quarantine method (default), the following # applies: # # A finer control of quarantining is available through variable # $virus_quarantine_to/$spam_quarantine_to. It may be a simple scalar # string, # or a ref to a hash lookup table, or a regexp lookup table object, # which makes possible to set up per-recipient quarantine addresses. I am very impressed with the amavisd-new, spamassassin, razor combination that I installed on my home machine a few days. Not had to manually process any spam since, and mailq provides a nice reference on what could be outright blocked. http://lawmonkey.org/anti-spam.html is a well written guide to get started, although most of the details were taken care of by the debian package used. /Allan -- Allan Wind P.O. Box 2022 Woburn, MA 01888-0022 USA pgp0.pgp Description: PGP signature
