[HACKERS] memory bug debugging

2005-10-04 Thread Markus Schiltknecht
Hello hackers, I'm fiddling around with the backend and have created a memory bug. I guess I'm overriding a palloced chunk, but can't figure out where. The backend's report is: WARNING: problem in alloc set MessageContext: req size alloc size for chunk 0x8383ca8 in block 0x83834a0

Re: [HACKERS] Replication on the backend

2005-12-06 Thread Markus Schiltknecht
On Tue, 2005-12-06 at 10:03 -0200, Gustavo Tonini wrote: But, wouldn't the performance be better? And wouldn't asynchronous messages be better processed? At least for synchronous multi-master replication, the performance bottelneck is going to be the interconnect between the nodes -

Re: [HACKERS] Replication on the backend

2005-12-06 Thread Markus Schiltknecht
Hello Jan, On Tue, 2005-12-06 at 10:10 -0500, Jan Wieck wrote: We need a general API. It should be possible to define on a per-database level which shared replication module to load on connect. The init function of that replication module then installs all the required callbacks at

Re: [HACKERS] Replication on the backend

2005-12-06 Thread Markus Schiltknecht
On Tue, 2005-12-06 at 23:19 -0500, Jan Wieck wrote: It's not so much the bandwidth but more the roundtrips that limit your maximum transaction throughput. I completely agree that the latency is counting, not the bandwith. Does anybody have latency / roundtrip measurements for current

Re: [HACKERS] Replication on the backend

2005-12-07 Thread Markus Schiltknecht
On Wed, 2005-12-07 at 01:04 -0800, J. Andrew Rogers wrote: Opteron boards get pretty damn close to Big Iron SMP fabric performance in a cheap package. Given how many companies have announced plans to produce Opteron server boards with Infiniband fabrics directly integrated into

Re: [HACKERS] Replication on the backend

2005-12-10 Thread Markus Schiltknecht
Hello, On Fri, 2005-12-09 at 08:47 -0500, Christopher Browne wrote: We *know* (particularly those of us that have had involvement in actually implementing replication systems used in production environments) that user space implementations of replication can function satisfactorily. We've

[HACKERS] Join with an array

2006-02-23 Thread Markus Schiltknecht
Hi, I'm trying to speed up a query with a lookup table. This lookup table gets very big and should still fit into memory. It does not change very often. Given these facts I decided to use an array, as follows: CREATE TABLE lookup_table (id INT PRIMARY KEY, items INT[] NOT NULL); I know this is

Re: [HACKERS] Join with an array

2006-02-23 Thread Markus Schiltknecht
Hello Martijn, On Thu, 2006-02-23 at 12:44 +0100, Martijn van Oosterhout wrote: SELECT i.id, i.title FROM item i JOIN lookup_table lut ON i.id = ANY(lut.items) WHERE lut.id = $LOOKUP_ID; At the very least you're going to have to tell us which version you are running plus the

Re: [HACKERS] Join with an array

2006-02-23 Thread Markus Schiltknecht
Hello Oleg, On Thu, 2006-02-23 at 15:02 +0300, Oleg Bartunov wrote: have you seen contrib/intarray ? Yes, but in the documentation I did not find anything like 'generate_series' or thelike. Maybe I'm looking at the wrong place, please give me a hint. Regards Markus

Re: [HACKERS] Support Parallel Query Execution in Executor

2006-04-08 Thread Markus Schiltknecht
Hi, On Sat, 2006-04-08 at 13:16 -0400, Jonah H. Harris wrote: On 4/8/06, Tom Lane [EMAIL PROTECTED] wrote: ... but I'm failing to follow where it says that parallel processing will fix that. All I can foresee in that direction is extra data transfer costs, bought at the price of

Re: [HACKERS] Support Parallel Query Execution in Executor

2006-04-10 Thread Markus Schiltknecht
Hi, On Sun, 2006-04-09 at 15:11 -0400, Tom Lane wrote: You can't just retarget a backend to operate in another database, at least not without major changes in that infrastructure. Why not? What would be needed to retarget a backend to operate in another database? Such a retargetting would be

Re: [HACKERS] Support Parallel Query Execution in Executor

2006-04-10 Thread Markus Schiltknecht
Hi Qingqing, On Mon, 2006-04-10 at 16:38 +0800, Qingqing Zhou wrote: Why not? What would be needed to retarget a backend to operate in another database? As Tom pointed out, without big change, a backend on database D1 can't connect to D2. This is because to connect to a database, we need

Re: [HACKERS] Support Parallel Query Execution in Executor

2006-04-10 Thread Markus Schiltknecht
On Mon, 2006-04-10 at 17:22 +0800, Qingqing Zhou wrote: Check the code InitPostgres(). These global varaibles are scattered in many places, so I am not sure if it is easy to write clean code to clear up these variables. But if you can come up with a patch to do reconnect without disconnect,

Re: [HACKERS] Support Parallel Query Execution in Executor

2006-04-10 Thread Markus Schiltknecht
Hi, On Mon, 2006-04-10 at 13:36 -0400, Alvaro Herrera wrote: An idea arising in chat with Joshua Drake: the retargetting code, if it turns out to work and not be excessively expensive, could also be useful to implement a server-side connection pooling of sorts: the postmaster could keep idle

[HACKERS] adding fields to pg_database

2006-04-11 Thread Markus Schiltknecht
Hi, I'm trying to add fields to pg_database in the system catalog. As long as I add fixed-size fields before the VAR LENGTH ones, that's all fine. But adding a VAR LENGTH (text) field initdb fails at: copying template1 to template0. The child process 'postgres' fails with signal 11. Strange

Re: [HACKERS] adding fields to pg_database

2006-04-11 Thread Markus Schiltknecht
On Tue, 2006-04-11 at 14:07 -0400, Jonah H. Harris wrote: Make sure you updated Natts_pg_database, the bootstrap DATA line, and the stuff in src/backend/commands/dbcommands.c. dbcommands.c was the missing peace, thank you! Markus ---(end of

[HACKERS] pseudo-type record arguments for PL-functions

2006-05-04 Thread Markus Schiltknecht
Hi, I'm trying to write a PL/Python function which is to be called from a rule. I'd need pass the OLD and NEW tuple records to the function. Unfortunately that does not work: 'pl/python functions cannot take type record'. What I have figured out by reading the source code: OLD and NEW are pseudo

Re: [HACKERS] Snowball and ispell in tsearch2

2006-06-07 Thread Markus Schiltknecht
Hello Teodor, I've just recently implemented an advanced full-text search function on top of tsearch2. Searching through the manuals and websites to get the snowball stemmer and compile my own module took me way to long. I'd rather go fetch a cup of coffee during a 30 minute download...

[HACKERS] Documenting Replication Solutions

2006-07-22 Thread Markus Schiltknecht
Hi, at the code sprint, we agreed to put together some documentation about current and upcoming replication solutions for PostgreSQL. Is somebody already working on that? For Postgres-R, you can find some information on www.postgres-r.org. And I've just opened a project on pgFoundry. This

Re: [HACKERS] Documenting Replication Solutions

2006-07-24 Thread Markus Schiltknecht
, PgCluster, etc... - What's the difference between Slony-II and Postgres-R? Where on the website do we put such a FAQ? Or should some of these questions be part of the main FAQ? Regards Markus Christopher Browne wrote: Quoth Markus Schiltknecht [EMAIL PROTECTED]: at the code sprint, we agreed

Re: [HACKERS] [PATCHES] Replication Documentation

2006-08-02 Thread Markus Schiltknecht
Hi, Bruce Momjian wrote: The idea being to define issues like multi/single master, async vs, sync, and mention the projects which are in each category. You could even add shared-nothing vs. shared-disk nodes. Generally I'd say it makes sense to 'educate' people, but does it really make

Re: [HACKERS] Replication Documentation

2006-08-02 Thread Markus Schiltknecht
Hi, Andrew Hammond wrote: I can see value in documenting what replication systems are known to work (for some definition of work) with a given release in the documentation for that release. Five years down the road when I'm trying to implement replication for a client who's somehow locked

Re: [HACKERS] standard interfaces for replication providers

2006-08-07 Thread Markus Schiltknecht
Hi, José Orlando Pereira wrote: I would argue that people haven't been able to build production-grade multi-master replication, in part, due to the barrier of not having a standard database-agnostic API. :-) In fact, the problem is not the lack of a standard API but the lack of an API at

Re: [HACKERS] standard interfaces for replication providers

2006-08-08 Thread Markus Schiltknecht
Hi, Jose Orlando Pereira wrote: Sorry, stuff was put twice in the zip file making it somewhat confusing. It is in postgresql-g-0.1/javasrc/GordaInterfaces/docs/gapi.pdf or directly on the web site at http://gorda.di.uminho.pt/download/reports/gapi.pdf. Thank you. I've just had a quick glance

Re: [HACKERS] standard interfaces for replication providers

2006-08-08 Thread Markus Schiltknecht
Hello Christopher, Christopher Browne wrote: Most databases that are interesting to replicate are implemented in C or C++, thereby implying that a suitably deep API needs to be implemented in C. I generally agree with you. Although it's probably worth mentioning that the API they propose

Re: [HACKERS] standard interfaces for replication providers

2006-08-09 Thread Markus Schiltknecht
Hello Alfranio, alfranio correia junior wrote: Of course not... It is impossible to build a replication system entirely by only using triggers... Hm, I don't think it's impossible, just unpractical. Anyway, if you say so yourself, I really doubt the use of GAPI. If you need to create your

[HACKERS] news server does not respond

2006-08-17 Thread Markus Schiltknecht
Hi, what's up with the news server? My reader can't connect since yesterday evening (MESZ, UTC+2). Telnetting gives: [EMAIL PROTECTED]:~$ telnet news.postgresql.org nntp Trying 200.46.204.72... telnet: Unable to connect to remote host: Connection refused A note about my current NNTP

Re: [HACKERS] Replication

2006-08-21 Thread Markus Schiltknecht
Hi, AgentM wrote: I would imagine that multi-master synchronous replication would be fairly trivial to implement with 2PC and wal-shipping available, no? Yes, that could be done. And AFAIK eigter pgpool or PgCluster (1) try to do sync, multi-master replication that way. The problem is that

Re: [HACKERS] Replication

2006-08-21 Thread Markus Schiltknecht
Jeff Davis wrote: How does WAL shipping help synchronous replication? The WAL is written _before_ commit, logging all the changes the transaction wants to write to the disk. This makes it look very similar to what is needed for synchronous replication. Instead of waiting for confirmation

Re: [HACKERS] Replication

2006-08-21 Thread Markus Schiltknecht
Alvaro Herrera wrote: But the confirmation that needs to come is that the WAL changes have been applied (fsync'ed), so the performance will be terrible. So bad, that I don't think anyone will want to use such a replication system ... Yeah, that's the big problem of sync, multi-master

Re: [HACKERS] Replication

2006-08-21 Thread Markus Schiltknecht
Gregory Maxwell wrote: infiniband is cheap.. Can I get one? I'd love to run some tests with Postgres-R ;-) ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq

Re: [HACKERS] Replication

2006-08-21 Thread Markus Schiltknecht
Jeff Davis wrote: Synchronous replication (to me) means that the data has been written to permanent storage on all masters and all slaves before any master or slave reports a successful COMMIT. Are you suggesting that you ship the WAL over the network, wait for it to be written to the slave, and

Re: [HACKERS] news server does not respond

2006-08-22 Thread Markus Schiltknecht
Marc G. Fournier wrote: Fixed, sorry for delay ... Good, thank you. But I've already switched back to IMAP, with subfolders and automatic filtering. Has the advantage of being available from any IMAP capable client _and_ saving the flags. Looks like the news server is not used that much,

Re: [HACKERS] news server does not respond

2006-08-23 Thread Markus Schiltknecht
Christopher Browne wrote: Yeah, and you can't complain when you're cut off... :-) yeah, known problem... I used gmane to track the list, but... Regards Markus ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ?

Re: [HACKERS] Replication

2006-08-23 Thread Markus Schiltknecht
Hannu Krosing wrote: But any sync _replication_ system will have severe impact on performance. My guess is that for a full sync replication, going from 1 server to 2 will actually lower performance andsome small gains would be possible only starting from 3rd server. Only testing will show

Re: [HACKERS] Replication

2006-08-23 Thread Markus Schiltknecht
Hannu Krosing wrote: But if you have very few writes, then there seems no reason to do sync anyway. I think there is one: high-availability. A standby-server which can continue if your primary fails. Of course sync is only needed if you absolutely cannot effort loosing any committed

Re: [HACKERS] Replication

2006-08-23 Thread Markus Schiltknecht
Hi, Hannu Krosing wrote: but it still needs to do at least one network roundtrip + any needed testing on all nodes + WAL sync on all nodes before it can COMMIT, no? No. It only needs the 'roundtrip' in the sense that a transaction sends out its writeset and has to wait for the GCS to have it

Re: [HACKERS] Replication

2006-08-24 Thread Markus Schiltknecht
Hi, Jeff Davis wrote: I disagree about high-availability. In fact, I would say that sync replication is trading availability and performance for synchronization (which is a valid tradeoff, but costly). In a way, replication is for databases systems what RAID1 is for hard drives. Having

Re: [HACKERS] Replication

2006-08-25 Thread Markus Schiltknecht
Jeff Davis wrote: Which doesn't work very well in the case of two groups of servers set up in two physical locations. I can see two possibilities: (1) You require a quorum to be effective, in which case your cluster of databases is only as reliable as the location which holds more servers. (2)

Re: [HACKERS] integration of pgcluster into postgresql

2006-08-26 Thread Markus Schiltknecht
Jonah H. Harris wrote: Support of PGCluster-I, which we're discussing here, is being dropped in favor of the shared-disk PGCluster-II which was demonstrated at the anniversary conference. IIRC, PGCluster-I does use command-based replication but is merged into the parser in such a way as to make

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, NikhilS wrote: The following things are TODOs: iv) Auto generate rules using the checks mentioned for the partitions, to handle INSERTs/DELETEs/UPDATEs to navigate them to the appropriate child. Note that checks specified directly on the master table will get inherited automatically.

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, NikhilS wrote: Our current partitioning solution is based on inheritance. With that in mind, for 8.3 I thought an implementation based on auto rules creation would be the way to go. That's completely reasonable. And as I've said, it's probably even a step towards what I've outlined

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, Simon Riggs wrote: I agree with much of your post, though this particular point caught my eye. If you'll forgive me for jumping on an isolated point in your post: No problem. Multi-table indexes sound like a good solution until you consider how big they would be. The reason we need a

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, Gregory Stark wrote: Put another way, multi-table indexes defeat the whole purpose of having partitioned the table in the first place. If you could have managed a single massive index then you wouldn't have bothered partitioning. That depends very much on the implementation of the

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, Simon Riggs wrote: Most high volume tables are Fact tables with potentially more than 1 row per Object/Dimension, so the unique index isn't appropriate in those cases. When partitioning a Major Entity its much easier to regard the PK as the partitioning key + unique key, which is

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, Gregory Stark wrote: However there are also cases such as where you have a=0..99 in one partition and a=100..199 in partition two, etc. It could still automatically build indexes on (a,b,c) on each partition and somehow note that the unique constraint is guaranteed across the whole

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, Joshua D. Drake wrote: If we don't have multi-table indexes how do we enforce a primary key against a partitioned set? The executor would have to be clever enough to not do a single index scan, but possibly scan through multiple indexes when asking for uniqueness, depending on the

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Simon Riggs wrote: The planner already uses the Append node to put together multiple plans. The great thing is it will put together IndexScans and SeqScans as applicable. No need for multi-scans as a special node type. Yes... only that mixing 'concurrent' index scans in the right order would

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Andrew Dunstan wrote: David Fetter wrote: That would be where the provably-distinct part comes in, so yes. That assumes you can provide some provably distinct test. In the general case I have in mind that isn't so. Could you please give a somewhat more concrete example, I'm not following

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, Andrew Dunstan wrote: I guess my point was really that multi-table indexes might have uses beyond partitioning. Aha, now I understand. Thanks for the clarification. Say I have two tables, each with a field FKed to a field in a third table. I'd like to create the values to be unique

Re: [HACKERS] Auto Partitioning

2007-04-05 Thread Markus Schiltknecht
Hi, Zeugswetter Andreas ADI SD wrote: CREATE INDEX x ON test(a, b, c); isn't the same as CRETAE INDEX x ON test(c, b, a); That is only a problem if you also want to avoid a sort (e.g. for an order by), ..or if you want to use that index for 'WHERE a = 5'. The first one is probably

Re: [HACKERS] Auto Partitioning

2007-04-05 Thread Markus Schiltknecht
Hi, Martijn van Oosterhout wrote: The executor would have to be clever enough to not do a single index scan, but possibly scan through multiple indexes when asking for uniqueness, depending on the partitioning rule set. But it's not the executor that checks uniqueness, it's built into the

Re: [HACKERS] Auto Partitioning

2007-04-06 Thread Markus Schiltknecht
Simon Riggs wrote: i.e. if we have partitions for each year (2001, 2002, 2003 2004, 2005, 2006, 2007) AND we have already proved that 2005 is excluded when we have a WHERE clause saying year = 2006, then we should be able to use the ordering to prove that partitions for 2004 and before are also

Re: [HACKERS] Hacking on PostgreSQL via GIT

2007-04-19 Thread Markus Schiltknecht
Hi, Alvaro Herrera wrote: Which is not always what happens in reality. Consider for example that we borrowed some files from NetBSD, OpenBSD, Tcl, zic and others. It would be nice to know exactly at what point we borrowed the file, so we can go to the upstream repo and check if there's any

Re: [HACKERS] Hacking on PostgreSQL via GIT

2007-04-19 Thread Markus Schiltknecht
Hi Jim C. Nasby wrote: I understand the argument about metadata and all, and largely agree with it. But on the other hand I think a version identifier is a critical piece of information; it's just as critical as the file name when it comes to identifying the information contained in the file.

Re: [HACKERS] COPYable logs status

2007-06-09 Thread Markus Schiltknecht
Hi, Tom Lane wrote: We *have* a log-writing process. The problem is in getting the data to it. Remember the imessages approach I'm using for Postgres-R? It passes messages around using shared memory and signals the receiver on incoming data. It's not perfect, sure, but it's a general

Re: [HACKERS] COPYable logs status

2007-06-09 Thread Markus Schiltknecht
Tom Lane wrote: Markus Schiltknecht [EMAIL PROTECTED] writes: Tom Lane wrote: We *have* a log-writing process. The problem is in getting the data to it. Remember the imessages approach I'm using for Postgres-R? It passes messages around using shared memory and signals the receiver

Re: [HACKERS] PG-MQ?

2007-06-20 Thread Markus Schiltknecht
Hi Chris, Chris Browne wrote: I'm seeing some applications where it appears that there would be value in introducing asynchronous messaging, ala message queueing. http://en.wikipedia.org/wiki/Message_queue ISTM that 'message queue' is a way too general term. There are hundreds of different

Re: [HACKERS] [FEATURE REQUEST] Streaming Onlinebackup (Maybe OFFTOPIC)

2007-09-07 Thread Markus Schiltknecht
Hi, apoc9009 wrote: Thadt is Replication NOT Backup I've now read all of your messages in this thread, but I simply fail to understand why you are that much opposed to the term 'replication'. I think the only thing which comes any close to what you're looking for is replication (in

Re: [HACKERS] [FEATURE REQUEST] Streaming Onlinebackup (Maybe OFFTOPIC)

2007-09-07 Thread Markus Schiltknecht
Hi, apoc9009 wrote: Translation for you: A Backup is a File or Set of Files thadt contains the Data of your Business critical Informations. It should not be Archived on the same place, the same House or the same Room. I disagree, a backup does not necessarily have to be a single file or a

[HACKERS] terms for database replication: synchronous vs eager

2007-09-07 Thread Markus Schiltknecht
Hi, I'm asking for advice and hints regarding terms in database replication, especially WRT Postgres-R. (Sorry for crossposting, but I fear not reaching enough people on the Postgres-R ML alone) I'm struggling on how to classify the Postgres-R algorithm. Up until recently, most people

Re: [HACKERS] WIP patch for latestCompletedXid method of computing snapshot xmax

2007-09-08 Thread Markus Schiltknecht
Hello Tom, Tom Lane wrote: So on the strength of that, I'm going to go ahead and commit the patch, but I'd be interested to see benchmarks from people with access to better hardware. I've just completed two dbt2 test runs on a mid-level system, with 4GB RAM and a 7 disk SATA RAID 1+0 w/ BBU.

Re: [HACKERS] terms for database replication: synchronous vs eager

2007-09-14 Thread Markus Schiltknecht
Hello Jan, thank you for your feedback. Jan Wieck wrote: On 9/7/2007 11:01 AM, Markus Schiltknecht wrote: This violates the common understanding of synchrony, because you can't commit on a node A and then query another node B and expect it be coherent immediately. That's right

Re: [HACKERS] terms for database replication: synchronous vs eager

2007-09-14 Thread Markus Schiltknecht
Hi, Chris Browne wrote: The approach that was going to be taken, in Slony-II, to apply locks as early as possible so as to find conflicts as soon as possible, rather than waiting, seems eager to me. Agreed. WRT locking, one might also call it pessimistic, but that sounds so... negative. I

Re: [HACKERS] [COMMITTERS] pgsql: Fix up text concatenation so that it accepts all the reasonable

2007-09-17 Thread Markus Schiltknecht
Hi, Gregory Stark wrote: Perhaps if you're doing some form of replication between different architectures you might want to use binary representation for your transfers. Or if you're doing something in a PL language like compressing or bundling up multiple data in a container format or

Re: [HACKERS] [COMMITTERS] pgsql: Fix up text concatenation so that it accepts all the reasonable

2007-09-17 Thread Markus Schiltknecht
Hello Tom, Tom Lane wrote: I think you'd be nuts to bet your data on the binary representations really being cross-platform compatible. Can you elaborate on this? AFAICT the send/recv functions use network byte ordering. What are the other problems between different architectures? There

Re: [HACKERS] [COMMITTERS] pgsql: Fix up text concatenation so that it accepts all the reasonable

2007-09-17 Thread Markus Schiltknecht
Hi, Tom Lane wrote: Well, you're probably fine with integers and text, but beyond that it gets a bit more dicey. I wouldn't want to assume that floats are compatible across any random pair of architectures, and in the more complex datatypes (such as arrays or geometric types) there might be

Re: [HACKERS] Raw device I/O for large objects

2007-09-18 Thread Markus Schiltknecht
Hi, Georgi Chulkov wrote: Please allow me to ask then: 1. In your opinion, would the above scenario indeed benefit from a raw-device interface for large objects? No, because file systems also try to do what you outline above. They certainly don't split sequential data up into blocks and

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Markus Schiltknecht
Hello Andrew, Andrew Sullivan wrote: Yes. And silent as ever. :-) Are the slides of your PgCon talk available for download somewhere? BTW: up until recently, there was yet another mailing list: [EMAIL PROTECTED] It was less focused on hooks and got at least some traffic. :-) Are those

Re: [HACKERS] Ordered Append Node

2007-11-22 Thread Markus Schiltknecht
Hello Gregory, Gregory Stark wrote: I've been hacking on the idea of an Append node which maintains the ordering of its subtables merging their records in order. This is important for partitioned tables since otherwise a lot of plans are not available such as merge joins. Cool! Some time

Re: [HACKERS] Ordered Append Node

2007-11-23 Thread Markus Schiltknecht
Hello Gregory, Gregory Stark wrote: It is kind of like a merge join but not quite. It's interleaving rows rather than matching them up. It's more like the final merge of a sort which also uses a heap to efficiently find the next value from the source tapes. Well, maybe my point here is: why

Re: [HACKERS] Ordered Append Node

2007-11-23 Thread Markus Schiltknecht
Hi, Florian Weimer wrote: Florian Weimer wrote: I think you need it because there are potentially many input types. Eh, tapes. Aha.. Given the partitioning case, I'd expect all rows to have an equal tuple descriptor. Maybe this is a matter of what to optimize, then? Could you elaborate

Re: [HACKERS] Ordered Append Node

2007-11-23 Thread Markus Schiltknecht
Hi, Florian Weimer wrote: I think you need it because there are potentially many input types. Given the partitioning case, I'd expect all rows to have an equal tuple descriptor. Maybe this is a matter of what to optimize, then? Could you elaborate on what use case you have in mind?

Re: [HACKERS] Ordered Append Node

2007-11-23 Thread Markus Schiltknecht
Gregory Stark wrote: Not quite the same since the Executor-based implementation would have a static tree structure based on the partitions. Even if the partitions are all empty except for one or two you would still have to push the result records through all the nodes for the empty partitions.

Re: [HACKERS] Ordered Append Node

2007-11-23 Thread Markus Schiltknecht
Hi, Heikki Linnakangas wrote: AFAICT it's roughly the same data structure as the zipper tree you envisioned, but not implemented with separate executor nodes for each level. Aha, that sounds good to me. ;-) As I've already mentioned, I think it's even better to simply show the user a

Re: [HACKERS] VLDB Features

2007-12-12 Thread Markus Schiltknecht
Hi, Josh Berkus wrote: Here's the other VLDB features we're missing: Parallel Query Uh.. this only makes sense in a distributed database, no? I've thought about parallel querying on top of Postgres-R. Does it make sense implementing some form of parallel querying apart from the

Re: [HACKERS] VLDB Features

2007-12-12 Thread Markus Schiltknecht
Hi Josh, Josh Berkus wrote: Sure. Imagine you have a 5TB database on a machine with 8 cores and only one concurrent user. You'd like to have 1 core doing I/O, and say 4-5 cores dividing the scan and join processing into 4-5 chunks. Ah, right, thank for enlightenment. Heck, I'm definitely

Re: [HACKERS] VLDB Features

2007-12-13 Thread Markus Schiltknecht
Hello Gregory, Gregory Stark wrote: Oracle is using Direct I/O so they need the reader and writer threads to avoid blocking on i/o all the time. We count on the OS doing readahead and buffering our writes so we don't have to. Direct I/O and needing some way to do asynchronous writes and reads

Re: [HACKERS] [GENERAL] Slow PITR restore

2007-12-14 Thread Markus Schiltknecht
Hi, Alvaro Herrera wrote: Simon Riggs wrote: ISTM its just autovacuum launcher + Hot Standby mixed. I don't think you need a launcher at all. Just get the postmaster to start a configurable number of wal-replay processes (currently the number is hardcoded to 1). I also see similarity to

Re: [HACKERS] [GENERAL] Slow PITR restore

2007-12-14 Thread Markus Schiltknecht
Hannu Krosing wrote: until N fubbers used ..whatever a fubber is :-) Nice typo! Markus ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings

Re: [HACKERS] [GENERAL] Slow PITR restore

2007-12-14 Thread Markus Schiltknecht
Hello Hannu, Hannu Krosing wrote: (For parallelized queries, superuser privileges might appear wrong, but I'm arguing that parallelizing the rights checking isn't worth the trouble, so the initiating worker backend should do that and only delegate safe jobs to hepler backends. Or is that a

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-04 Thread Markus Schiltknecht
Hello Simon, Simon Riggs wrote: I've come up with an alternative concept to allow us to discuss the particular merits of each. ISTM that this new proposal has considerable potential. Hm.. interesting idea. If we were able to keep track of which sections of a table are now read-only then we

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-04 Thread Markus Schiltknecht
Hi, Simon Riggs wrote: - any Fact table where measurements/observations/events are accumulated e.g. Web Hits (any Internet events) Call Detail Records Sales Security Events Scientific Measurements Process Control - any Major Entity where new entities are created from a sequence e.g. Orders,

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-04 Thread Markus Schiltknecht
Hi, Simon Riggs wrote: The smaller the partition size the greater the overhead of managing it. Also I've been looking at read-only tables and compression, as you may know. My idea was that in the future we could mark segments as either - read-only - compressed - able to be shipped off to

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-04 Thread Markus Schiltknecht
Hi, Simon Riggs wrote: On Fri, 2008-01-04 at 13:06 -0500, Andrew Sullivan wrote: On Fri, Jan 04, 2008 at 01:29:55PM +0100, Markus Schiltknecht wrote: Agreed. Just a minor note: I find marked read-only too strong, as it implies an impossibility to write. I propose speaking about mostly-read

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-05 Thread Markus Schiltknecht
Andrew Sullivan wrote: On Fri, Jan 04, 2008 at 10:26:54PM +0100, Markus Schiltknecht wrote: I'm still puzzled about how a DBA is expected to figure out which segments to mark. I think that part might be hand-wavy still. But once this facility is there, what's to prevent the current active

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-05 Thread Markus Schiltknecht
Hi, [EMAIL PROTECTED] wrote: The main proposal deliberately has few, if any, knobs and dials. That's a point of philosophy that I've had views on previously: my normal stance is that we need some knobs to allow the database to be tuned to individual circumstances. One thought I had back

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-05 Thread Markus Schiltknecht
Hi, Simon Riggs wrote: On Fri, 2008-01-04 at 22:26 +0100, Markus Schiltknecht wrote: I'm still puzzled about how a DBA is expected to figure out which segments to mark. Simon, are you assuming we are going to pass on segment numbers to the DBA one day? No Way! Ah, I'm glad ;-) Simon

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-05 Thread Markus Schiltknecht
Hi, Robert Treat wrote: Personally I cant say it complicates things, because it isn't clear how it will be managed. :-) Well, management of relations is easy enough, known to the DBA and most importantly: it already exists. Having to set up something which is *not* tied to a relation

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-06 Thread Markus Schiltknecht
Hi, Gokulakannan Somasundaram wrote: On Jan 5, 2008 6:15 PM, [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: One thought I had back then, with partitioned tables was gee -- B-tree index is already doing a partition; why do a manual partition on top of that?. Can you please

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-06 Thread Markus Schiltknecht
Hi, Robert Treat wrote: On Saturday 05 January 2008 14:02, Markus Schiltknecht wrote: To satisfy all the different requirements of partitioning with segments based partitioning, we'd have to allow a table to span multiple table spaces. I'm not very keen on going that way. Why? Uh

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-07 Thread Markus Schiltknecht
Hi Csaba, Csaba Nagy wrote: One additional thought: what about a kind of segment fill factor ? Meaning: each segment has some free space reserved for future updates/inserts of records in the same range of it's partitioning constraint. And when inserting/updating you put the new record into the

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-07 Thread Markus Schiltknecht
Hi, Csaba Nagy wrote: Sure, but it could be configurable and should only be enabled if the table is marked as partitioned on some condition... As I'm regarding SE as an optimization, I disagree here.. As all optimizations, SE should conceptually be reasonably close to cost-less when

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-07 Thread Markus Schiltknecht
Hi, Andrew Sullivan wrote: On Sat, Jan 05, 2008 at 08:02:41PM +0100, Markus Schiltknecht wrote: Well, management of relations is easy enough, known to the DBA and most importantly: it already exists. Having to set up something which is *not* tied to a relation complicates things just because

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-07 Thread Markus Schiltknecht
Hi, Andrew Sullivan wrote: On Mon, Jan 07, 2008 at 07:16:35PM +0100, Markus Schiltknecht wrote: Does anything speak against letting the DBA handle partitions as relations? Yes: it doesn't solve the problem I have, which is that I don't want to have to manage a whole bunch of tables. I want

Re: [HACKERS] Proposal - libpq Type System beta-0.8a (was PGparam)

2008-01-08 Thread Markus Schiltknecht
Hi, Andrew Chernow wrote: It might be something with the attachment, who knows. Most probably that was the case, yes. The -hackers list is limited, please use -patches to send patches. ;-) Regards Markus ---(end of broadcast)--- TIP 6:

[HACKERS] Named vs Unnamed Partitions

2008-01-08 Thread Markus Schiltknecht
Hi, IMO, the lengthy discussion about Segment Exclusion and Segment Visibility Maps has long turned into a discussion about partitioning in general. I'm thankful for all the new insights it has brought me and I want to continue sharing my view on things. What's following is highly

Re: [HACKERS] Some ideas about Vacuum

2008-01-09 Thread Markus Schiltknecht
Hi, Gokulakannan Somasundaram wrote: If we can ask the Vacuum process to scan the WAL log, it can get all the relevant details on where it needs to go. You seem to be assuming that only few tuples have changed between vacuums, so that WAL could quickly guide the VACUUM processes to the

[HACKERS] LD_LIBRARY_PATH not honored on Debian unstable

2008-01-09 Thread Markus Schiltknecht
Hi, I'm trying to run 'make check' on a 64bit Debian unstable. That aborts after 60 seconds due to not being able to connect to the postmaster. I figured that there's nothing wrong with the postmaster, rather psql can't start up, because it gets linked against an older libpq.so.5. It looks

Re: [HACKERS] LD_LIBRARY_PATH not honored on Debian unstable

2008-01-09 Thread Markus Schiltknecht
Andrew Dunstan wrote: Smells suspiciously like an rpath problem to me. What are your configure settings? Ah, yeah, I see. Using something else than --prefix=/usr helped. Thanks for the hint! Regards Markus ---(end of broadcast)--- TIP 5:

  1   2   >