Re: [HACKERS] Streaming replication and non-blocking I/O

Dimitri Fontaine escreveu: > It should be possible to be in contrib and installed by default, even > And it could be uninstall too. Let's not do it for core functionalities. -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgres

Re: [HACKERS] Streaming replication and non-blocking I/O

Heikki Linnakangas writes: > The module doesn't need to touch backend internals much at all, no > tinkering with shared memory for example, so I would feel much better > about moving that out of src/backend. Not sure where, though; it's not > an executable, so src/bin is hardly the right place, bu

Re: [HACKERS] Streaming replication and non-blocking I/O

Heikki Linnakangas wrote: > I've pushed that 'replication-dynmodule' branch in my git repo. The diff > is hard to read, because it mostly just moves code around, but I've > attached libpqwalreceiver.c here, which is the dynamic module part. You > can also browse the tree via the web interface > (ht

Re: [HACKERS] Streaming replication and non-blocking I/O

Tom Lane wrote: > Heikki Linnakangas writes: >> Before we sprinkle all the global variables it touches with that, let me >> explain what I meant by dividing walreceiver code differently between >> dynamically loaded module and backend code. Right now I have to go to >> sleep, though, but I'll try

Re: [HACKERS] Streaming replication and non-blocking I/O

Andrew Dunstan writes: > I ran an extra cycle. Still a bit of work to do: > Well, at least now we're down to the variables that haven't got PGDLLIMPORT, rather than wondering what's wrong with the build ...

Re: [HACKERS] Streaming replication and non-blocking I/O

Tom Lane wrote: Before we sprinkle all the global variables it touches with that, let me explain what I meant by dividing walreceiver code differently between dynamically loaded module and backend code. Right now I have to go to sleep, though, but I'll try to get back to during the weekend.

Re: [HACKERS] Streaming replication and non-blocking I/O

Heikki Linnakangas writes: > Tom Lane wrote: >> Specifically, I think you missed out $(BE_DLLLIBS) in SHLIB_LINK. >> We'll find out at the next mingw build... > Thanks. But what is BE_DLLLIBS? I can't find any description of it. It was the wrong theory anyway --- it already is included (in Makef

Re: [HACKERS] Streaming replication and non-blocking I/O

Tom Lane wrote: > I wrote: >> I think it's a tools/build process problem and should be attacked that >> way. > > Specifically, I think you missed out $(BE_DLLLIBS) in SHLIB_LINK. > We'll find out at the next mingw build... Thanks. But what is BE_DLLLIBS? I can't find any description of it. I sus

Re: [HACKERS] Streaming replication and non-blocking I/O

Aidan Van Dyk writes: > Maybe an insane question, but why can postmaster just not "exec" > walreceiver? It'd greatly complicate access to shared memory. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subsc

Re: [HACKERS] Streaming replication and non-blocking I/O

I wrote: > I think it's a tools/build process problem and should be attacked that > way. Specifically, I think you missed out $(BE_DLLLIBS) in SHLIB_LINK. We'll find out at the next mingw build... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@po

Re: [HACKERS] Streaming replication and non-blocking I/O

* Heikki Linnakangas [100115 15:20]: > Ok. I'll look at splitting walreceiver code between the shared module > and backend binary slightly differently. At first glance, it doesn't > seem that hard after all, and will make the code more modular anyway. Maybe an insane question, but why can postma

Re: [HACKERS] Streaming replication and non-blocking I/O

Heikki Linnakangas writes: > Magnus Hagander wrote: >> Yeah. FWIW, I don't use mingw do do any windows development, but >> definitely +1 on working hard to keep support for it if at all >> possible. > Ok. I'll look at splitting walreceiver code between the shared module > and backend binary sligh

Re: [HACKERS] Streaming replication and non-blocking I/O

Magnus Hagander wrote: > 2010/1/15 Andrew Dunstan : >> >> Heikki Linnakangas wrote: >>> Do people still use MinGW for any real work? Could we just drop >>> walreceiver support from MinGW builds? >>> >>> Or maybe we should consider splitting walreceiver into two parts after >>> all. Only the bare mi

Re: [HACKERS] Streaming replication and non-blocking I/O

2010/1/15 Andrew Dunstan : > > > Heikki Linnakangas wrote: >> >> Do people still use MinGW for any real work? Could we just drop >> walreceiver support from MinGW builds? >> >> Or maybe we should consider splitting walreceiver into two parts after >> all. Only the bare minimum that needs to access

Re: [HACKERS] Streaming replication and non-blocking I/O

Heikki Linnakangas wrote: Do people still use MinGW for any real work? Could we just drop walreceiver support from MinGW builds? Or maybe we should consider splitting walreceiver into two parts after all. Only the bare minimum that needs to access libpq would go into the shared object, and the

Re: [HACKERS] Streaming replication and non-blocking I/O

2010/1/15 Heikki Linnakangas : > Fujii Masao wrote: >> On Wed, Jan 13, 2010 at 3:37 AM, Magnus Hagander wrote: This change which moves walreceiver process into a dynamically loaded module caused the following compile error on my MinGW environment. >>> That sounds strange - it should pick

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao wrote: > On Wed, Jan 13, 2010 at 3:37 AM, Magnus Hagander wrote: >>> This change which moves walreceiver process into a dynamically loaded >>> module caused the following compile error on my MinGW environment. >> That sounds strange - it should pick those up from the -lpostgres. Any >>

Re: [HACKERS] Streaming replication and non-blocking I/O

On Thu, Jan 14, 2010 at 9:14 PM, Heikki Linnakangas wrote: > After reading up on SSL_read() and SSL_pending(), it seems that there is > unfortunately no reliable way of checking if there is incoming data that > can be read using SSL_read() without blocking, short of putting the > socket to non-blo

Re: [HACKERS] Streaming replication and non-blocking I/O

2010/1/14 Heikki Linnakangas : > Fujii Masao wrote: >> On Wed, Jan 13, 2010 at 7:27 PM, Heikki Linnakangas >> wrote: >>> the frontend always puts the >>> connection to non-blocking mode, while the backend uses blocking mode. >> >> Really? By default (i.e., without the expressly setting by using >>

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao wrote: > On Wed, Jan 13, 2010 at 7:27 PM, Heikki Linnakangas > wrote: >> the frontend always puts the >> connection to non-blocking mode, while the backend uses blocking mode. > > Really? By default (i.e., without the expressly setting by using > PQsetnonblocking()), the connection is

Re: [HACKERS] Streaming replication and non-blocking I/O

On Wed, Jan 13, 2010 at 7:27 PM, Heikki Linnakangas wrote: > the frontend always puts the > connection to non-blocking mode, while the backend uses blocking mode. Really? By default (i.e., without the expressly setting by using PQsetnonblocking()), the connection is set to blocking mode even in f

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao wrote: > Done. Currently there is no new libpq function for replication. The > walreceiver uses only existing functions like PQconnectdb, PQexec, > PQgetCopyData, etc. > > git://git.postgresql.org/git/users/fujii/postgres.git > branch: replication Thanks! I'm afraid we haven't qu

Re: [HACKERS] Streaming replication and non-blocking I/O

Thanks for your advice! On Wed, Jan 13, 2010 at 3:37 AM, Magnus Hagander wrote: >> This change which moves walreceiver process into a dynamically loaded >> module caused the following compile error on my MinGW environment. > > That sounds strange - it should pick those up from the -lpostgres. Any

Re: [HACKERS] Streaming replication and non-blocking I/O

On Tue, Jan 12, 2010 at 17:58, Fujii Masao wrote: > On Tue, Dec 22, 2009 at 8:49 PM, Heikki Linnakangas > wrote: >>> Umm.., I still cannot find the place where the walreceiver module is >>> loaded by using load_external_function() in your 'replication' branch. >>> Also the compilation of that bra

Re: [HACKERS] Streaming replication and non-blocking I/O

On Tue, Dec 22, 2009 at 8:49 PM, Heikki Linnakangas wrote: >> Umm.., I still cannot find the place where the walreceiver module is >> loaded by using load_external_function() in your 'replication' branch. >> Also the compilation of that branch fails. Is the 'pushed' branch the >> latest? Sorry if

Re: [HACKERS] Streaming replication and non-blocking I/O

On Tue, Jan 5, 2010 at 12:22 AM, Heikki Linnakangas wrote: > I've merged the replication branch with PostgreSQL CVS HEAD now, > including the patch for end-of-backup WAL records I committed earlier > today. See 'replication' branch in my git repository. > > There's also a couple of other small cha

Re: [HACKERS] Streaming replication and non-blocking I/O

I've merged the replication branch with PostgreSQL CVS HEAD now, including the patch for end-of-backup WAL records I committed earlier today. See 'replication' branch in my git repository. There's also a couple of other small changes: I believe the SSL stuff isn't really necessary, so I removed it

Re: [HACKERS] Streaming replication and non-blocking I/O

On Tue, Dec 22, 2009 at 8:49 PM, Heikki Linnakangas wrote: > Ah, I see. The changes were not included in the merge commit after all, > but I had simple forgot to "git add" them. Sorry about that, should be > there now. Thanks for doing "git push" again! But the compilation still fails. Attached

Re: [HACKERS] Streaming replication and non-blocking I/O

Greg Stark wrote: > On Tue, Dec 22, 2009 at 6:30 AM, Heikki Linnakangas > wrote: >> I think we can just use load_external_function() to load the library and >> call WalReceiverMain from AuxiliaryProcessMain(). Ie. hard-code the >> library name. Walreceiver is quite tightly coupled with the rest of

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao wrote: > On Tue, Dec 22, 2009 at 3:30 PM, Heikki Linnakangas > wrote: >> I think we can just use load_external_function() to load the library and >> call WalReceiverMain from AuxiliaryProcessMain(). Ie. hard-code the >> library name. Walreceiver is quite tightly coupled with the rest o

Re: [HACKERS] Streaming replication and non-blocking I/O

On Tue, Dec 22, 2009 at 6:30 AM, Heikki Linnakangas wrote: > I think we can just use load_external_function() to load the library and > call WalReceiverMain from AuxiliaryProcessMain(). Ie. hard-code the > library name. Walreceiver is quite tightly coupled with the rest of the > backend anyway, so

Re: [HACKERS] Streaming replication and non-blocking I/O

On Tue, Dec 22, 2009 at 3:30 PM, Heikki Linnakangas wrote: > I think we can just use load_external_function() to load the library and > call WalReceiverMain from AuxiliaryProcessMain(). Ie. hard-code the > library name. Walreceiver is quite tightly coupled with the rest of the > backend anyway, so

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao wrote: > On Tue, Dec 22, 2009 at 2:31 AM, Heikki Linnakangas > wrote: >> 2. Move walreceiver altogether into a loadable module, which is linked >> as usual to libpq. Like e.g contrib/dblink. >> >> Thoughts? Both seem reasonable to me. I tested the 2nd option (see >> 'replication' branc

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao writes: > Though I seem not to understand what a loadable module means, I wonder > how the walreceiver module is loaded. Put it in shared_preload_libraries, perhaps. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To m

Re: [HACKERS] Streaming replication and non-blocking I/O

On Tue, Dec 22, 2009 at 2:31 AM, Heikki Linnakangas wrote: > 2. Move walreceiver altogether into a loadable module, which is linked > as usual to libpq. Like e.g contrib/dblink. > > Thoughts? Both seem reasonable to me. I tested the 2nd option (see > 'replication' branch in my git repository), spl

Re: [HACKERS] Streaming replication and non-blocking I/O

Heikki Linnakangas writes: > Fujii Masao wrote: > I'm not sure which problem in that thread you're referring to, but I can > see two options: > 1. Use dlopen()/dlsym() in walreceiver to use libpq. A bit awkward, > though we could write a bunch of macros to hide that and make the libpq > calls loo

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao wrote: > On Tue, Dec 15, 2009 at 4:11 AM, Tom Lane wrote: >> Hm. Perhaps it should be a loadable plugin and not hard-linked into the >> backend? Compare dblink. > > You mean that such plugin is supplied in shared_preload_libraries, > a new process is forked and the shared-memory rel

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao wrote: > On Fri, Dec 18, 2009 at 11:42 AM, Fujii Masao wrote: >> Okey. Design clarification again; >> >> 0. Begin by connecting to the master using PQconnectdb() with new conninfo >> option specifying the request of replication. The startup packet with the >> request is sent to the mas

Re: [HACKERS] Streaming replication and non-blocking I/O

On Fri, Dec 18, 2009 at 11:42 AM, Fujii Masao wrote: > Okey. Design clarification again; > > 0. Begin by connecting to the master using PQconnectdb() with new conninfo > option specifying the request of replication. The startup packet with the > request is sent to the master, then the backend swit

Re: [HACKERS] Streaming replication and non-blocking I/O

On Thu, Dec 17, 2009 at 10:25 PM, Heikki Linnakangas wrote: > Fujii Masao wrote: >> I'm thinking of making the standby send the "walsender-switch-code" the same >> way >> as application_name; walreceiver always specifies the option like >> "replication=on" >> in conninfo string and calls PQconnec

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao wrote: > I'm thinking of making the standby send the "walsender-switch-code" the same > way > as application_name; walreceiver always specifies the option like > "replication=on" > in conninfo string and calls PQconnectdb(), which sends the code as a part of > startup packet. And, the

Re: [HACKERS] Streaming replication and non-blocking I/O

On Thu, Dec 17, 2009 at 9:02 PM, Heikki Linnakangas wrote: >> And, I think that such backend should switch to walsender mode when the >> startup >> packet arrives. Otherwise, we would have to authenticate such backend twice >> on different context, i.e., a normal backend and walsender. So the set

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao wrote: > On Wed, Dec 16, 2009 at 6:53 PM, Heikki Linnakangas > wrote: >> 0. Begin by connecting to the master just like a normal backend does. We >> don't necessarily need the new ProtocolVersion code either, though it's >> probably still a good idea to reject connections to older serv

Re: [HACKERS] Streaming replication and non-blocking I/O

On Wed, Dec 16, 2009 at 6:53 PM, Heikki Linnakangas wrote: > Great! The logical next step is move the handling of TimelineID and > system identifier out of libpq as well. All right. > 0. Begin by connecting to the master just like a normal backend does. We > don't necessarily need the new Protoc

Re: [HACKERS] Streaming replication and non-blocking I/O

I'm interested in abstracting out features of replication from libpq too. It would be nice if we could implement different communication bus modules. For example if you have dozens of replicas you may want to use something like spread to distribute the records using multicast. Sorry for top posti

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao wrote: > On Tue, Dec 15, 2009 at 3:47 AM, Heikki Linnakangas > wrote: >> Tom Lane wrote: >>> The very, very large practical problem with this is that if you decide >>> to change the behavior at any time, the only way to be sure that the WAL >>> receiver is using the right libpq version

Re: [HACKERS] Streaming replication and non-blocking I/O

On Tue, Dec 15, 2009 at 3:47 AM, Heikki Linnakangas wrote: > Tom Lane wrote: >> The very, very large practical problem with this is that if you decide >> to change the behavior at any time, the only way to be sure that the WAL >> receiver is using the right libpq version is to perform a soname maj

Re: [HACKERS] Streaming replication and non-blocking I/O

On Tue, Dec 15, 2009 at 4:11 AM, Tom Lane wrote: > Hm.  Perhaps it should be a loadable plugin and not hard-linked into the > backend?  Compare dblink. You mean that such plugin is supplied in shared_preload_libraries, a new process is forked and the shared-memory related to walreceiver is create

Re: [HACKERS] Streaming replication and non-blocking I/O

Tom Lane wrote: > Heikki Linnakangas writes: >> BTW, something that's been bothering me a bit with this patch is that we >> now have to link the backend with libpq. I don't see an immediate >> problem with that, but I'm not a packager. Does anyone see a problem >> with that? > > Yeah, I have a pr

Re: [HACKERS] Streaming replication and non-blocking I/O

Heikki Linnakangas writes: > Tom Lane wrote: >> Yeah, I have a problem with that. What's the backend doing with libpq? >> It's not receiving this data, it's sending it. > walreceiver is a postmaster subprocess too. Hm. Perhaps it should be a loadable plugin and not hard-linked into the backend

Re: [HACKERS] Streaming replication and non-blocking I/O

Heikki Linnakangas writes: > It's going to be a bit more complicated in walsender/walreceiver to work > with the libpq COPY API. We're going to need a WAL sending/receiving > protocol on top of it, defined in terms of rows and columns passed > through the COPY protocol. AFAIR, libpq knows essenti

Re: [HACKERS] Streaming replication and non-blocking I/O

Tom Lane wrote: > The very, very large practical problem with this is that if you decide > to change the behavior at any time, the only way to be sure that the WAL > receiver is using the right libpq version is to perform a soname major > version bump. The transformations done by libpq will essent

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao writes: > I'm going to set the new function calling ereport as the current notice > processor by using PQsetNoticeProcessor. But the problem is that only the > completed message like "NOTICE: xxx" is passed to such notice processor, > i.e., the error level itself is not passed. Use PQ

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao writes: > On Mon, Dec 14, 2009 at 11:38 AM, Tom Lane wrote: >> Do we need a new "PQgetXLogData" function at all?  Seems like you could >> shove the data through the COPY protocol and not have to touch libpq >> at all, rather than duplicating a nontrivial amount of code there. > Yeah,

Re: [HACKERS] Streaming replication and non-blocking I/O

On Sat, Dec 12, 2009 at 5:09 PM, Heikki Linnakangas wrote: > Could we change the API of PQgetXLogData to be more like PQgetCopyData? > I'm thinking of removing the timeout argument, and instead looping with > select/poll and PQconsumeInput in the caller. That probably means > introducing a new sta

Re: [HACKERS] Streaming replication and non-blocking I/O

On Mon, Dec 14, 2009 at 11:38 AM, Tom Lane wrote: > Do we need a new "PQgetXLogData" function at all?  Seems like you could > shove the data through the COPY protocol and not have to touch libpq > at all, rather than duplicating a nontrivial amount of code there. Yeah, I also think that all data

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao writes: > On Sun, Dec 13, 2009 at 5:42 AM, Heikki Linnakangas > wrote: >> To implement the timeout in PQgetXLogData(), pqWaitTimed() was changed >> to take a timeout instead of finishing_time argument. Which is a mistake >> because it breaks PQconnectdb, and as I said I don't think >>

Re: [HACKERS] Streaming replication and non-blocking I/O

On Sun, Dec 13, 2009 at 5:42 AM, Heikki Linnakangas wrote: > Walreceiver wants to wait for data to arrive from the master or a > signal. PQgetXLogData(), which is the libpq function to read a piece of > WAL, takes a timeout argument to support that. Walreceiver calls > PQgetXLogData() in an endles

Re: [HACKERS] Streaming replication and non-blocking I/O

Tom Lane wrote: > Heikki Linnakangas writes: >> Changing the finish_time argument to pqWaitTimed into timeout_ms changes >> the behavior connect_timeout option to PQconnectdb. It should wait for >> max connect_timeout seconds in total, but now it is waiting for >> connect_timeout seconds at each s

Re: [HACKERS] Streaming replication and non-blocking I/O

Heikki Linnakangas writes: > Changing the finish_time argument to pqWaitTimed into timeout_ms changes > the behavior connect_timeout option to PQconnectdb. It should wait for > max connect_timeout seconds in total, but now it is waiting for > connect_timeout seconds at each step in the connection

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao wrote: > On Thu, Dec 10, 2009 at 12:00 AM, Tom Lane wrote: >>> The OS buffer is expected to be able to store a large number of >>> XLogRecPtr messages, because its size is small. So it's also OK >>> to just drop it. >> It certainly seems to be something we could improve later, when and

Re: [HACKERS] Streaming replication and non-blocking I/O

On Thu, Dec 10, 2009 at 12:00 AM, Tom Lane wrote: >> The OS buffer is expected to be able to store a large number of >> XLogRecPtr messages, because its size is small. So it's also OK >> to just drop it. > > It certainly seems to be something we could improve later, when and > if evidence emerges

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao writes: > On Wed, Dec 9, 2009 at 3:58 PM, Heikki Linnakangas > wrote: >> But if everyone is happy with just relying on the OS buffer to not fill >> up, let's just drop it. > The OS buffer is expected to be able to store a large number of > XLogRecPtr messages, because its size is sma

Re: [HACKERS] Streaming replication and non-blocking I/O

On Wed, Dec 9, 2009 at 3:58 PM, Heikki Linnakangas wrote: > True. Another idea I had was to introduce a callback that backend libpq > can call when the buffer fills. Walsender would set the callback to > ProcessStreamMsgs(). > > But if everyone is happy with just relying on the OS buffer to not fi

Re: [HACKERS] Streaming replication and non-blocking I/O

Fujii Masao wrote: > On Tue, Dec 8, 2009 at 11:23 PM, Heikki Linnakangas > wrote: >> If that's deemed not good enough, we could modify just internal_flush() >> so that it uses secure_poll to wait for the possibility to either read >> or write, instead of blocking for just write. Whenever there's i

Re: [HACKERS] Streaming replication and non-blocking I/O

On Tue, Dec 8, 2009 at 11:23 PM, Heikki Linnakangas wrote: > The first question is: do we really need to be prepared for that? The > XLogRecPtr acknowledgment messages the client sends are very small, and > if the client is mindful about not sending them too often - perhaps max > 1 ack per 1 recei

[HACKERS] Streaming replication and non-blocking I/O

I find the backend libpq changes related to non-blocking I/O quite complex. Can we find a simpler solution? The problem we're trying to solve is that while the walsender backend sends a lot of WAL records to the client, the client can send a lot of messages to the backend. If volume of the message