Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
The module doesn't need to touch backend internals much at all, no
tinkering with shared memory for example, so I would feel much better
about moving that out of src/backend. Not sure where, though; it's not
an executable, so
Dimitri Fontaine escreveu:
It should be possible to be in contrib and installed by default, even
And it could be uninstall too. Let's not do it for core functionalities.
--
Euler Taveira de Oliveira
http://www.timbira.com/
--
Sent via pgsql-hackers mailing list
Fujii Masao wrote:
On Wed, Jan 13, 2010 at 3:37 AM, Magnus Hagander mag...@hagander.net wrote:
This change which moves walreceiver process into a dynamically loaded
module caused the following compile error on my MinGW environment.
That sounds strange - it should pick those up from the
2010/1/15 Heikki Linnakangas heikki.linnakan...@enterprisedb.com:
Fujii Masao wrote:
On Wed, Jan 13, 2010 at 3:37 AM, Magnus Hagander mag...@hagander.net wrote:
This change which moves walreceiver process into a dynamically loaded
module caused the following compile error on my MinGW
Heikki Linnakangas wrote:
Do people still use MinGW for any real work? Could we just drop
walreceiver support from MinGW builds?
Or maybe we should consider splitting walreceiver into two parts after
all. Only the bare minimum that needs to access libpq would go into the
shared object, and
2010/1/15 Andrew Dunstan and...@dunslane.net:
Heikki Linnakangas wrote:
Do people still use MinGW for any real work? Could we just drop
walreceiver support from MinGW builds?
Or maybe we should consider splitting walreceiver into two parts after
all. Only the bare minimum that needs to
Magnus Hagander wrote:
2010/1/15 Andrew Dunstan and...@dunslane.net:
Heikki Linnakangas wrote:
Do people still use MinGW for any real work? Could we just drop
walreceiver support from MinGW builds?
Or maybe we should consider splitting walreceiver into two parts after
all. Only the bare
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
Magnus Hagander wrote:
Yeah. FWIW, I don't use mingw do do any windows development, but
definitely +1 on working hard to keep support for it if at all
possible.
Ok. I'll look at splitting walreceiver code between the shared
* Heikki Linnakangas heikki.linnakan...@enterprisedb.com [100115 15:20]:
Ok. I'll look at splitting walreceiver code between the shared module
and backend binary slightly differently. At first glance, it doesn't
seem that hard after all, and will make the code more modular anyway.
Maybe an
I wrote:
I think it's a tools/build process problem and should be attacked that
way.
Specifically, I think you missed out $(BE_DLLLIBS) in SHLIB_LINK.
We'll find out at the next mingw build...
regards, tom lane
--
Sent via pgsql-hackers mailing list
Aidan Van Dyk ai...@highrise.ca writes:
Maybe an insane question, but why can postmaster just not exec
walreceiver?
It'd greatly complicate access to shared memory.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes
Tom Lane wrote:
I wrote:
I think it's a tools/build process problem and should be attacked that
way.
Specifically, I think you missed out $(BE_DLLLIBS) in SHLIB_LINK.
We'll find out at the next mingw build...
Thanks. But what is BE_DLLLIBS? I can't find any description of it.
I suspect
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
Tom Lane wrote:
Specifically, I think you missed out $(BE_DLLLIBS) in SHLIB_LINK.
We'll find out at the next mingw build...
Thanks. But what is BE_DLLLIBS? I can't find any description of it.
It was the wrong theory anyway --- it
Tom Lane wrote:
Before we sprinkle all the global variables it touches with that, let me
explain what I meant by dividing walreceiver code differently between
dynamically loaded module and backend code. Right now I have to go to
sleep, though, but I'll try to get back to during the weekend.
Andrew Dunstan and...@dunslane.net writes:
I ran an extra cycle. Still a bit of work to do:
http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=dawn_batdt=2010-01-15%2023:04:54
Well, at least now we're down to the variables that haven't got
PGDLLIMPORT, rather than wondering what's wrong with
Tom Lane wrote:
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
Before we sprinkle all the global variables it touches with that, let me
explain what I meant by dividing walreceiver code differently between
dynamically loaded module and backend code. Right now I have to go to
Heikki Linnakangas wrote:
I've pushed that 'replication-dynmodule' branch in my git repo. The diff
is hard to read, because it mostly just moves code around, but I've
attached libpqwalreceiver.c here, which is the dynamic module part. You
can also browse the tree via the web interface
On Wed, Jan 13, 2010 at 7:27 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
the frontend always puts the
connection to non-blocking mode, while the backend uses blocking mode.
Really? By default (i.e., without the expressly setting by using
PQsetnonblocking()), the connection
Fujii Masao wrote:
On Wed, Jan 13, 2010 at 7:27 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
the frontend always puts the
connection to non-blocking mode, while the backend uses blocking mode.
Really? By default (i.e., without the expressly setting by using
2010/1/14 Heikki Linnakangas heikki.linnakan...@enterprisedb.com:
Fujii Masao wrote:
On Wed, Jan 13, 2010 at 7:27 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
the frontend always puts the
connection to non-blocking mode, while the backend uses blocking mode.
Really? By
On Thu, Jan 14, 2010 at 9:14 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
After reading up on SSL_read() and SSL_pending(), it seems that there is
unfortunately no reliable way of checking if there is incoming data that
can be read using SSL_read() without blocking, short
Thanks for your advice!
On Wed, Jan 13, 2010 at 3:37 AM, Magnus Hagander mag...@hagander.net wrote:
This change which moves walreceiver process into a dynamically loaded
module caused the following compile error on my MinGW environment.
That sounds strange - it should pick those up from the
Fujii Masao wrote:
Done. Currently there is no new libpq function for replication. The
walreceiver uses only existing functions like PQconnectdb, PQexec,
PQgetCopyData, etc.
git://git.postgresql.org/git/users/fujii/postgres.git
branch: replication
Thanks!
I'm afraid we haven't quite
On Tue, Dec 22, 2009 at 8:49 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Umm.., I still cannot find the place where the walreceiver module is
loaded by using load_external_function() in your 'replication' branch.
Also the compilation of that branch fails. Is the 'pushed'
On Tue, Jan 12, 2010 at 17:58, Fujii Masao masao.fu...@gmail.com wrote:
On Tue, Dec 22, 2009 at 8:49 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Umm.., I still cannot find the place where the walreceiver module is
loaded by using load_external_function() in your
On Tue, Jan 5, 2010 at 12:22 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
I've merged the replication branch with PostgreSQL CVS HEAD now,
including the patch for end-of-backup WAL records I committed earlier
today. See 'replication' branch in my git repository.
There's
I've merged the replication branch with PostgreSQL CVS HEAD now,
including the patch for end-of-backup WAL records I committed earlier
today. See 'replication' branch in my git repository.
There's also a couple of other small changes: I believe the SSL stuff
isn't really necessary, so I removed
On Tue, Dec 22, 2009 at 6:30 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
I think we can just use load_external_function() to load the library and
call WalReceiverMain from AuxiliaryProcessMain(). Ie. hard-code the
library name. Walreceiver is quite tightly coupled with the
Fujii Masao wrote:
On Tue, Dec 22, 2009 at 3:30 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
I think we can just use load_external_function() to load the library and
call WalReceiverMain from AuxiliaryProcessMain(). Ie. hard-code the
library name. Walreceiver is quite
Greg Stark wrote:
On Tue, Dec 22, 2009 at 6:30 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
I think we can just use load_external_function() to load the library and
call WalReceiverMain from AuxiliaryProcessMain(). Ie. hard-code the
library name. Walreceiver is quite
On Tue, Dec 22, 2009 at 8:49 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Ah, I see. The changes were not included in the merge commit after all,
but I had simple forgot to git add them. Sorry about that, should be
there now.
Thanks for doing git push again!
But the
On Fri, Dec 18, 2009 at 11:42 AM, Fujii Masao masao.fu...@gmail.com wrote:
Okey. Design clarification again;
0. Begin by connecting to the master using PQconnectdb() with new conninfo
option specifying the request of replication. The startup packet with the
request is sent to the master, then
Fujii Masao wrote:
On Fri, Dec 18, 2009 at 11:42 AM, Fujii Masao masao.fu...@gmail.com wrote:
Okey. Design clarification again;
0. Begin by connecting to the master using PQconnectdb() with new conninfo
option specifying the request of replication. The startup packet with the
request is sent
Fujii Masao wrote:
On Tue, Dec 15, 2009 at 4:11 AM, Tom Lane t...@sss.pgh.pa.us wrote:
Hm. Perhaps it should be a loadable plugin and not hard-linked into the
backend? Compare dblink.
You mean that such plugin is supplied in shared_preload_libraries,
a new process is forked and the
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
Fujii Masao wrote:
I'm not sure which problem in that thread you're referring to, but I can
see two options:
1. Use dlopen()/dlsym() in walreceiver to use libpq. A bit awkward,
though we could write a bunch of macros to hide that
On Tue, Dec 22, 2009 at 2:31 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
2. Move walreceiver altogether into a loadable module, which is linked
as usual to libpq. Like e.g contrib/dblink.
Thoughts? Both seem reasonable to me. I tested the 2nd option (see
'replication'
Fujii Masao masao.fu...@gmail.com writes:
Though I seem not to understand what a loadable module means, I wonder
how the walreceiver module is loaded.
Put it in shared_preload_libraries, perhaps.
regards, tom lane
--
Sent via pgsql-hackers mailing list
Fujii Masao wrote:
On Tue, Dec 22, 2009 at 2:31 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
2. Move walreceiver altogether into a loadable module, which is linked
as usual to libpq. Like e.g contrib/dblink.
Thoughts? Both seem reasonable to me. I tested the 2nd option
On Tue, Dec 22, 2009 at 3:30 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
I think we can just use load_external_function() to load the library and
call WalReceiverMain from AuxiliaryProcessMain(). Ie. hard-code the
library name. Walreceiver is quite tightly coupled with the
On Wed, Dec 16, 2009 at 6:53 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Great! The logical next step is move the handling of TimelineID and
system identifier out of libpq as well.
All right.
0. Begin by connecting to the master just like a normal backend does. We
don't
Fujii Masao wrote:
On Wed, Dec 16, 2009 at 6:53 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
0. Begin by connecting to the master just like a normal backend does. We
don't necessarily need the new ProtocolVersion code either, though it's
probably still a good idea to
On Thu, Dec 17, 2009 at 9:02 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
And, I think that such backend should switch to walsender mode when the
startup
packet arrives. Otherwise, we would have to authenticate such backend twice
on different context, i.e., a normal
Fujii Masao wrote:
I'm thinking of making the standby send the walsender-switch-code the same
way
as application_name; walreceiver always specifies the option like
replication=on
in conninfo string and calls PQconnectdb(), which sends the code as a part of
startup packet. And, the
On Thu, Dec 17, 2009 at 10:25 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Fujii Masao wrote:
I'm thinking of making the standby send the walsender-switch-code the same
way
as application_name; walreceiver always specifies the option like
replication=on
in conninfo
Fujii Masao wrote:
On Tue, Dec 15, 2009 at 3:47 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Tom Lane wrote:
The very, very large practical problem with this is that if you decide
to change the behavior at any time, the only way to be sure that the WAL
receiver is using
I'm interested in abstracting out features of replication from libpq too. It
would be nice if we could implement different communication bus modules.
For example if you have dozens of replicas you may want to use something
like spread to distribute the records using multicast.
Sorry for top
On Tue, Dec 15, 2009 at 3:47 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Tom Lane wrote:
The very, very large practical problem with this is that if you decide
to change the behavior at any time, the only way to be sure that the WAL
receiver is using the right libpq
On Sat, Dec 12, 2009 at 5:09 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Could we change the API of PQgetXLogData to be more like PQgetCopyData?
I'm thinking of removing the timeout argument, and instead looping with
select/poll and PQconsumeInput in the caller. That
Fujii Masao masao.fu...@gmail.com writes:
On Mon, Dec 14, 2009 at 11:38 AM, Tom Lane t...@sss.pgh.pa.us wrote:
Do we need a new PQgetXLogData function at all? Seems like you could
shove the data through the COPY protocol and not have to touch libpq
at all, rather than duplicating a nontrivial
Fujii Masao masao.fu...@gmail.com writes:
I'm going to set the new function calling ereport as the current notice
processor by using PQsetNoticeProcessor. But the problem is that only the
completed message like NOTICE: xxx is passed to such notice processor,
i.e., the error level itself is not
Tom Lane wrote:
The very, very large practical problem with this is that if you decide
to change the behavior at any time, the only way to be sure that the WAL
receiver is using the right libpq version is to perform a soname major
version bump. The transformations done by libpq will
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
It's going to be a bit more complicated in walsender/walreceiver to work
with the libpq COPY API. We're going to need a WAL sending/receiving
protocol on top of it, defined in terms of rows and columns passed
through the COPY
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
Tom Lane wrote:
Yeah, I have a problem with that. What's the backend doing with libpq?
It's not receiving this data, it's sending it.
walreceiver is a postmaster subprocess too.
Hm. Perhaps it should be a loadable plugin and
Tom Lane wrote:
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
BTW, something that's been bothering me a bit with this patch is that we
now have to link the backend with libpq. I don't see an immediate
problem with that, but I'm not a packager. Does anyone see a problem
with
On Tue, Dec 15, 2009 at 4:11 AM, Tom Lane t...@sss.pgh.pa.us wrote:
Hm. Perhaps it should be a loadable plugin and not hard-linked into the
backend? Compare dblink.
You mean that such plugin is supplied in shared_preload_libraries,
a new process is forked and the shared-memory related to
On Sun, Dec 13, 2009 at 5:42 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Walreceiver wants to wait for data to arrive from the master or a
signal. PQgetXLogData(), which is the libpq function to read a piece of
WAL, takes a timeout argument to support that. Walreceiver
Fujii Masao masao.fu...@gmail.com writes:
On Sun, Dec 13, 2009 at 5:42 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
To implement the timeout in PQgetXLogData(), pqWaitTimed() was changed
to take a timeout instead of finishing_time argument. Which is a mistake
because it
On Mon, Dec 14, 2009 at 11:38 AM, Tom Lane t...@sss.pgh.pa.us wrote:
Do we need a new PQgetXLogData function at all? Seems like you could
shove the data through the COPY protocol and not have to touch libpq
at all, rather than duplicating a nontrivial amount of code there.
Yeah, I also think
Fujii Masao wrote:
On Thu, Dec 10, 2009 at 12:00 AM, Tom Lane t...@sss.pgh.pa.us wrote:
The OS buffer is expected to be able to store a large number of
XLogRecPtr messages, because its size is small. So it's also OK
to just drop it.
It certainly seems to be something we could improve later,
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
Changing the finish_time argument to pqWaitTimed into timeout_ms changes
the behavior connect_timeout option to PQconnectdb. It should wait for
max connect_timeout seconds in total, but now it is waiting for
connect_timeout seconds
Tom Lane wrote:
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
Changing the finish_time argument to pqWaitTimed into timeout_ms changes
the behavior connect_timeout option to PQconnectdb. It should wait for
max connect_timeout seconds in total, but now it is waiting for
Fujii Masao masao.fu...@gmail.com writes:
On Wed, Dec 9, 2009 at 3:58 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
But if everyone is happy with just relying on the OS buffer to not fill
up, let's just drop it.
The OS buffer is expected to be able to store a large number
On Thu, Dec 10, 2009 at 12:00 AM, Tom Lane t...@sss.pgh.pa.us wrote:
The OS buffer is expected to be able to store a large number of
XLogRecPtr messages, because its size is small. So it's also OK
to just drop it.
It certainly seems to be something we could improve later, when and
if
I find the backend libpq changes related to non-blocking I/O quite
complex. Can we find a simpler solution?
The problem we're trying to solve is that while the walsender backend
sends a lot of WAL records to the client, the client can send a lot of
messages to the backend. If volume of the
On Tue, Dec 8, 2009 at 11:23 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
The first question is: do we really need to be prepared for that? The
XLogRecPtr acknowledgment messages the client sends are very small, and
if the client is mindful about not sending them too often
Fujii Masao wrote:
On Tue, Dec 8, 2009 at 11:23 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
If that's deemed not good enough, we could modify just internal_flush()
so that it uses secure_poll to wait for the possibility to either read
or write, instead of blocking for
On Wed, Dec 9, 2009 at 3:58 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
True. Another idea I had was to introduce a callback that backend libpq
can call when the buffer fills. Walsender would set the callback to
ProcessStreamMsgs().
But if everyone is happy with just
67 matches
Mail list logo
