Re: [HACKERS] Backup throttling

2014-02-28 Thread Antonin Houska
On 02/27/2014 11:04 PM, Alvaro Herrera wrote: I pushed this patch with a few further tweaks. In your changes to address the above point, you made the suffix mandatory in the pg_basebackup -r option. This seemed a strange restriction, so I removed it. It seems more user-friendly to me to

Re: [HACKERS] Backup throttling

2014-02-27 Thread Alvaro Herrera
Antonin Houska escribió: Why did you choose bytes per second as a valid rate which we can specify? Since the minimum rate is 32kB, isn't it better to use KB per second for that? If we do that, we can easily increase the maximum rate from 1GB to very large number in the future if

Re: [HACKERS] Backup throttling

2014-02-03 Thread Antonin Houska
On 01/31/2014 06:26 AM, Fujii Masao wrote: Is there a good place to define the constant, so that both backend and client can use it? I'd say 'include/common' but no existing file seems to be appropriate. I'm not sure if it's worth to add a new file there. If there is no good place to define

Re: [HACKERS] Backup throttling

2014-01-30 Thread Fujii Masao
On Tue, Jan 21, 2014 at 1:10 AM, Antonin Houska antonin.hou...@gmail.com wrote: On 01/15/2014 10:52 PM, Alvaro Herrera wrote: I gave this patch a look. There was a bug that the final bounds check for int32 range was not done when there was no suffix, so in effect you could pass numbers larger

Re: [HACKERS] Backup throttling

2014-01-21 Thread Antonin Houska
I realize the following should be applied on the top of v7: index a0216c1..16dd939 100644 --- a/src/backend/replication/basebackup.c +++ b/src/backend/replication/basebackup.c @@ -1263,7 +1263,7 @@ throttle(size_t increment) throttling_counter %= throttling_sample; /* Once the

Re: [HACKERS] Backup throttling

2014-01-20 Thread Antonin Houska
On 01/15/2014 10:52 PM, Alvaro Herrera wrote: I gave this patch a look. There was a bug that the final bounds check for int32 range was not done when there was no suffix, so in effect you could pass numbers larger than UINT_MAX and pg_basebackup would not complain until the number reached the

Re: [HACKERS] Backup throttling

2014-01-16 Thread Andres Freund
Hi, On 2014-01-15 18:52:32 -0300, Alvaro Herrera wrote: Another thing I found a bit strange was the use of the latch. What this patch does is create a separate latch which is used for the throttling. This means that if the walsender process receives a signal, it will not wake up if it's

Re: [HACKERS] Backup throttling

2014-01-16 Thread Peter Geoghegan
On Thu, Jan 16, 2014 at 12:03 PM, Andres Freund and...@2ndquadrant.com wrote: slightly related: we should start to reuse procLatch for walsenders instead of having a separate latch someday. +1. The potential for bugs from failing to account for this within signal handlers seems like a concern.

Re: [HACKERS] Backup throttling

2014-01-16 Thread Michael Paquier
On Fri, Jan 17, 2014 at 5:03 AM, Andres Freund and...@2ndquadrant.com wrote: slightly related: we should start to reuse procLatch for walsenders instead of having a separate latch someday. + 1 on that. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] Backup throttling

2014-01-15 Thread Alvaro Herrera
Alvaro Herrera escribió: Antonin Houska escribió: Thanks for checking. The new version addresses your findings. I gave this patch a look. BTW I also moved the patch the commitfest currently running, and set it waiting-on-author. Your move. -- Álvaro Herrera

Re: [HACKERS] Backup throttling

2013-12-10 Thread Antonin Houska
Thanks for checking. The new version addresses your findings. // Antonin Houska (Tony) On 12/09/2013 03:49 PM, Fujii Masao wrote: On Fri, Dec 6, 2013 at 6:43 PM, Boszormenyi Zoltan z...@cybertec.at wrote: Hi, 2013-12-05 15:36 keltezéssel, Antonin Houska írta: On 12/02/2013 02:23 PM,

Re: [HACKERS] Backup throttling

2013-12-09 Thread Fujii Masao
On Fri, Dec 6, 2013 at 6:43 PM, Boszormenyi Zoltan z...@cybertec.at wrote: Hi, 2013-12-05 15:36 keltezéssel, Antonin Houska írta: On 12/02/2013 02:23 PM, Boszormenyi Zoltan wrote: Hi, I am reviewing your patch. Thanks. New version attached. I have reviewed and tested it and marked it

Re: [HACKERS] Backup throttling

2013-12-06 Thread Boszormenyi Zoltan
Hi, 2013-12-05 15:36 keltezéssel, Antonin Houska írta: On 12/02/2013 02:23 PM, Boszormenyi Zoltan wrote: Hi, I am reviewing your patch. Thanks. New version attached. I have reviewed and tested it and marked it as ready for committer. Best regards, Zoltán Böszörményi --

Re: [HACKERS] Backup throttling

2013-12-05 Thread Antonin Houska
On 12/02/2013 02:23 PM, Boszormenyi Zoltan wrote: Hi, I am reviewing your patch. Thanks. New version attached. * Does it follow the project coding guidelines? Yes. A nitpicking: this else branch below might need brackets because there is also a comment in that branch: +

Re: [HACKERS] Backup throttling

2013-12-02 Thread Boszormenyi Zoltan
Hi, I am reviewing your patch. 2013-10-10 15:32 keltezéssel, Antonin Houska írta: On 10/09/2013 08:56 PM, Robert Haas wrote: There seem to be several review comments made since you posted this version. I'll mark this Waiting on Author in the CommitFest application, since it seems that the

Re: [HACKERS] Backup throttling

2013-10-10 Thread Antonin Houska
On 10/09/2013 08:56 PM, Robert Haas wrote: There seem to be several review comments made since you posted this version. I'll mark this Waiting on Author in the CommitFest application, since it seems that the patch needs to be further updated. Since it was waiting for reviewer, I was not sure

Re: [HACKERS] Backup throttling

2013-10-09 Thread Robert Haas
On Tue, Sep 3, 2013 at 8:35 AM, Antonin Houska antonin.hou...@gmail.com wrote: On 07/24/2013 09:20 AM, Antonin Houska wrote: Hello, the purpose of this patch is to limit impact of pg_backup on running server. Attached is a new version. Server-side implementation this time. Antonin Houska

Re: [HACKERS] Backup throttling

2013-09-03 Thread Antonin Houska
On 07/24/2013 09:20 AM, Antonin Houska wrote: Hello, the purpose of this patch is to limit impact of pg_backup on running server. Attached is a new version. Server-side implementation this time. Antonin Houska (Tony) diff --git a/doc/src/sgml/ref/pg_basebackup.sgml

Re: [HACKERS] Backup throttling

2013-09-03 Thread Andres Freund
Hi, On 2013-09-03 14:35:18 +0200, Antonin Houska wrote: + /* + * THROTTLING_SAMPLE_MIN / MAX_RATE_LOWER (in seconds) should be the + * longest possible time to sleep. + */ + pg_usleep((long) sleep); + else + +

Re: [HACKERS] Backup throttling

2013-09-03 Thread Alvaro Herrera
Antonin Houska wrote: + para +Suffixes literalk/literal (kilobytes) and literalm/literal +(megabytes) are accepted. For example: literal10m/literal + /para m is for meters, or milli. Please use M here. +static uint32 +parse_max_rate(char *src) +{ + int

Re: [HACKERS] Backup throttling

2013-09-03 Thread Antonin Houska
On 09/03/2013 06:56 PM, Alvaro Herrera wrote: +/* + * Only the following suffixes are allowed. It's not too useful to + * restrict the rate to gigabytes: such a rate will probably bring + * significant impact

Re: [HACKERS] Backup throttling

2013-09-03 Thread Andres Freund
On 2013-09-03 12:56:53 -0400, Alvaro Herrera wrote: Antonin Houska wrote: + para +Suffixes literalk/literal (kilobytes) and literalm/literal +(megabytes) are accepted. For example: literal10m/literal + /para m is for meters, or milli. Please use M here.

Re: [HACKERS] Backup throttling

2013-08-27 Thread Craig Ringer
On 08/27/2013 01:56 AM, Antonin Houska wrote: However what you stress now is control of the (continuous) WAL stream and thus something that affects normal operation, rather than setup. I still think the pg_basebackup does not have to throttle the WAL stream, so this your request does not

Re: [HACKERS] Backup throttling

2013-08-27 Thread Robert Haas
On Tue, Aug 20, 2013 at 2:37 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Throttling in the client seems much better to me. TCP is designed to handle a slow client. Other people have already offered some good points in this area, but let me just add one thought that I don't think has

Re: [HACKERS] Backup throttling

2013-08-27 Thread Benedikt Grundmann
On Tue, Aug 27, 2013 at 12:58 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Aug 20, 2013 at 2:37 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Throttling in the client seems much better to me. TCP is designed to handle a slow client. Other people have already offered some

Re: [HACKERS] Backup throttling

2013-08-27 Thread Antonin Houska
On 08/27/2013 01:58 PM, Robert Haas wrote: On Tue, Aug 20, 2013 at 2:37 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Throttling in the client seems much better to me. TCP is designed to handle a slow client. Other people have already offered some good points in this area, but let

Re: [HACKERS] Backup throttling

2013-08-27 Thread Greg Smith
On 8/27/13 7:58 AM, Robert Haas wrote: We have a *general* need to be able to throttle server-side resource utilization, particularly I/O. This is a problem not only for pg_basebackup, but for COPY, CLUSTER, VACUUM, and even things like UPDATE. Of all of those, the only one for which we

Re: [HACKERS] Backup throttling

2013-08-26 Thread Antonin Houska
On 08/22/2013 03:33 PM, Craig Ringer wrote: On 08/22/2013 01:39 PM, PostgreSQL - Hans-Jürgen Schönig wrote: what would be a reasonable scenario where limiting streaming would make sense? i cannot think of any to be honest. I tend to agree. If anything we're likely to want the reverse -

Re: [HACKERS] Backup throttling

2013-08-26 Thread Hannu Krosing
On 08/26/2013 12:50 PM, Antonin Houska wrote: On 08/22/2013 03:33 PM, Craig Ringer wrote: On 08/22/2013 01:39 PM, PostgreSQL - Hans-Jürgen Schönig wrote: what would be a reasonable scenario where limiting streaming would make sense? i cannot think of any to be honest. I tend to agree. If

Re: [HACKERS] Backup throttling

2013-08-26 Thread Craig Ringer
On 08/26/2013 08:15 PM, Hannu Krosing wrote: On 08/26/2013 12:50 PM, Antonin Houska wrote: On 08/22/2013 03:33 PM, Craig Ringer wrote: On 08/22/2013 01:39 PM, PostgreSQL - Hans-Jürgen Schönig wrote: what would be a reasonable scenario where limiting streaming would make sense? i cannot

Re: [HACKERS] Backup throttling

2013-08-26 Thread Antonin Houska
On 08/26/2013 02:33 PM, Craig Ringer wrote: On 08/26/2013 08:15 PM, Hannu Krosing wrote: On 08/26/2013 12:50 PM, Antonin Houska wrote: On 08/22/2013 03:33 PM, Craig Ringer wrote: On 08/22/2013 01:39 PM, PostgreSQL - Hans-Jürgen Schönig wrote: what would be a reasonable scenario where

Re: [HACKERS] Backup throttling

2013-08-24 Thread Peter Eisentraut
On Wed, 2013-07-24 at 09:20 +0200, Antonin Houska wrote: the purpose of this patch is to limit impact of pg_backup on running server. Feedback is appreciated. Please replace the tabs in the SGML files with spaces. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] Backup throttling

2013-08-22 Thread Craig Ringer
On 08/22/2013 01:39 PM, PostgreSQL - Hans-Jürgen Schönig wrote: what would be a reasonable scenario where limiting streaming would make sense? i cannot think of any to be honest. I tend to agree. If anything we're likely to want the reverse - the ability to throttle WAL *generation* on the

Re: [HACKERS] Backup throttling

2013-08-22 Thread Andres Freund
On 2013-08-22 07:39:41 +0200, PostgreSQL - Hans-Jürgen Schönig wrote: regarding the client side implementation: we have chosen this way because it is less invasive. i cannot see a reason to do this on the server side because we won't have 10 pg_basebackup-style tools making use of

Re: [HACKERS] Backup throttling

2013-08-21 Thread PostgreSQL - Hans-Jürgen Schönig
On Aug 19, 2013, at 9:11 PM, Andres Freund wrote: On 2013-08-19 20:15:51 +0200, Boszormenyi Zoltan wrote: 2013-08-19 19:20 keltezéssel, Andres Freund írta: Hi, On 2013-07-24 09:20:52 +0200, Antonin Houska wrote: Hello, the purpose of this patch is to limit impact of pg_backup on running

Re: [HACKERS] Backup throttling

2013-08-21 Thread Andres Freund
On 2013-08-21 08:10:42 +0200, PostgreSQL - Hans-Jürgen Schönig wrote: On Aug 19, 2013, at 9:11 PM, Andres Freund wrote: On 2013-08-19 20:15:51 +0200, Boszormenyi Zoltan wrote: 2013-08-19 19:20 keltezéssel, Andres Freund írta: Hi, On 2013-07-24 09:20:52 +0200, Antonin Houska wrote:

Re: [HACKERS] Backup throttling

2013-08-21 Thread PostgreSQL - Hans-Jürgen Schönig
On Aug 21, 2013, at 10:57 AM, Andres Freund wrote: On 2013-08-21 08:10:42 +0200, PostgreSQL - Hans-Jürgen Schönig wrote: On Aug 19, 2013, at 9:11 PM, Andres Freund wrote: On 2013-08-19 20:15:51 +0200, Boszormenyi Zoltan wrote: 2013-08-19 19:20 keltezéssel, Andres Freund írta: Hi, On

Re: [HACKERS] Backup throttling

2013-08-20 Thread Heikki Linnakangas
On 19.08.2013 21:15, Boszormenyi Zoltan wrote: 2013-08-19 19:20 keltezéssel, Andres Freund írta: Based on a quick look it seems like you're throttling on the receiving side. Is that a good idea? Especially over longer latency links, TCP buffering will reduce the effect on the sender side

Re: [HACKERS] Backup throttling

2013-08-20 Thread Boszormenyi Zoltan
2013-08-20 08:37 keltezéssel, Heikki Linnakangas írta: On 19.08.2013 21:15, Boszormenyi Zoltan wrote: 2013-08-19 19:20 keltezéssel, Andres Freund írta: Based on a quick look it seems like you're throttling on the receiving side. Is that a good idea? Especially over longer latency links, TCP

Re: [HACKERS] Backup throttling

2013-08-19 Thread Boszormenyi Zoltan
2013-07-31 22:50 keltezéssel, Antonin Houska írta: On 07/31/2013 07:13 AM, Gibheer wrote: Hi, That is a really nice feature. I don't pretend it's my idea, I just coded it. My boss proposed the feature as such :-) I took a first look at your patch and some empty lines you added (e.g. line 60

Re: [HACKERS] Backup throttling

2013-08-19 Thread Andres Freund
Hi, On 2013-07-24 09:20:52 +0200, Antonin Houska wrote: Hello, the purpose of this patch is to limit impact of pg_backup on running server. Feedback is appreciated. Based on a quick look it seems like you're throttling on the receiving side. Is that a good idea? Especially over longer latency

Re: [HACKERS] Backup throttling

2013-08-19 Thread Boszormenyi Zoltan
2013-08-19 19:20 keltezéssel, Andres Freund írta: Hi, On 2013-07-24 09:20:52 +0200, Antonin Houska wrote: Hello, the purpose of this patch is to limit impact of pg_backup on running server. Feedback is appreciated. Based on a quick look it seems like you're throttling on the receiving side.

Re: [HACKERS] Backup throttling

2013-08-19 Thread Andres Freund
On 2013-08-19 20:15:51 +0200, Boszormenyi Zoltan wrote: 2013-08-19 19:20 keltezéssel, Andres Freund írta: Hi, On 2013-07-24 09:20:52 +0200, Antonin Houska wrote: Hello, the purpose of this patch is to limit impact of pg_backup on running server. Feedback is appreciated. Based on a quick

Re: [HACKERS] Backup throttling

2013-08-19 Thread Boszormenyi Zoltan
2013-08-19 21:11 keltezéssel, Andres Freund írta: On 2013-08-19 20:15:51 +0200, Boszormenyi Zoltan wrote: 2013-08-19 19:20 keltezéssel, Andres Freund írta: Hi, On 2013-07-24 09:20:52 +0200, Antonin Houska wrote: Hello, the purpose of this patch is to limit impact of pg_backup on running

Re: [HACKERS] Backup throttling

2013-07-31 Thread Alvaro Herrera
Gibheer escribió: Why did you move localGetCurrentTimestamp() into streamutil.c? Is sys/time.h still needed in receivelog.c after the move? I think we should have GetCurrentTimestamp() in src/common; there are way too many copies of that functionality now. I looked into this awhile back, but

Re: [HACKERS] Backup throttling

2013-07-31 Thread Antonin Houska
On 07/31/2013 07:13 AM, Gibheer wrote: Hi, That is a really nice feature. I don't pretend it's my idea, I just coded it. My boss proposed the feature as such :-) I took a first look at your patch and some empty lines you added (e.g. line 60 your patch). Can you remove them? Sure, will do in

Re: [HACKERS] Backup throttling

2013-07-31 Thread Gibheer
On Wed, 31 Jul 2013 22:50:19 +0200 Antonin Houska antonin.hou...@gmail.com wrote: On 07/31/2013 07:13 AM, Gibheer wrote: Hi, That is a really nice feature. I don't pretend it's my idea, I just coded it. My boss proposed the feature as such :-) I took a first look at your patch and

Re: [HACKERS] Backup throttling

2013-07-30 Thread Gibheer
On Wed, 24 Jul 2013 09:20:52 +0200 Antonin Houska antonin.hou...@gmail.com wrote: Hello, the purpose of this patch is to limit impact of pg_backup on running server. Feedback is appreciated. // Antonin Houska (Tony) Hi, That is a really nice feature. I took a first look at your patch and

[HACKERS] Backup throttling

2013-07-24 Thread Antonin Houska
Hello, the purpose of this patch is to limit impact of pg_backup on running server. Feedback is appreciated. // Antonin Houska (Tony) diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml index eb0c1d6..3b7ecfd 100644 --- a/doc/src/sgml/ref/pg_basebackup.sgml

Re: [HACKERS] Backup throttling

2013-07-24 Thread Fujii Masao
On Wed, Jul 24, 2013 at 4:20 PM, Antonin Houska antonin.hou...@gmail.com wrote: Hello, the purpose of this patch is to limit impact of pg_backup on running server. Feedback is appreciated. Interesting. Please add this patch to the next commitfeat.