Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-07 Thread Tom Lane
Amit Kapila writes: > On Tue, Jun 6, 2017 at 10:14 PM, Tom Lane wrote: >> By definition, the address range we're trying to reuse worked successfully >> in the postmaster process. I don't see how forcing a specific address >> could do anything but

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-07 Thread Amit Kapila
On Tue, Jun 6, 2017 at 10:14 PM, Tom Lane wrote: > Robert Haas writes: >> I think the idea of retrying process creation (and I definitely agree >> with Tom and Magnus that we have to retry process creation, not just >> individual mappings) is a good

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-06 Thread Robert Haas
On Tue, Jun 6, 2017 at 1:27 PM, Tom Lane wrote: > Robert Haas writes: >> On Tue, Jun 6, 2017 at 12:44 PM, Tom Lane wrote: >>> By definition, the address range we're trying to reuse worked successfully >>> in the postmaster process.

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-06 Thread Tom Lane
Robert Haas writes: > On Tue, Jun 6, 2017 at 12:44 PM, Tom Lane wrote: >> By definition, the address range we're trying to reuse worked successfully >> in the postmaster process. I don't see how forcing a specific address >> could do anything but

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-06 Thread Robert Haas
On Tue, Jun 6, 2017 at 12:44 PM, Tom Lane wrote: > By definition, the address range we're trying to reuse worked successfully > in the postmaster process. I don't see how forcing a specific address > could do anything but create an additional risk of postmaster startup >

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-06 Thread Tom Lane
Robert Haas writes: > I think the idea of retrying process creation (and I definitely agree > with Tom and Magnus that we have to retry process creation, not just > individual mappings) is a good place to start. Now if we find that we > are having to retry frequently, then

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-06 Thread Robert Haas
On Mon, Jun 5, 2017 at 10:10 PM, Amit Kapila wrote: > Agreed. By the way, while browsing about this problem, I found that > one other open source (nginx) has used a solution similar to what > Andres was proposing upthread to solve this problem. Refer: >

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-05 Thread Amit Kapila
On Mon, Jun 5, 2017 at 7:26 PM, Tom Lane wrote: > Amit Kapila writes: >> Sure. I think it is slightly tricky because specs don't say clearly >> how ASLR can impact the behavior of any API and in my last attempt I >> could not reproduce the issue. >

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-05 Thread Tom Lane
Amit Kapila writes: > Sure. I think it is slightly tricky because specs don't say clearly > how ASLR can impact the behavior of any API and in my last attempt I > could not reproduce the issue. > I can try to do basic verification with the patch you have proposed, > but

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-05 Thread Magnus Hagander
On Mon, Jun 5, 2017 at 1:35 PM, Amit Kapila wrote: > On Mon, Jun 5, 2017 at 4:58 PM, Magnus Hagander > wrote: > > On Mon, Jun 5, 2017 at 1:16 PM, Amit Kapila > wrote: > >> > >> On Mon, Jun 5, 2017 at 9:15 AM, Tom Lane

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-05 Thread Amit Kapila
On Mon, Jun 5, 2017 at 4:58 PM, Magnus Hagander wrote: > On Mon, Jun 5, 2017 at 1:16 PM, Amit Kapila wrote: >> >> On Mon, Jun 5, 2017 at 9:15 AM, Tom Lane wrote: >> > Amit Kapila writes: >> > >> >> I

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-05 Thread Magnus Hagander
On Mon, Jun 5, 2017 at 1:16 PM, Amit Kapila wrote: > On Mon, Jun 5, 2017 at 9:15 AM, Tom Lane wrote: > > Amit Kapila writes: > > > >> I think the same problem can happen during reattach as well. > >> Basically,

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-05 Thread Amit Kapila
On Mon, Jun 5, 2017 at 9:15 AM, Tom Lane wrote: > Amit Kapila writes: > >> I think the same problem can happen during reattach as well. >> Basically, MapViewOfFileEx can fail to load image at predefined >> address (UsedShmemSegAddr). > > Once we've

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-04 Thread Tom Lane
Amit Kapila writes: > On Mon, Jun 5, 2017 at 4:00 AM, Tom Lane wrote: >> I took a quick look at this, and it seems rather beside the point. > What I understood from the randomization shm allocation behavior due > to ASLR is that when we try to

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-04 Thread Amit Kapila
On Mon, Jun 5, 2017 at 4:00 AM, Tom Lane wrote: > Amit Kapila writes: >> Okay, I have added the comment to explain the same. I have also >> modified the patch to adjust the looping as per your suggestion. > > I took a quick look at this, and it seems

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-04 Thread Tom Lane
Amit Kapila writes: > Okay, I have added the comment to explain the same. I have also > modified the patch to adjust the looping as per your suggestion. I took a quick look at this, and it seems rather beside the point. You can't just loop inside an already-forked

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-03 Thread Amit Kapila
On Fri, Jun 2, 2017 at 7:20 PM, Petr Jelinek wrote: > On 02/06/17 15:37, Amit Kapila wrote: >> >> No, it is to avoid calling free of memory which is not reserved on >> retry. See the comment: >> + * On the first try, release memory region reservation that was made

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-02 Thread Petr Jelinek
On 02/06/17 15:37, Amit Kapila wrote: > On Thu, Jun 1, 2017 at 10:36 PM, Petr Jelinek > wrote: >> On 01/06/17 15:25, Tom Lane wrote: >>> Robert Haas writes: So, are you going to, perhaps, commit this? Or who is picking this up? >>>

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-02 Thread Amit Kapila
On Thu, Jun 1, 2017 at 10:36 PM, Petr Jelinek wrote: > On 01/06/17 15:25, Tom Lane wrote: >> Robert Haas writes: >>> So, are you going to, perhaps, commit this? Or who is picking this up? >> >>> /me knows precious little about Windows. >> >>

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-02 Thread Noah Misch
On Fri, May 26, 2017 at 05:50:45PM +0530, Amit Kapila wrote: > On Fri, May 26, 2017 at 5:30 AM, Tsunakawa, Takayuki > wrote: > > I guessed that the reason Noah suggested 1 - 5 seconds of retry is based > > on the expectation that the address space might be freed

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-01 Thread Petr Jelinek
On 01/06/17 15:25, Tom Lane wrote: > Robert Haas writes: >> So, are you going to, perhaps, commit this? Or who is picking this up? > >> /me knows precious little about Windows. > > I'm not going to be the one to commit this either, but seems like someone > should. >

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-06-01 Thread Tom Lane
Robert Haas writes: > So, are you going to, perhaps, commit this? Or who is picking this up? > /me knows precious little about Windows. I'm not going to be the one to commit this either, but seems like someone should. regards, tom lane -- Sent

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-31 Thread Robert Haas
On Fri, May 26, 2017 at 10:51 AM, Magnus Hagander wrote: > I would definitely suggest putting it in HEAD (and thus, v10) for a while to > get some real world exposure before backpatching. But if it does work out > well in the end, then we can certainly consider backpatching

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-29 Thread Amit Kapila
On Fri, May 26, 2017 at 8:21 PM, Magnus Hagander wrote: > > On Fri, May 26, 2017 at 8:24 AM, Michael Paquier > wrote: >> >> On Fri, May 26, 2017 at 8:20 AM, Amit Kapila >> wrote: >> > I think the real question here is,

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-26 Thread Magnus Hagander
On Fri, May 26, 2017 at 8:24 AM, Michael Paquier wrote: > On Fri, May 26, 2017 at 8:20 AM, Amit Kapila > wrote: > > I think the real question here is, shall we backpatch this fix or we > > want to do this just in Head or we want to consider it

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-26 Thread Michael Paquier
On Fri, May 26, 2017 at 8:20 AM, Amit Kapila wrote: > I think the real question here is, shall we backpatch this fix or we > want to do this just in Head or we want to consider it as a new > feature for PostgreSQL-11. I think it should be fixed in Head and the > change

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-26 Thread Amit Kapila
On Fri, May 26, 2017 at 5:30 AM, Tsunakawa, Takayuki wrote: > From: pgsql-hackers-ow...@postgresql.org >> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Amit Kapila >> Yes, I also share this opinion, the shm attach failures are due to >> randomization

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-25 Thread Amit Kapila
On Thu, May 25, 2017 at 8:01 PM, Tom Lane wrote: > Amit Kapila writes: >> Yes, I also share this opinion, the shm attach failures are due to >> randomization behavior, so sleep won't help much. So, I will change >> the patch to use 100 retries unless

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-25 Thread Tsunakawa, Takayuki
From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Amit Kapila > Yes, I also share this opinion, the shm attach failures are due to > randomization behavior, so sleep won't help much. So, I will change the > patch to use 100 retries unless people

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-25 Thread Tom Lane
Amit Kapila writes: > Yes, I also share this opinion, the shm attach failures are due to > randomization behavior, so sleep won't help much. So, I will change > the patch to use 100 retries unless people have other opinions. Sounds about right to me.

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-25 Thread Amit Kapila
On Thu, May 25, 2017 at 8:41 AM, Noah Misch wrote: > On Thu, May 25, 2017 at 11:41:19AM +0900, Michael Paquier wrote: > >> Indeed, pgrename() does so with a 100ms sleep time between each >> iteration. Perhaps we could do that and limit to 50 iterations? > > pgrename() is

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-24 Thread Noah Misch
On Thu, May 25, 2017 at 11:41:19AM +0900, Michael Paquier wrote: > On Thu, May 25, 2017 at 11:34 AM, Tsunakawa, Takayuki > wrote: > > From: pgsql-hackers-ow...@postgresql.org > >> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Noah Misch > >> Ten feels

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-24 Thread Michael Paquier
On Thu, May 25, 2017 at 11:34 AM, Tsunakawa, Takayuki wrote: > From: pgsql-hackers-ow...@postgresql.org >> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Noah Misch >> Ten feels low to me. The value should be be low enough so users don't give >> up and

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-24 Thread Tsunakawa, Takayuki
From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Noah Misch > Ten feels low to me. The value should be be low enough so users don't give > up and assume a permanent hang, but there's little advantage to making it > lower. > I'd set it such that

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-24 Thread Noah Misch
On Wed, May 24, 2017 at 09:29:11AM -0400, Michael Paquier wrote: > On Tue, May 23, 2017 at 8:14 AM, Amit Kapila wrote: > > So it seems both you and Tom are leaning towards some sort of retry > > mechanism for shm reattach on Windows. I also think that is a viable > >

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-24 Thread Amit Kapila
On Wed, May 24, 2017 at 6:59 PM, Michael Paquier wrote: > On Tue, May 23, 2017 at 8:14 AM, Amit Kapila wrote: >> So it seems both you and Tom are leaning towards some sort of retry >> mechanism for shm reattach on Windows. I also think that is

Re: retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-24 Thread Michael Paquier
On Tue, May 23, 2017 at 8:14 AM, Amit Kapila wrote: > So it seems both you and Tom are leaning towards some sort of retry > mechanism for shm reattach on Windows. I also think that is a viable > option to negate the impact of ASLR. Attached patch does that. Note >

retry shm attach for windows (WAS: Re: [HACKERS] OK, so culicidae is *still* broken)

2017-05-23 Thread Amit Kapila
On Sat, May 20, 2017 at 5:56 PM, Noah Misch wrote: > On Sat, Apr 15, 2017 at 02:30:18PM -0700, Andres Freund wrote: >> On 2017-04-15 17:24:54 -0400, Tom Lane wrote: >> > Andres Freund writes: >> > > On 2017-04-15 17:09:38 -0400, Tom Lane wrote: >> > >> Why