[go-nuts] Re: How to design HTTP request between HTTP servers

[Van Fury]

OK, if I understand you correctly, the outer for loop is not needed and that
there should be two for loops.
the first for loop does step 1,2 and 3 and then "break" out of the loop 
when valid response is received
and then set the "interval".
The second for loop start a for _ := range ticker.C loop and also break out 
when there is a PATCH failure response.

Thanks for the advice 


 


On Saturday, March 20, 2021 at 12:48:15 PM UTC+2 Brian Candler wrote:

> One other minor point.  When you write
>
> for _ = range ticker.C {
> ...
> }
>
> then you have a timer leak if you break or return from that loop but don't 
> call ticker.Stop().  If you only ever exit by "return" from the function, 
> then you can simply do:
>
> ticker := time.NewTicker(time.Duration(Timer) * time.Second)
> *defer ticker.Stop()*
>
> as per the example here .  
> However since your code could never exit from that loop, it wasn't a 
> problem (and the outer "for { ... }" loop was unnecessary too)
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/7f59c23e-6f07-4498-b56d-09435bff916an%40googlegroups.com.

[go-nuts] Re: How to design HTTP request between HTTP servers

[Van Fury]

Hi Brian,

Thanks for the reply. I intentionally left the error checks out so as to 
make the code short. 
Also using the Client.Do was an error and its suppose to be 
transport.Client.Do.
Also in the main() does other stuff after the PUT and PATCH messages but 
does not have any function that is 
using the data concurrently.

Thanks for the advice on the code structure. I will restructure my code to 
the PuTAndPatch().

BR
Van






On Saturday, March 20, 2021 at 12:04:33 PM UTC+2 Brian Candler wrote:

> *> Any help and comments about my application design*
>
> A few general observations first:
>
> 1. In several places you are ignoring the error status, in particular at 
> json.Unmarshal(bodybytes, ) and the error from transport.Client.Do
> 2. In HandlePUTMessage you are ignoring the HTTP status code
> 3. I can't see why you're using transport.Client.Do in one function, and 
> Client.Do in another
> 4. The "Location" response header, that you read in 
> SendPUTMessageToServerB, should only be set on 3xx (redirect) responses,  
> but you only accept 200 / 204
> 5. Your main function starts a goroutine ("go 
> SendPATCHMessageEveryTimerValue()"), but then immediately exits: 
> https://play.golang.org/p/dLXvxfCaki4 .  If there's no other stuff going 
> on, then just remove the "go".  (But maybe this is because this is an 
> incomplete program fragment, and your real main() does other stuff)
> 6. You have clearly realised that you can't have two different goroutines 
> reading and writing the same variable concurrently.  However the way you've 
> used mutex seems wrong here. Firstly, you're keeping the mutex locked 
> indefinitely (you should only lock it for a short time, while performing an 
> action that touches those variables), and secondly, I don't see any code in 
> the main goroutine which would be touching the same data concurrently - 
> which would also need to be protected by the mutex. But again, this could 
> also be because the program is incomplete.
>
> *> restarting from step 1 when 404 status code is received does not work*
>
> I believe the problem is that you're not returning an "error" status from 
> PATCHMessage under that condition.  You can synthesise one of your own:
>
>} else if status == http.StatusNotFound {
>
> // Here I would like to restart PUT message if status code is 404
>*return fmt.Errorf("Got http.StatusNotFound")*
> }
>
> ... which would then be caught by "if err != nil" in 
> SendPATCHMessageEveryTimerValue.
>
> If you want, you could make your own object (compatible with the "error" 
> interface) to carry the HTTP status as a value:
>
> type HttpError struct {
> Codeint
> Description string
> }
>
> func (e HttpError) Error() string {
> return e.Description
> }
>
> ...
> if status != http.StatusOK {
> return {
> Code:resp.StatusCode,
> Description: fmt.Sprintf("Unexpected status: %s", 
> resp.Status),
> }
> }
>
> This can be useful if the caller wants to see the HTTP status value and 
> act on it:
>
> err := Foo()
> if e, ok := err.(*HttpError); ok && e.Code == http.StatusForbidden 
> {
>  do something
> }
>
> However, personally, I think you should structure the code more closely 
> along the lines of your original description of the problem.
>
> What I mean is: instead of having SendPATCHMessageEveryTimerValue 
> internally call out to SendPUTMessage if there's an error, I'd make it 
> terminate.  The caller is then responsible for going back to step 1 and 
> sending the PUT before sending more PATCHes.
>
> This gives the overall structure something like this:
>
> func PutAndPatch() {
> for {
> var interval time.Duration
> for {
> ... do your step 1, 2, 3
> ... break out when you have the valid PUT response and have 
> set 'interval'
> }
> for {
> ... do your step 4, 5, 6
> ... break out when there is a failure to PATCH
> }
> }
> }
>
> The caller can then choose to run this as a go routine, with "go 
> PutAndPatch()", or not, as they prefer.  Note also that the state ("var 
> interval") is private to this function, so there's no need for a mutex.  
> This would also allow you to run multiple instances of PutAndPatch, talking 
> to different servers, in different goroutines.
>
> Of course, as it stands, if you run PutAndPatch() in a goroutine then it 
> will run autonomously, and the main program will have no idea of its status 
> - and not even have a way to stop it.  If you want to add that sort of 
> functionality, then the cleanest way is likely to be using channels.   For 
> example, the standard way to signal a "stop" is to have an empty channel, 
> which the caller closes when they want the stop to take place; or to use 
> the "context 

Re: [go-nuts] Go routines stuck in runtime_pollwait

[Rahul Yadav]

Any updates, I am also facing this issue in my case Golang server just 
stuck and openFD gets spiked.
-Rahul

On Thursday, August 27, 2020 at 8:40:19 PM UTC+5:30 Siddhesh Divekar wrote:

> Ok, so your data collectors never complete. A simple change to make this 
>> easier to diagnose is to not spin up another collector controller at the 2 
>> min mark if the previous has not completed. 
>>
>> I would determine if the stuck collector is BQ or Elastic and check the 
>> server side logs. 
>>
> We start a data collector (DC) every 2 mins irrespective of whether the 
> previous DC has completed or not, so we would have a few DCs stuck but not 
> all. (2 from backtrace)
>
> I would determine if the stuck collector is BQ or Elastic and check the 
>> server side logs. 
>>
> In server side logs we don't see any activity at all. That is the 
> confusing part why regular user requests are not reaching the go server 
> from the users.
> Will check BQ and elastic logs around that time. But after restart 
> everything was working fine.
>
> Could be a bug in the http/2 implementation. I would disable http/2 and 
>> see if you encounter the problem. 
>>
> If we hit it again we will collect github.com/robaho/goanalyzer & see 
> what is happening.
> Between is there a compile time flag to disable http2 while building ?
>
> Thanks,
>
> On Thu, Aug 27, 2020 at 4:05 AM Robert Engels  
> wrote:
>
>> Could be a bug in the http/2 implementation. I would disable http/2 and 
>> see if you encounter the problem. 
>>
>> On Aug 27, 2020, at 6:02 AM, Robert Engels  wrote:
>>
>> 
>> Ok, so your data collectors never complete. A simple change to make this 
>> easier to diagnose is to not spin up another collector controller at the 2 
>> min mark if the previous has not completed. 
>>
>> I would determine if the stuck collector is BQ or Elastic and check the 
>> server side logs. 
>>
>> On Aug 26, 2020, at 11:29 PM, Kurtis Rader  wrote:
>>
>> 
>> On Wed, Aug 26, 2020 at 8:51 PM Siddhesh Divekar  
>> wrote:
>>
>>> Right, then it looks less likely that we are blocked on a mutex.
>>>
>>> Every 2 minutes we spin up a go routine which then in turn spins up a 
>>> bunch of go routines to collect data from big query & elastic (data 
>>> collector routines).
>>> The 2 minutes go routine then in turn waits on waitgroup for data 
>>> collector routines to come back. So its not single go routine from our side 
>>> at least.
>>> From backtrace we have from sigabort we see only one such data collector 
>>> go routine blocked and other 2 2 mins thread waiting on waitgroup.
>>>
>>
>> Are you spinning up a Go routine every 2 minutes regardless of whether 
>> the previous instance had completed? Your comment is not clear whether you 
>> have one, or more than one, goroutine issuing a BigQuery query at the time 
>> you saw the problem and captured the backtraces. If you are, in fact, 
>> starting those Go routines without regard to whether the prior instance had 
>> completed that seems like a "yellow flag", "here be dragons", situation. 
>>
>> -- 
>> Kurtis Rader
>> Caretaker of the exceptional canines Junior and Hank
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD-y%3DQg_dx5W0qDWiMV9sB0eFToTEoiD83_mbn7RHC%3DQ%3Dg%40mail.gmail.com
>>  
>> 
>> .
>>
>>
>
> -- 
> -Siddhesh.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8a8a4445-7b57-410a-bb33-fa5d603b5f9fn%40googlegroups.com.

Re: [go-nuts] Trying to use a tool in my build (a friction log)

[Manlio Perillo]

Il giorno venerdì 19 marzo 2021 alle 23:20:49 UTC+1 Tim Hockin ha scritto:

> Thanks for feedback, comments below
>
> On Thu, Mar 18, 2021 at 3:35 PM Jay Conrod  wrote:
>
>>
>> > [...] 

>  
>
>> *2. "writing go.mod cache" error messages*
>>
>> This error message should be a lot better. Sorry about that.
>>
>
> I mean, "mkdir /home/thockin/go: not a directory" is pretty concise - it's 
> not a directory (on purpose) because I really don't want Go randomly 
> deciding to write stuff to my homedir (which I suspected was happening, and 
> that's why I made it a non-directory :)
>  
>

If you don't want the go tool to write files to your home directory, you 
can just define a custom GOPATH or GOMODCACHE.  As an example, on my 
sysytem I have set GOPATH to ~/.local/lib/go, and this directory is 
excluded from the backup.

Manlio 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8002edf8-1a9e-44a6-a427-3fbc2b4e1dc1n%40googlegroups.com.

[go-nuts] Re: How to design HTTP request between HTTP servers

[Brian Candler]

On Saturday, 20 March 2021 at 10:04:33 UTC Brian Candler wrote:

> 2. In HandlePUTMessage you are ignoring the HTTP status code
>

My mistake, ignore that one.  You are handling it in the caller, 
SendPUTMessageToServerB.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/7d189520-03a1-4f98-a924-ecd41f8ae2c6n%40googlegroups.com.

[go-nuts] Re: How to design HTTP request between HTTP servers

[Brian Candler]

One other minor point.  When you write

for _ = range ticker.C {
...
}

then you have a timer leak if you break or return from that loop but don't 
call ticker.Stop().  If you only ever exit by "return" from the function, 
then you can simply do:

ticker := time.NewTicker(time.Duration(Timer) * time.Second)
*defer ticker.Stop()*

as per the example here .  However 
since your code could never exit from that loop, it wasn't a problem (and 
the outer "for { ... }" loop was unnecessary too)

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/87fcc13a-21e8-4700-b4ed-b0855505259an%40googlegroups.com.

[go-nuts] Re: How to design HTTP request between HTTP servers

[Brian Candler]

*> Any help and comments about my application design*

A few general observations first:

1. In several places you are ignoring the error status, in particular at 
json.Unmarshal(bodybytes, ) and the error from transport.Client.Do
2. In HandlePUTMessage you are ignoring the HTTP status code
3. I can't see why you're using transport.Client.Do in one function, and 
Client.Do in another
4. The "Location" response header, that you read in 
SendPUTMessageToServerB, should only be set on 3xx (redirect) responses,  
but you only accept 200 / 204
5. Your main function starts a goroutine ("go 
SendPATCHMessageEveryTimerValue()"), but then immediately exits: 
https://play.golang.org/p/dLXvxfCaki4 .  If there's no other stuff going 
on, then just remove the "go".  (But maybe this is because this is an 
incomplete program fragment, and your real main() does other stuff)
6. You have clearly realised that you can't have two different goroutines 
reading and writing the same variable concurrently.  However the way you've 
used mutex seems wrong here. Firstly, you're keeping the mutex locked 
indefinitely (you should only lock it for a short time, while performing an 
action that touches those variables), and secondly, I don't see any code in 
the main goroutine which would be touching the same data concurrently - 
which would also need to be protected by the mutex. But again, this could 
also be because the program is incomplete.

*> restarting from step 1 when 404 status code is received does not work*

I believe the problem is that you're not returning an "error" status from 
PATCHMessage under that condition.  You can synthesise one of your own:

   } else if status == http.StatusNotFound {

// Here I would like to restart PUT message if status code is 404
   *return fmt.Errorf("Got http.StatusNotFound")*
}

... which would then be caught by "if err != nil" in 
SendPATCHMessageEveryTimerValue.

If you want, you could make your own object (compatible with the "error" 
interface) to carry the HTTP status as a value:

type HttpError struct {
Codeint
Description string
}

func (e HttpError) Error() string {
return e.Description
}

...
if status != http.StatusOK {
return {
Code:resp.StatusCode,
Description: fmt.Sprintf("Unexpected status: %s", 
resp.Status),
}
}

This can be useful if the caller wants to see the HTTP status value and act 
on it:

err := Foo()
if e, ok := err.(*HttpError); ok && e.Code == http.StatusForbidden {
 do something
}

However, personally, I think you should structure the code more closely 
along the lines of your original description of the problem.

What I mean is: instead of having SendPATCHMessageEveryTimerValue 
internally call out to SendPUTMessage if there's an error, I'd make it 
terminate.  The caller is then responsible for going back to step 1 and 
sending the PUT before sending more PATCHes.

This gives the overall structure something like this:

func PutAndPatch() {
for {
var interval time.Duration
for {
... do your step 1, 2, 3
... break out when you have the valid PUT response and have set 
'interval'
}
for {
... do your step 4, 5, 6
... break out when there is a failure to PATCH
}
}
}

The caller can then choose to run this as a go routine, with "go 
PutAndPatch()", or not, as they prefer.  Note also that the state ("var 
interval") is private to this function, so there's no need for a mutex.  
This would also allow you to run multiple instances of PutAndPatch, talking 
to different servers, in different goroutines.

Of course, as it stands, if you run PutAndPatch() in a goroutine then it 
will run autonomously, and the main program will have no idea of its status 
- and not even have a way to stop it.  If you want to add that sort of 
functionality, then the cleanest way is likely to be using channels.   For 
example, the standard way to signal a "stop" is to have an empty channel, 
which the caller closes when they want the stop to take place; or to use 
the "context " library which is a wrapper 
around this idea.  For any sort of concurrency it's also well worth getting 
your head around the contents of this video: GopherCon 2018: Bryan C. Mills 
- Rethinking Classical Concurrency Patterns 
.

If you do decide to read and/or modify global state from PutAndPatch(), 
then you'd want to lock the mutex just before accessing it and unlock it 
immediately afterwards - and the main program would need to do the same.  
It's very easy to introduce bugs if you forget to do this.  Use the race 
detector  to help find such problems.

HTH,

Brian.

-- 
You received this message because you 

[go-nuts] How to design HTTP request between HTTP servers

[Afriyie Abraham Kwabena]

Hi,

I have been learning Golang for some few months now but still find it 
difficult to design some applications. I have two HTTP servers A and B and 
would like to do the following
1. server A send an HTTP PUT request with JSON body to server B.
2. If server B is not available, server A retry until server B is up and 
respond to the request
3. In server B response body is a timer value received by server A  
4. Server B uses the timer value (eg 45) to send a PATCH request using the 
value as interval. For example in this case, server A send the PATCH 
request to server B every 45 seconds.
5. Server B respond to the PATCH request with either HTTP status code 204 
or status code 404.
6. If status code is 204, that is OK, server A continues to send the PATCH 
request using the 45 seconds interval.
7. If the response status is 404, server A have to start the process again 
from step 1 again to step 6 when 204 status code is received. 

I have tried to write some code which works but restarting from step 1 when 
404 status code is received does not work. Also am not sure if my 
application design is good enough. Any help and comments about my 
application design. Below is my code  

var (
ContentLocation string
Timer  int32
mu  sync.Mutex
)

func SendPUTMessageToServerB() {

msgbytes, err := ioutil.ReadFile("./msg.json")
...

locProfilebytes, err := json.Marshal(pcfprofile)
  ...
 
locVarNRFUrl := "server B url"

profile, resp, err := HandlePUTMessage(locProfilebytes, locVarNRFUrl)

status := resp.StatusCode

if status == http.StatusOK {
logrus.Println("PUT mesage Update SUCCESS")
} else if status == http.StatusCreated {
logrus.Println("PUT message SUCCESS")
} else {

logrus.Println(fmt.Errorf("Wrong status code returned by server B 
%d", status))
}

ContentLocation = resp.Header.Get("Location")
Timer = profile.TimerValue
}

func HandlePUTMessage(filebyte []byte, VarPath string) (Profile, 
*http.Response, error) {

var (
// Set client and set url
localVarHTTPMethod = http.MethodPut
nfp  Profile
)

req, err := http.NewRequest(localVarHTTPMethod, VarPath, 
bytes.NewBuffer(filebyte))
if err != nil {
logrus.Error(err)
}
req.Close = true
req.Header.Set("Content-Type", "application/json")

backoff := 1
for {
res, err := Client.Do(req) 
if err != nil || res == nil {
logrus.Println("Server A Trying to send PUT request ...")
backoff *= 2
if backoff > 20 {
backoff = 20
}
time.Sleep(time.Duration(backoff) * time.Second)
continue
}

defer func() {
if resCloseErr := res.Body.Close(); resCloseErr != nil {
logrus.Errorf("Response body cannot close: %+v", 
resCloseErr)
}
}()

bodybytes, err := ioutil.ReadAll(res.Body)
//localVarHTTPResponse.Body.Close()
if err != nil {
logrus.Error(err)
return nfp, res, err
}

json.Unmarshal(bodybytes, )
return nfp, res, nil
}
}

// Send PATCH message to server B 
func PATCHMessage() (err error) {
 ...

patchitembytes, err := json.Marshal(patchitem)

...

req, err := http.NewRequest("PATCH", ContentLocation, 
bytes.NewBuffer(patchitembytes))
req.Header.Set("Content-Type", "application/json-patch+json")

response, err := transport.Client.Do(req) // for dev

defer response.Body.Close()

status := response.StatusCode
if status == http.StatusNoContent {

logrus.Info("PATCH message SUCCESS")
} else if status == http.StatusNotFound {

// Here I would like to restart PUT message if status code is 404

logrus.Println("Heart-Beat Message FAILED") 
}
return err
}

// Send a patch message every Timer value
func SendPATCHMessageEveryTimerValue() {

for {
ticker := time.NewTicker(time.Duration(Timer) * time.Second)
mu.Lock()

for _ = range ticker.C {
err := PATCHMessage()
if err != nil {
SendPUTMessage()
}
}
mu.Unlock()
}
}


func main() {

SendPUTMessageToServerB()
go SendPATCHMessageEveryTimerValue()
}



-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/a2481944-2a5c-467b-ba39-6e355fe219aan%40googlegroups.com.