Re: [go-nuts] net/http: question about RoundTripper
Thanks Kevin for these insights. It does seem like the documentation
notes were meant for Go core devs. It would have helped, if the
authors threw in more insight.
I have also been using RoundTripper as client middleware, but so far
largely for authentication. I wanted to expand the scope of the client
middleware in my implementations to do more but looking at the
RoundTripper documentation, I wanted to have views on its use and see
if I can find anti-patterns that I should be aware of.
In fact, since I could not find a lot of useful information around it,
I even felt like writing a blog post highlighting good and bad
patterns using RoundTripper based on the notes I collect.
On Tue, May 12, 2020 at 7:38 AM Kevin Conway wrote:
>
> I'll make an attempt to answer this question but I could be wrong given some
> deeper historical context between the early Go developers.
>
> There are two chunks of the http.RoundTripper comments that folks typically
> ask about: "should not attempt to handle higher-level protocol details" and
> "should not modify the request". Unfortunately, the rationale for why these
> statements are made is neither presented in the comments nor in the patches
> that introduced them.
>
> It appears "should not modify the request" most likely refers to a
> concurrency issue that must be mitigated by copying the request before
> mutating it in the RoundTripper. Here's the change that claims to have
> resolved an issue surrounding this: https://codereview.appspot.com/5284041.
> The original comments explicitly allowed for mutating the request but this
> was changed to "should not" after this patch due to the bug that it resolved.
>
> It's a little harder to find direct evident of the author's intent for
> "should not attempt to handle higher-level protocol details". This part of
> the comment has been in the code for nearly a decade and it becomes fairly
> difficult to track the origin past a set of major renames and large movements
> of files from place to place within the early Go source code. Reading
> https://github.com/golang/go/commit/e0a2c5d4b540934e06867710fe7137661a2a39ec
> makes it seem like these notes were meant for the author or for other Go core
> devs who were building the original HTTP stack rather than those of us who
> would use it later. For example, it appears to signal that standard library
> developers should isolate higher level features within the Client type rather
> than in the ClientTransport (now RoundTripper) type. I haven't found
> anything, yet, that suggests the comments are meant for anyone other than
> developers of the http package in the Go standard library.
>
> From a more practical perspective, you don't really have another choice when
> it comes to HTTP client middleware that are generally useful in Go
> applications than the http.RoundTripper interface. If everyone applied the
> "accept interfaces, return structs" guidelines then you would have more
> options. For example, if everything that needed an HTTP client accepted a
> "type Doer { Do(r *http.Request) (*http.Response, Error) }" style interface
> then you could target your middleware as wrappers for the http.Client.
> Unfortunately, most projects that allow for injection of a custom HTTP client
> do so by accepting an instance of *http.Client. Accepting that specific,
> concrete type makes wrapping anything other than the http.RoundTripper a
> practical impossibility.
>
> Personally, I've been using http.RoundTripper middleware for several years
> without issue. It's a solid pattern that can provide an enormous amount of
> value by allowing re-usable layers of behavior that can be injected into
> virtually any library or framework that uses an HTTP client. I don't worry
> about the comments in the standard library for the reasons I listed.
>
> On Mon, May 11, 2020 at 2:01 PM Anuj Agrawal wrote:
>>
>> I am trying to understand in what cases would it make sense to
>> implement my own RoundTripper.
>>
>> If I search about it, I come across examples of RoundTripper that try
>> to do things like caching, retries, authentication, etc. I also read
>> somewhere that there are many RoundTripper implementations that just
>> set the User-Agent header on a request.
>>
>> I know that the documentation says "RoundTrip should not attempt to
>> handle higher-level protocol details such as redirects,
>> authentication, or cookies." And, I also understand that RoundTripper
>> would be a bad place for things like caching.
>>
>> However, I am not able to figure out why is it such a bad idea to use
>> RoundTripper as a middleware that allows me to do so
[go-nuts] net/http: question about RoundTripper
I am trying to understand in what cases would it make sense to implement my own RoundTripper. If I search about it, I come across examples of RoundTripper that try to do things like caching, retries, authentication, etc. I also read somewhere that there are many RoundTripper implementations that just set the User-Agent header on a request. I know that the documentation says "RoundTrip should not attempt to handle higher-level protocol details such as redirects, authentication, or cookies." And, I also understand that RoundTripper would be a bad place for things like caching. However, I am not able to figure out why is it such a bad idea to use RoundTripper as a middleware that allows me to do some of the higher level things like authentication. After all, authentication is just about interpreting and/or manipulating some headers. In some cases, it could be just as good as setting the User-Agent where all that happens is setting the Authorization header with a token. In some other cases, it could mean interpreting the challenge thrown by the server and then making the same call again with a response to the challenge. Can someone please help me understand this better? -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CABnk5p0k77he6c7Pw0APQJF%2B_FDJFOFdJp_BJ8eS66jbw6%3DG1w%40mail.gmail.com.
[go-nuts] http middleware and request context
Hi, I have a situation where I have a chain of middlewares with a logger middleware that comes before an authentication middleware. Logger middleware logs about the incoming request, calls the next middleware in the chain and then logs about the response. Authentication middleware authenticates the incoming request and adds the authenticated user to the request context (which ends up creating a new request object to be passed downstream) before invoking the next middleware. Problem statement is that I want to log the authenticated user id in the logger middleware. But because the request object known to the logger middleware is not the same request object that the authentication middleware set the user context with, I do not have the user information available in the logger middleware. I am deliberating on the ways of achieving this. Right now, I am contemplating to add a pointer to a zero value struct with agreed upon attributes (or with just a context attribute) or a map to the context of the request object in logger middleware or an earlier middleware. Then, in any downstream middleware I can set values of struct attributes (or manipulate the context attribute) or set key value pairs to the map. This makes the "context" being set by downstream middlewares available to upstream middlewares after a downstream middleware returns. Are there better ways of doing it? Hopefully, others in the community may have also faced similar problems and I would like to know about any alternate ways that others may have used. Thanks, Anuj -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CABnk5p0H0YUm2vkL9Nb_CXVpgRy%2BJ87O6NH8FmgR5ewL4%3DSnsg%40mail.gmail.com.
Re: [go-nuts] Re: avoiding os thread creation due to logging
I am not sure if that is indeed a better solution. I am probably better off
delegating the responsibility of writing logs to a dedicated goroutine.
On Sun, Aug 5, 2018 at 10:51 PM, wrote:
> You can use a semaphore-like pattern described in
> https://tip.golang.org/ref/mem to limit the number of goroutines trying
> to concurrently write to the log file. This remove the need for the
> buffering scheme.
>
>
> On Sunday, August 5, 2018 at 9:32:06 AM UTC-7, Anuj Agrawal wrote:
>>
>> I have an application with more than 50k goroutines. I pass a context to
>> each of these goroutines which contains a logger and that logger writes to
>> a file. At some point, the number of goroutines trying to write a log
>> peaked and my application crashed with - runtime: program exceeds
>> 1-thread limitfatal error: thread exhaustion.
>>
>> To provide a quick fix, I added a writer in between the logger and the
>> file writer which would buffer all logs in a channel and a separate
>> goroutine can then read from the channel and write to the file, thus,
>> avoiding having to create so many OS threads. Something like:
>>
>> type writer struct {
>> wg sync.WaitGroup
>> buffer chan []byte
>> outio.WriteCloser
>> pool sync.Pool
>> }
>>
>> func (w *writer) Write(p []byte) (int, error) {
>> tmp := w.pool.Get().([]byte)
>> tmp = append(tmp[:0], p...)
>> w.buffer <- tmp
>> return len(tmp), nil
>> }
>>
>> go func() {
>> for p := range w.buffer {
>> if _, err := w.out.Write(p); err != nil {
>> log.Printf("error writing to log file: %s\n%s\n", err.Error(), string(p))
>> }
>> w.pool.Put(p)
>> }
>> w.wg.Done()
>> }()
>>
>>
>>
>> I want to have suggestions if it can be fixed in a better way.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Re: How to reuse go fix tool code
ch only if they denote the same object. This allows correct
> matching even in the presence of dot imports, named imports and
> locally shadowed package names in the input program.
>
> Matching of type syntax is semantic, not syntactic: type syntax in the
> pattern matches type syntax in the input if the types are identical.
> Thus, func(x int) matches func(y int).
>
> This tool was inspired by other example-based refactoring tools,
> 'gofmt -r' for Go and Refaster for Java.
>
>
> LIMITATIONS
> ===
>
> EXPRESSIVENESS
>
> Only refactorings that replace one expression with another, regardless
> of the expression's context, may be expressed. Refactoring arbitrary
> statements (or sequences of statements) is a less well-defined problem
> and is less amenable to this approach.
>
> A pattern that contains a function literal (and hence statements)
> never matches.
>
> There is no way to generalize over related types, e.g. to express that
> a wildcard may have any integer type, for example.
>
> It is not possible to replace an expression by one of a different
> type, even in contexts where this is legal, such as x in fmt.Print(x).
>
> The struct literals T{x} and T{K: x} cannot both be matched by a single
> template.
>
>
> SAFETY
>
> Verifying that a transformation does not introduce type errors is very
> complex in the general case. An innocuous-looking replacement of one
> constant by another (e.g. 1 to 2) may cause type errors relating to
> array types and indices, for example. The tool performs only very
> superficial checks of type preservation.
>
>
> IMPORTS
>
> Although the matching algorithm is fully aware of scoping rules, the
> replacement algorithm is not, so the replacement code may contain
> incorrect identifier syntax for imported objects if there are dot
> imports, named imports or locally shadowed package names in the input
> program.
>
> Imports are added as needed, but they are not removed as needed.
> Run 'goimports' on the modified file for now.
>
> Dot imports are forbidden in the template.
>
>
> TIPS
>
>
> Sometimes a little creativity is required to implement the desired
> migration. This section lists a few tips and tricks.
>
> To remove the final parameter from a function, temporarily change the
> function signature so that the final parameter is variadic, as this
> allows legal calls both with and without the argument. Then use eg to
> remove the final argument from all callers, and remove the variadic
> parameter by hand. The reverse process can be used to add a final
> parameter.
>
> To add or remove parameters other than the final one, you must do it in
> stages: (1) declare a variant function f' with a different name and the
> desired parameters; (2) use eg to transform calls to f into calls to f',
> changing the arguments as needed; (3) change the declaration of f to
> match f'; (4) use eg to rename f' to f in all calls; (5) delete f'.
>
>
> --thepudds
>
> On Friday, March 16, 2018 at 9:51:58 AM UTC-4, peterGo wrote:
>>
>> Anuj Agrawal,
>>
>> Exporting an API carries a commitment to maintain and support that API.
>> go fix was a special-purpose command, that was useful before the Go1
>> compatibility guarantee. I can see no reason to export a go fix API.
>> Peter
>>
>>
>> On Thursday, March 15, 2018 at 6:20:47 AM UTC-4, Anuj Agrawal wrote:
>>>
>>> Hi,
>>>
>>> I am looking to create a gofix for one of my projects. I thought of
>>> importing the code in
>>> https://github.com/golang/go/tree/master/src/cmd/fix so that I could
>>> simply write a plugin, build a binary and give it to people who import
>>> my code.
>>>
>>> However, I do not see any exported symbols in this code. That leaves
>>> me with only one choice - that I copy the code and then write the
>>> plugin on top of it. That does not sound like a very great idea as
>>> compared to being able to import the package itself.
>>>
>>> Is there any particular reason for not exporting any symbols in the
>>> gofix code?
>>>
>>> Thanks,
>>> Anuj Agrawal
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[go-nuts] avoiding os thread creation due to logging
I have an application with more than 50k goroutines. I pass a context to
each of these goroutines which contains a logger and that logger writes to
a file. At some point, the number of goroutines trying to write a log
peaked and my application crashed with - runtime: program exceeds
1-thread limitfatal error: thread exhaustion.
To provide a quick fix, I added a writer in between the logger and the file
writer which would buffer all logs in a channel and a separate goroutine
can then read from the channel and write to the file, thus, avoiding having
to create so many OS threads. Something like:
type writer struct {
wg sync.WaitGroup
buffer chan []byte
outio.WriteCloser
pool sync.Pool
}
func (w *writer) Write(p []byte) (int, error) {
tmp := w.pool.Get().([]byte)
tmp = append(tmp[:0], p...)
w.buffer <- tmp
return len(tmp), nil
}
go func() {
for p := range w.buffer {
if _, err := w.out.Write(p); err != nil {
log.Printf("error writing to log file: %s\n%s\n", err.Error(), string(p))
}
w.pool.Put(p)
}
w.wg.Done()
}()
I want to have suggestions if it can be fixed in a better way.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[go-nuts] How to reuse go fix tool code
Hi, I am looking to create a gofix for one of my projects. I thought of importing the code in https://github.com/golang/go/tree/master/src/cmd/fix so that I could simply write a plugin, build a binary and give it to people who import my code. However, I do not see any exported symbols in this code. That leaves me with only one choice - that I copy the code and then write the plugin on top of it. That does not sound like a very great idea as compared to being able to import the package itself. Is there any particular reason for not exporting any symbols in the gofix code? Thanks, Anuj Agrawal -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[go-nuts] Re: [mysql] 2017/12/23 11:09:37 packets.go:141: write tcp 127.0.0.1:20630->127.0.0.1:3306: write: broken pipe
I am seeing similar logs even after I add db.SetConnMaxLifetime(10 * time.Second). Test code here: https://play.golang.org/p/itTUXDYv4Ge Any suggestions on how to debug further? On Monday, December 25, 2017 at 12:10:02 PM UTC+5:30, Naoki INADA wrote: > > >> >> To prevent further errors set connection lifetime to value less than >> configured. >> >> >> eg. db.SetConnMaxLifetime(time.Second * 14400) >> >> >> > Thanks, but I (go-sql-driver/mysql maintainer) recommend 10~60 sec > rather than 1/2 of wait_timeout. > > wait_timeout is not the only reason which close TCP connection. > TCP connection may be killed by various reasons. (OS, router, etc...) > > When people ask this question, they doesn't have enough skill to > check all possibilities by themselves. > On the other hand, 10sec is large enough to reduce overhead of > reconnection. > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
