[go-nuts] Re: best practices of client middleware

[Sangjin Lee]

I offered a PR on alice (which is a middleware library for the handlers) 
that does that. It's essentially an application of the same pattern. We're 
getting a lot of mileage for this.

https://github.com/justinas/alice/pull/40

On Tuesday, March 6, 2018 at 6:41:53 AM UTC-8, Eyal Posener wrote:
>
> Good stuff,
> So I see that this library wraps the http.Client and doesn't use the 
> roundtripper.
> Pretty elegant!
>
> I still have two questions about the standard library:
>
>- Didn't understand yet why it is not allowed to add headers in the 
>roundtripper.
>- Is this a bit strange that the standard library provides a beautiful 
>way to have server middleware, but no way to have client middlewares?
>
> Cheers,
> Eyal
>
> On Tuesday, March 6, 2018 at 2:57:57 AM UTC+2, Bojan Delić wrote:
>>
>> As far as I am aware, there is very little information about best 
>> practices regarding client side middlewares. 
>>
>> I though that having such support is neat idea and I have implemented 
>> something (that I use for some time now, though still in beta) that you 
>> might find useful for your use case. I have written small library that 
>> describes client middleware protocol , 
>> some useful middlewares  
>> and HTTP client  that's using these 
>> libraries. 
>>
>> This might solve your problem directly (writing new middleware is 
>> trivial), but it might introduce dependency that you don't want, so I hope 
>> this will provide inspiration on how you would do something similar 
>> yourself. 
>>
>> On Monday, March 5, 2018 at 3:03:14 PM UTC+1, Eyal Posener wrote:
>>>
>>> Hi
>>>
>>> I want to implement a client middleware - for example: sign the request 
>>> body and add the signature as an HTTP header.
>>> I thought that the best way to do it is to implement a RoundTripper 
>>> interface  which up on 
>>> request, calculate the signature, adds the header, and invoke a "next" 
>>> ToundTripper.
>>>
>>> This could be a very generic implementation, which i can inject to any 
>>> client that uses the standard library http.Client.
>>>
>>> However, I found the following in the doc:
>>>
>>> // RoundTrip should not modify the request, except for
>>> // consuming and closing the Request's Body. RoundTrip may
>>> // read fields of the request in a separate goroutine. Callers
>>> // should not mutate the request until the Response's Body has
>>> // been closed.
>>>
>>>
>>> Is there a standard way to do it?
>>>
>>> Thanks,
>>> Eyal
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] dependency repo internal mirroring/hosting

[Sangjin Lee]

What do folks do in terms of managing/vetting dependencies and also hosting 
them internally? In other language ecosystems, there are tools such as 
Nexus or Artifactory that work as caching mirrors for binary dependencies. 
That way, a company or a team can have a stronger control/governance on 
dependencies they use. I'm curious if there is a mature solution for Go.

One strawman idea is to set up an internal git (or github) repo that acts 
as a repository of dependencies, and have our dep toml point to that 
internal repo for each and every dependency. But it is not quite an 
automated solution to the problem. I'm curious what others are doing for 
this.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: understanding memory profile

[Sangjin Lee]

It is actually with the latest (1.9.2). I'll file an issue.

Thanks.

On Tuesday, December 5, 2017 at 1:51:07 PM UTC-8, Dave Cheney wrote:
>
> Can you please check if this is happening with the current version of Go 
> and if so raise a bug, https://golang.org/issue/new. 
>
> Thanks 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: understanding memory profile

[Sangjin Lee]

Mystery solved. It turns out it was coming from bytes.NewBuffer which was 
being invoked in the method I listed. Somehow that was attributed to the 
import line for bytes rather than the line of invocation.

On Tuesday, December 5, 2017 at 10:51:33 AM UTC-8, Dave Cheney wrote:
>
> No idea. Which version of Go are you using?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: understanding memory profile

[Sangjin Lee]

Thanks for the pointer! I am still puzzled by the former (allocation 
associated with the import line). I thought it could be an init() function 
in this file, but that does not explain it...

On Monday, December 4, 2017 at 5:47:09 PM UTC-8, Dave Cheney wrote:
>
>
>
> On Tuesday, 5 December 2017 12:09:54 UTC+11, Sangjin Lee wrote:
>>
>> A couple of questions on understanding go pprof's memory profile...
>>
>> When I list a certain function (with --alloc_objects), I see an 
>> allocation number associated with an import line mysteriously. I am pretty 
>> certain I can rule out any source line offset. For example,
>>
>> (pprof) list foo
>> Total: 534804146
>> ROUTINE  (snip)
>>   24057960   27540717 (flat, cum)  5.15% of Total
>>  .  .  1:package protocol
>>  .  .  2:
>>  .  .  3:import (
>>   12092676   12092676  4:   "bytes"
>>  .  .  5:   "fmt"
>>  .  .  6:   "net/url"
>>  .  .  7:   "strconv"
>>  .  .  8:   "strings"
>>  .  .  9:   "time"
>>  .  . 10:   "unicode"
>>  .  . 11:)
>>
>> Any idea how to interpret this?
>>
>> In addition, is there a way to drill down further into exactly what type 
>> of memory allocation is contributing to the allocation when I list a 
>> function? The tree/web/pdf view doesn't break down further function names 
>> (only shows the boxes with what I believe is the individual allocation 
>> size). I'm kind of guessing from the line of code on what it is, but it'd 
>> be great if there is a way to track down exactly what type of memory 
>> allocation it is (string copy, map growth, byte allocation, etc.). For 
>> example,
>>
>> (pprof) list add
>> Total: 534804146
>> ROUTINE  (snip)
>>   17248777   17248777 (flat, cum)  3.23% of Total
>>  .  .220: l *logger
>>  .  .221:}
>>  .  .222:
>>  .  .223:func (l *logger) add(data 
>> map[string]interface{}) {
>>  .  .224: if id := l.Id(); id != "" {
>>   17248777   17248777225: data[ID] = id
>>  .  .226: }
>>  .  .227:}
>>  .  .228:
>>
>> where id is a string.
>>
>
> The allocations are taking the address of the value of id when it is boxed 
> into an interface{} 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] understanding memory profile

[Sangjin Lee]

A couple of questions on understanding go pprof's memory profile...

When I list a certain function (with --alloc_objects), I see an allocation 
number associated with an import line mysteriously. I am pretty certain I 
can rule out any source line offset. For example,

(pprof) list foo
Total: 534804146
ROUTINE  (snip)
  24057960   27540717 (flat, cum)  5.15% of Total
 .  .  1:package protocol
 .  .  2:
 .  .  3:import (
  12092676   12092676  4:   "bytes"
 .  .  5:   "fmt"
 .  .  6:   "net/url"
 .  .  7:   "strconv"
 .  .  8:   "strings"
 .  .  9:   "time"
 .  . 10:   "unicode"
 .  . 11:)

Any idea how to interpret this?

In addition, is there a way to drill down further into exactly what type of 
memory allocation is contributing to the allocation when I list a function? 
The tree/web/pdf view doesn't break down further function names (only shows 
the boxes with what I believe is the individual allocation size). I'm kind 
of guessing from the line of code on what it is, but it'd be great if there 
is a way to track down exactly what type of memory allocation it is (string 
copy, map growth, byte allocation, etc.). For example,

(pprof) list add
Total: 534804146
ROUTINE  (snip)
  17248777   17248777 (flat, cum)  3.23% of Total
 .  .220: l *logger
 .  .221:}
 .  .222:
 .  .223:func (l *logger) add(data 
map[string]interface{}) {
 .  .224: if id := l.Id(); id != "" {
  17248777   17248777225: data[ID] = id
 .  .226: }
 .  .227:}
 .  .228:

where id is a string.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] http.Transport: separating the connection pooling aspect

[Sangjin Lee]

We enjoy using http.Transport as it gives us a real solid HTTP client 
functionality and faithful protocol implementation. The connection 
pooling/management is also bundled into http.Transport. That's the part I'd 
like to discuss. The http.Transport connection management takes a stance on 
a few areas. For example,

- it does not limit the number of *active* connections
- it reuses available connections in a LIFO manner

There are real needs and use cases where we need a different behavior 
there. We may want to limit the number of active connections. We may want 
to have a different connection pooling policy (e.g. FIFO). But today it is 
not possible if you use http.Transport. The only option is to implement the 
HTTP client, but we like the protocol implementation that exists in 
http.Transport.

This is probably more of a go 2 suggestion, but it would be great if these 
policies are configurable first of all so that we can realize different 
policies than the current hard-coded ones. Even better, it would be awesome 
if the connection management aspect of http.Transport is separated from the 
protocol aspect of http.Transport and become pluggable. Then we could 
choose to provide a different connection management implementation and mix 
it with the protocol support of http.Transport.

Thoughts?

Regards,
Sangjin

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: bufio.Writer and timeouts

[Sangjin Lee]

Hmm maybe I misunderstood your use case. I thought you were basically
setting a fixed write duration via SetWriteDeadline() before each Write()
call, in which case the same treatment probably should be done for Flush().


Sangjin

On Thu, Sep 14, 2017 at 3:33 PM, Juliusz Chroboczek  wrote:

> > Is extending the deadline right before flushing an option?
>
> No.  Before the Flush (or the Write) times out, I don't know that this
> particular peer is congested, and hence I have no reason to reschedule
> requests queued to this peer on a different connection.
>
> -- Juliusz
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/to
> pic/golang-nuts/y0lHtwaeeQk/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: bufio.Writer and timeouts

[Sangjin Lee]

Is extending the deadline right before flushing an option? If your deadline 
is based on a relative duration, you could apply the same delay on Flush() 
too because Flush() may incur write, no?

On Saturday, September 9, 2017 at 8:17:24 AM UTC-7, Juliusz Chroboczek 
wrote:
>
> > bufio.Writer is about 200 lines, and you probably don't even need all 
> > of them.  Just copy it and modify it for your purposes. 
>
> Ok, thanks. 
>
> -- Juliusz 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] fast infoset encoding handler lib in go?

[Sangjin Lee]

Is there a fast infoset encoding handler in go out there that can transform 
it to and from a more textual (e.g. xml) format? I'm referring to the fast 
infoset format that is used for some (old style) SOAP communication (see 
https://en.wikipedia.org/wiki/Fast_Infoset). Google search doesn't turn up 
much... Thanks!

Sangjin

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.