Re: [go-nuts] os/exec stdout bytes.Buffer and timeout behavior?

[Ian Lance Taylor]

On Sun, Oct 1, 2017 at 2:40 PM, roger peppe  wrote:
> One might argue that Run should respect the context deadline even in the
> presence of pipes that don't EOF.

Fair point.

Ian


> On 29 Sep 2017 22:26, "Ian Lance Taylor"  wrote:
>>
>> On Fri, Sep 29, 2017 at 12:33 PM, Alex Buchanan 
>> wrote:
>> >
>> > package main
>> >
>> > import (
>> > "bytes"
>> >   "context"
>> >   "log"
>> >   "os/exec"
>> >   "time"
>> > )
>> >
>> > func main() {
>> > var stdout bytes.Buffer
>> >
>> >   ctx, cancel := context.WithTimeout(context.Background(), time.Second)
>> >   defer cancel()
>> >
>> >   cmd := exec.CommandContext(ctx, "/bin/sh", "-c", "sleep 10; echo foo")
>> >   cmd.Stdout = 
>> >
>> >   err := cmd.Run()
>> >   log.Printf("%s, %s, '%s'", err, ctx.Err(), stdout.String())
>> > }
>> >
>> >
>> > This runs for 10 seconds, but I was expecting 1 second. Can someone help
>> > me
>> > understand what is happening? I think it has something to do with the
>> > stdout
>> > bytes.Buffer?
>>
>> Yes.  The problem is that when the context passed to CommandContext
>> expires, it kills the process started by the command, but does not
>> kill any subprocesses that that command may have started.  So when the
>> context expires the /bin/sh is killed, but the sleep subprocess is
>> still running.  Because you use a bytes.Buffer, the program has
>> created a pipe to capture the standard output of the command.  After
>> the process dies, cmd.Run is waiting for that pipe to be closed to
>> make sure that it gathers all the data.  But the pipe is held open by
>> the sleep subprocess.  It is only after that subprocess exits that the
>> pipe is closed and your program continues.
>>
>> Ian
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] os/exec stdout bytes.Buffer and timeout behavior?

[roger peppe]

One might argue that Run should respect the context deadline even in the
presence of pipes that don't EOF.

On 29 Sep 2017 22:26, "Ian Lance Taylor"  wrote:

> On Fri, Sep 29, 2017 at 12:33 PM, Alex Buchanan 
> wrote:
> >
> > package main
> >
> > import (
> > "bytes"
> >   "context"
> >   "log"
> >   "os/exec"
> >   "time"
> > )
> >
> > func main() {
> > var stdout bytes.Buffer
> >
> >   ctx, cancel := context.WithTimeout(context.Background(), time.Second)
> >   defer cancel()
> >
> >   cmd := exec.CommandContext(ctx, "/bin/sh", "-c", "sleep 10; echo foo")
> >   cmd.Stdout = 
> >
> >   err := cmd.Run()
> >   log.Printf("%s, %s, '%s'", err, ctx.Err(), stdout.String())
> > }
> >
> >
> > This runs for 10 seconds, but I was expecting 1 second. Can someone help
> me
> > understand what is happening? I think it has something to do with the
> stdout
> > bytes.Buffer?
>
> Yes.  The problem is that when the context passed to CommandContext
> expires, it kills the process started by the command, but does not
> kill any subprocesses that that command may have started.  So when the
> context expires the /bin/sh is killed, but the sleep subprocess is
> still running.  Because you use a bytes.Buffer, the program has
> created a pipe to capture the standard output of the command.  After
> the process dies, cmd.Run is waiting for that pipe to be closed to
> make sure that it gathers all the data.  But the pipe is held open by
> the sleep subprocess.  It is only after that subprocess exits that the
> pipe is closed and your program continues.
>
> Ian
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] os/exec stdout bytes.Buffer and timeout behavior?

[Ian Lance Taylor]

On Fri, Sep 29, 2017 at 1:33 PM, Alex Buchanan  wrote:
>
> Ok, that's a great explanation, thanks. Do you have a recommendation for how 
> to exit early? Would StdoutPipe() help? I tried setting Setpgid to true, but 
> probably os/exec doesn't pay attention to process groups.
>
> Alternatively, I can configure my shell scripts to close stdout/err on exit? 
> Not pretty, but might work.
>
> My best approach so far is to use time.After and panic, instead of context. 
> This code is being used in a test, so panic is somewhat acceptable.

I would say that CommandContext is really only useful for simple
programs, not for shell scripts.  So using time.After seems reasonable
to me.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] os/exec stdout bytes.Buffer and timeout behavior?

[Alex Buchanan]

Ok, that's a great explanation, thanks. Do you have a recommendation for how to 
exit early? Would StdoutPipe() help? I tried setting Setpgid to true, but 
probably os/exec doesn't pay attention to process groups.

Alternatively, I can configure my shell scripts to close stdout/err on exit? 
Not pretty, but might work.

My best approach so far is to use time.After and panic, instead of context. 
This code is being used in a test, so panic is somewhat acceptable.

Thanks again.




On 9/29/17, 1:26 PM, "Ian Lance Taylor"  wrote:

>On Fri, Sep 29, 2017 at 12:33 PM, Alex Buchanan  
>wrote:
>>
>> package main
>>
>> import (
>> "bytes"
>>   "context"
>>   "log"
>>   "os/exec"
>>   "time"
>> )
>>
>> func main() {
>> var stdout bytes.Buffer
>>
>>   ctx, cancel := context.WithTimeout(context.Background(), time.Second)
>>   defer cancel()
>>
>>   cmd := exec.CommandContext(ctx, "/bin/sh", "-c", "sleep 10; echo foo")
>>   cmd.Stdout = 
>>
>>   err := cmd.Run()
>>   log.Printf("%s, %s, '%s'", err, ctx.Err(), stdout.String())
>> }
>>
>>
>> This runs for 10 seconds, but I was expecting 1 second. Can someone help me
>> understand what is happening? I think it has something to do with the stdout
>> bytes.Buffer?
>
>Yes.  The problem is that when the context passed to CommandContext
>expires, it kills the process started by the command, but does not
>kill any subprocesses that that command may have started.  So when the
>context expires the /bin/sh is killed, but the sleep subprocess is
>still running.  Because you use a bytes.Buffer, the program has
>created a pipe to capture the standard output of the command.  After
>the process dies, cmd.Run is waiting for that pipe to be closed to
>make sure that it gathers all the data.  But the pipe is held open by
>the sleep subprocess.  It is only after that subprocess exits that the
>pipe is closed and your program continues.
>
>Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] os/exec stdout bytes.Buffer and timeout behavior?

[Ian Lance Taylor]

On Fri, Sep 29, 2017 at 12:33 PM, Alex Buchanan  wrote:
>
> package main
>
> import (
> "bytes"
>   "context"
>   "log"
>   "os/exec"
>   "time"
> )
>
> func main() {
> var stdout bytes.Buffer
>
>   ctx, cancel := context.WithTimeout(context.Background(), time.Second)
>   defer cancel()
>
>   cmd := exec.CommandContext(ctx, "/bin/sh", "-c", "sleep 10; echo foo")
>   cmd.Stdout = 
>
>   err := cmd.Run()
>   log.Printf("%s, %s, '%s'", err, ctx.Err(), stdout.String())
> }
>
>
> This runs for 10 seconds, but I was expecting 1 second. Can someone help me
> understand what is happening? I think it has something to do with the stdout
> bytes.Buffer?

Yes.  The problem is that when the context passed to CommandContext
expires, it kills the process started by the command, but does not
kill any subprocesses that that command may have started.  So when the
context expires the /bin/sh is killed, but the sleep subprocess is
still running.  Because you use a bytes.Buffer, the program has
created a pipe to capture the standard output of the command.  After
the process dies, cmd.Run is waiting for that pipe to be closed to
make sure that it gathers all the data.  But the pipe is held open by
the sleep subprocess.  It is only after that subprocess exits that the
pipe is closed and your program continues.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] os/exec stdout bytes.Buffer and timeout behavior?

[Alex Buchanan]

package main

import (
"bytes"
  "context"
  "log"
  "os/exec"
  "time"
)

func main() {
var stdout bytes.Buffer

  ctx, cancel := context.WithTimeout(context.Background(), time.Second)
  defer cancel()

  cmd := exec.CommandContext(ctx, "/bin/sh", "-c", "sleep 10; echo foo")
  cmd.Stdout = 

  err := cmd.Run()
  log.Printf("%s, %s, '%s'", err, ctx.Err(), stdout.String())
}


This runs for 10 seconds, but I was expecting 1 second. Can someone help me 
understand what is happening? I think it has something to do with the 
stdout bytes.Buffer?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.