Re: Patch 8.0.0105
Bram Moolenaar wrote: > Skywind wrote: > > > Bram Moolenaar wrote: > > > > > You can improve this a lot by changing: > > > > > >if ch_canread(ch) > > > let text = ch_read(ch, {'timeout':0}) > > > caddexpr text > > > cbottom > > > endif > > > > > > To: > > > > > >if ch_canread(ch) > > > while ch_canread(ch) > > > let text = ch_read(ch, {'timeout':0}) > > > caddexpr text > > > endwhile > > > cbottom > > > endif > > > > Yes, I intentionally do so to make reading really slower than writing. > > > > > But, you might still miss some messages if the job exits early. > > > > > > I suppose we will need to add an option to tell Vim that you will read > > > the messages, not using a callback. I think this should do it: > > > > > > "drop"Specifies when to drop messages: > > > "auto" When there is no callback to handle a message. > > > The "close_cb" is also considered for this. > > > "never" All messages will be kept. > > > > > > > It's very nice if I can have the "drop" option in job_start ! > > Should work with patch 8.0.0107. Took a bit of debugging to figure out > why the channel was reported as being closed. I thought messages were > being dropped somewhere, but it turned out to be a problem with > ch_canread(). > Finally, thanks very much :D -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Patch 8.0.0105
Skywind wrote: > Bram Moolenaar wrote: > > > You can improve this a lot by changing: > > > >if ch_canread(ch) > > let text = ch_read(ch, {'timeout':0}) > > caddexpr text > > cbottom > > endif > > > > To: > > > >if ch_canread(ch) > > while ch_canread(ch) > > let text = ch_read(ch, {'timeout':0}) > > caddexpr text > > endwhile > > cbottom > > endif > > Yes, I intentionally do so to make reading really slower than writing. > > > But, you might still miss some messages if the job exits early. > > > > I suppose we will need to add an option to tell Vim that you will read > > the messages, not using a callback. I think this should do it: > > > > "drop" Specifies when to drop messages: > > "auto" When there is no callback to handle a message. > > The "close_cb" is also considered for this. > > "never" All messages will be kept. > > > > It's very nice if I can have the "drop" option in job_start ! Should work with patch 8.0.0107. Took a bit of debugging to figure out why the channel was reported as being closed. I thought messages were being dropped somewhere, but it turned out to be a problem with ch_canread(). -- hundred-and-one symptoms of being an internet addict: 65. The last time you looked at the clock it was 11:30pm, and in what seems like only a few seconds later, your sister runs past you to catch her 7am school bus. /// Bram Moolenaar -- b...@moolenaar.net -- http://www.Moolenaar.net \\\ ///sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\ \\\ an exciting new programming language -- http://www.Zimbu.org/// \\\help me help AIDS victims -- http://ICCF-Holland.org/// -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Patch 8.0.0105
Bram Moolenaar wrote: > You can improve this a lot by changing: > >if ch_canread(ch) > let text = ch_read(ch, {'timeout':0}) > caddexpr text > cbottom > endif > > To: > >if ch_canread(ch) > while ch_canread(ch) > let text = ch_read(ch, {'timeout':0}) > caddexpr text > endwhile > cbottom > endif Yes, I intentionally do so to make reading really slower than writing. > But, you might still miss some messages if the job exits early. > > I suppose we will need to add an option to tell Vim that you will read > the messages, not using a callback. I think this should do it: > > "drop"Specifies when to drop messages: > "auto" When there is no callback to handle a message. > The "close_cb" is also considered for this. > "never" All messages will be kept. > It's very nice if I can have the "drop" option in job_start ! -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Patch 8.0.0105
Skywind wrote: > Bram Moolenaar wrote: > > When writing the test I also had problems with messages being dropped. > > I thought that adding a close callback should fix it, but perhaps there > > is another situation where they are still dropped. > > > > Can you enable the channel log and see where the messages are dropped? > > > > I reduced loop times in timerjob.py from 8 to 100 times > > The full file is still too long, I have uploaded it on gist: > https://gist.github.com/skywind3000/3a2f6016240e33e97587e1181563b163 > > Here is an abbreviated version: Thanks. I had a try myself to get the full log. Apparently the close callback is called quite soon, and since you only read a line every 10 msec you miss all the ones after a second. You can improve this a lot by changing: if ch_canread(ch) let text = ch_read(ch, {'timeout':0}) caddexpr text cbottom endif To: if ch_canread(ch) while ch_canread(ch) let text = ch_read(ch, {'timeout':0}) caddexpr text endwhile cbottom endif But, you might still miss some messages if the job exits early. I suppose we will need to add an option to tell Vim that you will read the messages, not using a callback. I think this should do it: "drop" Specifies when to drop messages: "auto" When there is no callback to handle a message. The "close_cb" is also considered for this. "never" All messages will be kept. -- hundred-and-one symptoms of being an internet addict: 60. As your car crashes through the guardrail on a mountain road, your first instinct is to search for the "back" button. /// Bram Moolenaar -- b...@moolenaar.net -- http://www.Moolenaar.net \\\ ///sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\ \\\ an exciting new programming language -- http://www.Zimbu.org/// \\\help me help AIDS victims -- http://ICCF-Holland.org/// -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Patch 8.0.0105
Bram Moolenaar wrote: > When writing the test I also had problems with messages being dropped. > I thought that adding a close callback should fix it, but perhaps there > is another situation where they are still dropped. > > Can you enable the channel log and see where the messages are dropped? > I reduced loop times in timerjob.py from 8 to 100 times The full file is still too long, I have uploaded it on gist: https://gist.github.com/skywind3000/3a2f6016240e33e97587e1181563b163 Here is an abbreviated version: start log session 14.983871 : Starting job: /usr/bin/python timerjob.py 14.983992 on 0: Created channel 15.006013 : looking for messages on channels 15.013262 RECV on 0: 'timer job: this is line 0 timer job: this is line 1 timer job: this is line 2 timer job: this is line 3 timer job: this is line 4 timer job: this is line 5 ' 15.013299 RECV on 0: 'timer job: this is line 6 timer job: this is line 7 timer job: this is line 8 ' 15.013314 RECV on 0: 'timer job: this is line 9 timer job: this is line 10 ' () 15.014157 RECV on 0: 'timer job: this is line 142 timer job: this is line 143 ' 15.014168 RECV on 0: 'timer job: this is line 144 timer job: this is line 145 ' 15.014253 on 0: Blocking NL read, timeout: 0 msec 15.014271 on 0: Returning 25 bytes 15.014401 RECV on 0: 'timer job: this is line 146 timer job: this is line 147 timer job: this is line 148 timer job: this is line 149 () timer job: this is line 176 timer job: this is line 177 timer job: this is line 178 timer job: this is line 179 timer job: this is line 180 ' 15.014903 RECV on 0: 'timer job: this is line 181 timer job: this is line 182 timer job: this is line 183 timer job: this is line 184 timer job: this is line 185 timer job: this is line 186 timer job: this is line 187 timer job: this is line 188 timer job: this is line 189 timer job: this is line 190 timer job: this is line 191 timer job: this is line 192 timer job: this is line 193 timer job: this is line 194 timer job: this is line 195 timer job: this is line 196 timer job: this is line 197 timer job: this is line 198 timer job: this is line 199 ' 15.027015 : looking for messages on channels 15.027154 on 0: Blocking NL read, timeout: 0 msec 15.027171 on 0: Returning 25 bytes 15.039816 : looking for messages on channels 15.039939 on 0: Blocking NL read, timeout: 0 msec 15.039956 on 0: Returning 25 bytes 15.052561 : looking for messages on channels 15.052716 on 0: Blocking NL read, timeout: 0 msec 15.052733 on 0: Returning 25 bytes () 15.974655 : looking for messages on channels 15.974805 on 0: Blocking NL read, timeout: 0 msec 15.974822 on 0: Returning 26 bytes 15.987733 : looking for messages on channels 15.987887 on 0: Blocking NL read, timeout: 0 msec 15.987904 on 0: Returning 26 bytes 16.000733 : looking for messages on channels 16.000856 on 0: Blocking NL read, timeout: 0 msec 16.000873 on 0: Returning 26 bytes 16.013597 : looking for messages on channels 16.013730 on 0: Blocking NL read, timeout: 0 msec 16.013746 on 0: Returning 26 bytes 16.018591 RECV on 0: '[done in 0 seconds] ' 16.018602 : looking for messages on channels 16.019356 on 0: channel_select_check(): Read EOF from ch_part[1], closing 16.019387 : looking for messages on channels 16.019399 on 0: Closing channel because all readable fds are closed 16.019409 on 0: Closing channel 16.019425 on 0: Invoking callbacks before closing 16.019438 on 0: Invoking close callback MyClose 16.022167 on 0: Dropping message 'timer job: this is line 78 timer job: this is line 79 ' 16.022207 on 0: Dropping message 'timer job: this is line 80 timer job: this is line 81 ' 16.00 on 0: Dropping message 'timer job: this is line 82 ' 16.022231 on 0: Dropping message 'timer job: this is line 83 timer job: this is line 84 ' 16.022241 on 0: Dropping message 'timer job: this is line 85 timer job: this is line 86 ' 16.022252 on 0: Dropping message 'timer job: this is line 87 timer job: this is line 88 ' 16.022263 on 0: Dropping message 'timer job: this is line 89 timer job: this is line 90 ' 16.022273 on 0: Dropping message 'timer job: this is line 91 timer job: this is line 92 ' 16.022299 on 0: Dropping message 'timer job: this is line 93 timer job: this is line 94 ' 16.022311 on 0: Dropping message 'timer job: this is line 95 timer job: this is line 96 ' 16.022321 on 0: Dropping message 'timer job: this is line 97 ' 16.022332 on 0: Dropping message 'timer job: this is line 98 timer job: this is line 99 ' 16.022342 on 0: Dropping message 'timer job: this is line 100 timer job: this is line 101 ' 16.022352 on 0: Dropping message 'timer job: this is line 102 timer job: this is line 103 ' 16.022362 on 0: Dropping message 'timer job: this is line 104 timer job: this is line 105 timer job: this is line 106 timer job: this is line 107 timer job:
Re: Patch 8.0.0105
Skywind wrote: > Bram Moolenaar wrote: > > Patch 8.0.0105 > > Problem:When using ch_read() with zero timeout, can't tell the > > difference > > between reading an empty line and nothing available. > > Solution: Add ch_canread(). > > Files: src/evalfunc.c, src/channel.c, src/proto/channel.pro, > > src/testdir/test_channel.vim, src/testdir/shared.vim, > > runtime/doc/eval.txt, runtime/doc/channel.txt > > This is fabulous, thanks for ch_canread() > However, a lot of messages are still dropped in spite of adding a sleep > before exiting: > > - > timerjob.vim > - > function! MyTimer(id) > let ch = job_getchannel(g:job) > let st = ch_status(ch) > if st == "fail" || st == "closed" > return > endif > if ch_canread(ch) > let text = ch_read(ch, {'timeout':0}) > caddexpr text > cbottom > endif > endfunc > > " according to channel.txt, add a close callback > function! MyClose(channel) > caddexpr "[channel closed]" > cbottom > endfunc > > copen 8 > cexpr "[python output]" > wincmd k > > let cmd = ['/usr/bin/python', 'timerjob.py'] > let opt = {"out_io": "pipe", "err_io": "out", "close_cb": "MyClose"} > > let job = job_start(cmd, opt) > let timer = timer_start(10, "MyTimer", {"repeat":-1}) > > - > timerjob.py > - > #! /usr/bin/env python2 > import sys, time > > for i in xrange(8): > sys.stdout.write('timer job: this is line %d\n'%(i)) > sys.stdout.flush() > > # as channel.txt said, we need a sleep here > time.sleep(1) > > - > output in quickfix > - > || [python output] > || timer job: this is line 0 > || timer job: this is line 1 > || timer job: this is line 2 > || timer job: this is line 3 > || timer job: this is line 4 > || timer job: this is line 5 > . > || timer job: this is line 57 > || timer job: this is line 58 > || timer job: this is line 59 > || timer job: this is line 60 > || timer job: this is line 61 > || [channel closed] > > Only 62 messages have been received, almost 8 - 62 = 79938 messages have > been dropped. > > I can sleep longer in timerjob.py, but I can't add a single sleep if I start > grep as a job. And I can't decide how long should I sleep to avoid dropping. When writing the test I also had problems with messages being dropped. I thought that adding a close callback should fix it, but perhaps there is another situation where they are still dropped. Can you enable the channel log and see where the messages are dropped? -- hundred-and-one symptoms of being an internet addict: 57. You begin to wonder how on earth your service provider is allowed to call 200 hours per month "unlimited." /// Bram Moolenaar -- b...@moolenaar.net -- http://www.Moolenaar.net \\\ ///sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\ \\\ an exciting new programming language -- http://www.Zimbu.org/// \\\help me help AIDS victims -- http://ICCF-Holland.org/// -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Patch 8.0.0105
Kazunobu Kuriyama wrote: > Looks you are trying to simplify the issue too much. > > Your argument looks good for utilities such as grep, tail, cut and so on. > > However, Vim (or any other programs having UI which is driven by events) is > different from them in that, while Vim is made busy to handle a deluge of > input from an external program, it also has to ask the terminal or the GUI to > update its contents simultaneously (in appearance, at least). > > > Actually, the event loop cannot handle the both tasks simultaneously; it > tries to do only one thing when select() or poll() is called, and it can > execute a thing only after it makes sure another is staying idle at that > time. IOW, an illusion that both external input handling and UI update > appear to happen simultaneously is made possible only if both of them can > stay idle alternatively. > > > The GUI will be locked if it is not given a chance to run the event loop for > its own sake. > > > The third element UI makes your issue very tough, and that issue itself is > inherently difficult; while the user is asking an external program to do its > job asynchronously, he is also asking Vim to show the result as if they were > working synchronously, using the UI which inherently works asynchronously. A > awful set of simultaneous equations. > > > That said, I never mean that any effort done here so far doesn't make sense. > Actually, I already enjoy the fruit; as far as the examples skywind3000 > raised (except the one after 8.0.105) are concerned, the issue appears to be > fixed for me. > > > -- > > -- > > You received this message from the "vim_dev" maillist. > > Do not top-post! Type your reply below the text you are replying to. > > For more information, visit http://www.vim.org/maillist.php > > > > --- > > You received this message because you are subscribed to the Google Groups > "vim_dev" group. > > To unsubscribe from this group and stop receiving emails from it, send an > email to vim_dev+u...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. Thanks for your reply, You mean I spoke too much on the same topic ? I think I know what a event-loop is, I wrote a lot of gui programs with win32 and qt before. This is not as complex as you said before, for example: EditPlus/GEdit can get background jobs working correctly. Last week when I was trying to get grep working with vim's job system. It just worked, but wasn't so perfect. Vim got freezed when grep trying to output a lot of messages in a short time (this can be reproduced by running grep in your document root with a very simple word). Channel callback consumes too much time. I was curious that what will happen if I run the same grep command in other editors like EditPlus or GEdit. Switching to EditPlus and setting grep as an "User Tool" in EditPlus, I found EditPlus never got freezed, it just slow down the speed of grep. So I am trying to solve it by introducing a flow control like mechanism: invoking ch_read(..., {"timeout":0}) fixed times in a timer. If vim doesn't read as faster as the child process, child should block when system pipe is full. Child process could be slow down and Vim can run as smooth as usual. After digging a lot in the channel.txt, I suppose a channel (without callbacks) can work as a traditional non-blocking socket. Just like other runtime libraries of many programming language. It's much simply then the callback implementation. We don't even need to put these fds into the fd-set of select/poll if a channel is created without callbacks. Everything is fine if the channel (without callbacks) can just work like a nonblocking socket. (that is it has a nonblocking "read" function and can returns some messages or tells me EAGAIN/EWOULDBLOCK). Thanks to Bram, we have job/channel system now and sure it is awesome. And I am thinking these asychronous apis will be used in the next decade or even longer. Great plugins will be born under the advantage of them. If we can get grep to work perfectly with vim today. We can expect many gorgeous plugins tomorrow. -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Patch 8.0.0105
2016-11-30 19:06 GMT+09:00 skywind3000 : > skywind3000 wrote: > > > we can see: > > > > 1. there is a "sleep" in the parent process, parent works slower than > child. > > > > 2. calling "write" block the child process successfully when pipe is > full (4096 bytes on linux by default). The child process does not exit > immediately. > > > > 3. 10 seconds later after child process exist, we can still read > messages from the parent process. > > > > This is exactly the right behavior of inter-processes communication. > > > > Since we can't add a sleep in grep, tail, less, more, cut. and we > don't know how long should we sleep either. > > > > How can we prevent message dropping when we are using ch_read ? > > I can reproduce it with popen in python: > -- > parent.py > -- > import sys, os, time > > fp = os.popen('python child.py', 'r') > > while 1: > text = fp.readline() > if text == '': > break > print text.rstrip() > sys.stdout.flush() > time.sleep(0.001) > > > child.py > > #! /usr/bin/env python2 > import sys, time > > t = time.time() > for i in xrange(2): > sys.stdout.write('timer job: this is line %d\n'%(i)) > sys.stdout.flush() > > t = time.time() - t > print '[done in %d seconds]'%int(t) > > # as channel.txt said, we need a sleep here > time.sleep(1) > > > reproduce > > $ python parent.py > timer job: this is line 0 > timer job: this is line 1 > timer job: this is line 2 > . > timer job: this is line 19996 > timer job: this is line 19997 > timer job: this is line 19998 > timer job: this is line 1 > [done in 21 seconds] > > - > 1. there is a sleep in parent.py, parent works slower than child too. > 2. the child.py can be blocked successfully by parent.py > 3. no sleep in child.py > 4. no message dropped. > > it behaves the same way just like the c version in my previous post. > Looks you are trying to simplify the issue too much. Your argument looks good for utilities such as grep, tail, cut and so on. However, Vim (or any other programs having UI which is driven by events) is different from them in that, while Vim is made busy to handle a deluge of input from an external program, it also has to ask the terminal or the GUI to update its contents simultaneously (in appearance, at least). Actually, the event loop cannot handle the both tasks simultaneously; it tries to do only one thing when select() or poll() is called, and it can execute a thing only after it makes sure another is staying idle at that time. IOW, an illusion that both external input handling and UI update appear to happen simultaneously is made possible only if both of them can stay idle alternatively. The GUI will be locked if it is not given a chance to run the event loop for its own sake. The third element UI makes your issue very tough, and that issue itself is inherently difficult; while the user is asking an external program to do its job asynchronously, he is also asking Vim to show the result as if they were working synchronously, using the UI which inherently works asynchronously. A awful set of simultaneous equations. That said, I never mean that any effort done here so far doesn't make sense. Actually, I already enjoy the fruit; as far as the examples skywind3000 raised (except the one after 8.0.105) are concerned, the issue appears to be fixed for me. > > -- > -- > You received this message from the "vim_dev" maillist. > Do not top-post! Type your reply below the text you are replying to. > For more information, visit http://www.vim.org/maillist.php > > --- > You received this message because you are subscribed to the Google Groups > "vim_dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to vim_dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Patch 8.0.0105
skywind3000 wrote: > we can see: > > 1. there is a "sleep" in the parent process, parent works slower than child. > > 2. calling "write" block the child process successfully when pipe is full > (4096 bytes on linux by default). The child process does not exit immediately. > > 3. 10 seconds later after child process exist, we can still read messages > from the parent process. > > This is exactly the right behavior of inter-processes communication. > > Since we can't add a sleep in grep, tail, less, more, cut. and we don't > know how long should we sleep either. > > How can we prevent message dropping when we are using ch_read ? I can reproduce it with popen in python: -- parent.py -- import sys, os, time fp = os.popen('python child.py', 'r') while 1: text = fp.readline() if text == '': break print text.rstrip() sys.stdout.flush() time.sleep(0.001) child.py #! /usr/bin/env python2 import sys, time t = time.time() for i in xrange(2): sys.stdout.write('timer job: this is line %d\n'%(i)) sys.stdout.flush() t = time.time() - t print '[done in %d seconds]'%int(t) # as channel.txt said, we need a sleep here time.sleep(1) reproduce $ python parent.py timer job: this is line 0 timer job: this is line 1 timer job: this is line 2 . timer job: this is line 19996 timer job: this is line 19997 timer job: this is line 19998 timer job: this is line 1 [done in 21 seconds] - 1. there is a sleep in parent.py, parent works slower than child too. 2. the child.py can be blocked successfully by parent.py 3. no sleep in child.py 4. no message dropped. it behaves the same way just like the c version in my previous post. -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Patch 8.0.0105
skywind3000 wrote: > I can sleep longer in timerjob.py, but I can't add a single sleep if I start > grep as a job. And I can't decide how long should I sleep to avoid dropping. latest channel.txt said: Note that if the job exits before you read the output, the output may be lost. This depends on the system (on Unix this happens because closing the write end of a pipe causes the read end to get EOF). To avoid this make the job sleep for a short while before it exits. It is strange if a sleep is required in every command line utility, half of shell commands (with pipe) we used every day may break down. I wrote another test to verify this issue: the child process is repeating calling write the parent process is repeating calling read & sleep child process exits quickly, but we don't have any message dropped pipesleep.c #include #include #include #include #include #include #define READ 0 #define WRITE 1 int main(void) { int fd[2], pid; if (pipe(fd) < 0) { perror("pipe"); exit(1); } if ((pid = fork()) == -1) { perror("fork"); exit(1); } if (pid == 0) { // child char text[100]; int i; close(fd[READ]); for (i = 0; i < 2000; i++) { sprintf(text, "this is very loonnng line %d\n", i); write(fd[WRITE], text, strlen(text)); } fprintf(stderr, "[child exit here]\n"); fflush(stderr); } else { // parent char text[100]; int i; close(fd[WRITE]); for (i = 0; i < 1; i++) { int hr = read(fd[READ], text, 100); if (hr <= 0) { printf("\n[done, errno is %d]\n", errno); break; } else { text[hr] = 0; printf("%s", text); fflush(stdout); } usleep(10 * 1000); } } return 0; } output of pipesleep.c this is very loonnng line 0 this is very loonnng line 1 this is very loonnng line 2 this is very loonnng line 3 this is very loonnng line 4 this is very loonnng line 5 this is very loonnng line 6 this is very loonnng line 7 this is very loonnng line 8 .. this is very loonnng line 829 [child exit here] this is very loonnng line 830 this is very loonnng line 831 .. this is very loonnng line 1997 this is very loonnng line 1998 this is very loonnng line 1999 [done, errno is 0] we can see: 1. there is a "sleep" in the parent process, parent works slower than child. 2. calling "write" block the child process successfully when pipe is full (4096 bytes on linux by default). The child process does not exit immediately. 3. 10 seconds later after child process exist, we can still read messages from the parent process. This is exactly the right behavior of inter-processes communication. Since we can't add a sleep in grep, tail, less, more, cut. and we don't know how long should we sleep either. How can we prevent message dropping when we are using ch_read ? -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Patch 8.0.0105
Bram Moolenaar wrote: > Patch 8.0.0105 > Problem:When using ch_read() with zero timeout, can't tell the difference > between reading an empty line and nothing available. > Solution: Add ch_canread(). > Files: src/evalfunc.c, src/channel.c, src/proto/channel.pro, > src/testdir/test_channel.vim, src/testdir/shared.vim, > runtime/doc/eval.txt, runtime/doc/channel.txt This is fabulous, thanks for ch_canread() However, a lot of messages are still dropped in spite of adding a sleep before exiting: - timerjob.vim - function! MyTimer(id) let ch = job_getchannel(g:job) let st = ch_status(ch) if st == "fail" || st == "closed" return endif if ch_canread(ch) let text = ch_read(ch, {'timeout':0}) caddexpr text cbottom endif endfunc " according to channel.txt, add a close callback function! MyClose(channel) caddexpr "[channel closed]" cbottom endfunc copen 8 cexpr "[python output]" wincmd k let cmd = ['/usr/bin/python', 'timerjob.py'] let opt = {"out_io": "pipe", "err_io": "out", "close_cb": "MyClose"} let job = job_start(cmd, opt) let timer = timer_start(10, "MyTimer", {"repeat":-1}) - timerjob.py - #! /usr/bin/env python2 import sys, time for i in xrange(8): sys.stdout.write('timer job: this is line %d\n'%(i)) sys.stdout.flush() # as channel.txt said, we need a sleep here time.sleep(1) - output in quickfix - || [python output] || timer job: this is line 0 || timer job: this is line 1 || timer job: this is line 2 || timer job: this is line 3 || timer job: this is line 4 || timer job: this is line 5 . || timer job: this is line 57 || timer job: this is line 58 || timer job: this is line 59 || timer job: this is line 60 || timer job: this is line 61 || [channel closed] Only 62 messages have been received, almost 8 - 62 = 79938 messages have been dropped. I can sleep longer in timerjob.py, but I can't add a single sleep if I start grep as a job. And I can't decide how long should I sleep to avoid dropping. -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Patch 8.0.0105
Patch 8.0.0105 Problem:When using ch_read() with zero timeout, can't tell the difference between reading an empty line and nothing available. Solution: Add ch_canread(). Files: src/evalfunc.c, src/channel.c, src/proto/channel.pro, src/testdir/test_channel.vim, src/testdir/shared.vim, runtime/doc/eval.txt, runtime/doc/channel.txt *** ../vim-8.0.0104/src/evalfunc.c 2016-11-24 15:09:03.409856638 +0100 --- src/evalfunc.c 2016-11-29 20:23:04.633430178 +0100 *** *** 76,81 --- 76,82 static void f_ceil(typval_T *argvars, typval_T *rettv); #endif #ifdef FEAT_JOB_CHANNEL + static void f_ch_canread(typval_T *argvars, typval_T *rettv); static void f_ch_close(typval_T *argvars, typval_T *rettv); static void f_ch_close_in(typval_T *argvars, typval_T *rettv); static void f_ch_evalexpr(typval_T *argvars, typval_T *rettv); *** *** 499,504 --- 500,506 {"ceil", 1, 1, f_ceil}, #endif #ifdef FEAT_JOB_CHANNEL + {"ch_canread",1, 1, f_ch_canread}, {"ch_close", 1, 1, f_ch_close}, {"ch_close_in", 1, 1, f_ch_close_in}, {"ch_evalexpr", 2, 3, f_ch_evalexpr}, *** *** 1779,1784 --- 1781,1801 #ifdef FEAT_JOB_CHANNEL /* + * "ch_canread()" function + */ + static void + f_ch_canread(typval_T *argvars, typval_T *rettv) + { + channel_T *channel = get_channel_arg(&argvars[0], TRUE, TRUE, 0); + + rettv->vval.v_number = 0; + if (channel != NULL) + rettv->vval.v_number = channel_has_readahead(channel, PART_SOCK) + || channel_has_readahead(channel, PART_OUT) + || channel_has_readahead(channel, PART_ERR); + } + + /* * "ch_close()" function */ static void *** ../vim-8.0.0104/src/channel.c 2016-11-26 15:13:29.402218088 +0100 --- src/channel.c 2016-11-29 20:13:58.177247346 +0100 *** *** 2603,2609 /* * Return TRUE if "channel" has JSON or other typeahead. */ ! static int channel_has_readahead(channel_T *channel, ch_part_T part) { ch_mode_T ch_mode = channel->ch_part[part].ch_mode; --- 2603,2609 /* * Return TRUE if "channel" has JSON or other typeahead. */ ! int channel_has_readahead(channel_T *channel, ch_part_T part) { ch_mode_T ch_mode = channel->ch_part[part].ch_mode; *** ../vim-8.0.0104/src/proto/channel.pro 2016-11-26 15:13:29.402218088 +0100 --- src/proto/channel.pro 2016-11-29 20:14:03.549209705 +0100 *** *** 25,30 --- 25,31 int channel_collapse(channel_T *channel, ch_part_T part, int want_nl); int channel_can_write_to(channel_T *channel); int channel_is_open(channel_T *channel); + int channel_has_readahead(channel_T *channel, ch_part_T part); char *channel_status(channel_T *channel, int req_part); void channel_info(channel_T *channel, dict_T *dict); void channel_close(channel_T *channel, int invoke_close_cb); *** ../vim-8.0.0104/src/testdir/test_channel.vim2016-11-17 17:25:28.212093109 +0100 --- src/testdir/test_channel.vim2016-11-29 21:52:55.853041485 +0100 *** *** 58,63 --- 58,66 " string with ][ should work call assert_equal('this][that', ch_evalexpr(handle, 'echo this][that')) + " nothing to read now + call assert_equal(0, ch_canread(handle)) + " sending three messages quickly then reading should work for i in range(3) call ch_sendexpr(handle, 'echo hello ' . i) *** *** 368,374 endif call ch_setoptions(handle, {'mode': 'raw'}) ! " The message are sent raw, we do our own JSON strings here. call ch_sendraw(handle, "[1, \"hello!\"]\n", {'callback': 'Ch_handleRaw1'}) call WaitFor('g:Ch_reply1 != ""') call assert_equal("[1, \"got it\"]", g:Ch_reply1) --- 371,377 endif call ch_setoptions(handle, {'mode': 'raw'}) ! " The messages are sent raw, we do our own JSON strings here. call ch_sendraw(handle, "[1, \"hello!\"]\n", {'callback': 'Ch_handleRaw1'}) call WaitFor('g:Ch_reply1 != ""') call assert_equal("[1, \"got it\"]", g:Ch_reply1) *** *** 431,437 return endif call ch_log('Test_raw_pipe()') ! let job = job_start(s:python . " test_channel_pipe.py", {'mode': 'raw'}) call assert_equal(v:t_job, type(job)) call assert_equal("run", job_status(job)) --- 434,443 return endif call ch_log('Test_raw_pipe()') ! " Add a dummy