Re: adding additional network stuff in RealWaitForChar?
On 11/09/06, Gaspar Chilingarov [EMAIL PROTECTED] wrote: Hello all vimmers! I would like to ask - how should I add additional stuff in RealWaitForChar? I have some network protocol, when another side periodically send me keepalive messages(every 10-15 secs). I would like to receive them and process. In GUI based vim it was pretty simple -- using gdk_input_add or similar thing. In other hand I would like to console users have same benefits - I've added handlers in RealWaitForChar, but ... it processes only first message on it's own. All latter messages are received, but does not processed -- until I press some key periodically, thus causing remote site timeouting and aborting the connection. I've implemented it in a quite similar way to FEAT_MOUSE_GPM and USE_XSMP implementations. What can I miss there? Any recommendations where to look? Thanks in advance, Gaspar There is a patched Vim7 here (http://theclapp.org/repos/vim70+async+ecl/) that contains code to handle network callbacks. However, beware! I think there are subtle bugs in way this works with Vim's code path. You cannot just randomly go calling functions from RealWaitForChar (for example, appending text), and expect everything to be OK. You probably want to look in network_io.c and os_unix.c. Cheers Brad
Re: Fastest way to append line or char to a buffer
On 29/08/06, Ilya [EMAIL PROTECTED] wrote:
Brad Beveridge wrote:
On 26/08/06, Brad Beveridge [EMAIL PROTECTED] wrote:
SNIP
Cheers
Brad
Hi, sorry for all the trouble :) I'm having redraw issues, where
running this code to append chars is causing strange problems.
The scenario is this (always using the code below for char output):
1) Write a char to a buffer (line1)
2) Write a newline (line 2)
3) Move the cursor to the bottom of the buffer (line2 empty)
4) Write another Char (appears in line2)
5) Press Ctrl-L
6) Notice that line1 has vanished.
moving the cursor to the new blank line at the bottom of the line is
critical here. I'm sure that I'm doing something wrong, but I really
can't figure it out.
As a side note, it feels very clumsy to have to manually call changed
line functions when calling ml_* funcs - is there some notes I can
read about why this is?
Anyhow, here is the code I'm using to put chars in a buffer - I really
appreciate any thoughts, this feels so close to working, but I've
spend several hours trying to get it right. Very frustrating :)
Cheers
Brad
static void vim_append_char (buf_T target_buf, char c)
{
static char string[2] = {0};
Should not you have = {0, 0} here? Second element never get
initialized but it could be accessed by ml_append_string.
That might be more clear perhaps, but when you initialize an array
like that in C, the last element is propagated for the whole array.
Cheers
Brad
Re: How to trigger a screen update from RealWaitForChar
SNIP Q1) Is using ml_append from (effectively) within RealWaitForChar a bad thing? At the moment it appears to work properly. Yes. You never know where in Vim you are, the caller may not handle the situation that the text changes unexpectedly. It's similar to autocommands, which have caused crashes in the past. At what level in the stack is it safe to call ml_append, etc? How do I tell Vim to retreat up to that level, is there code that I can look at? At the moment I'm trying to do it by adding to an entry to keymap.h:key_extra, pushing 3 bytes into buf in os_unix.c (just like the cursorhold), but I have no idea how to get right out into main_loop. Cheers Brad
Re: Fastest way to append line or char to a buffer
On 26/08/06, Brad Beveridge [EMAIL PROTECTED] wrote:
SNIP
Cheers
Brad
Hi, sorry for all the trouble :) I'm having redraw issues, where
running this code to append chars is causing strange problems.
The scenario is this (always using the code below for char output):
1) Write a char to a buffer (line1)
2) Write a newline (line 2)
3) Move the cursor to the bottom of the buffer (line2 empty)
4) Write another Char (appears in line2)
5) Press Ctrl-L
6) Notice that line1 has vanished.
moving the cursor to the new blank line at the bottom of the line is
critical here. I'm sure that I'm doing something wrong, but I really
can't figure it out.
As a side note, it feels very clumsy to have to manually call changed
line functions when calling ml_* funcs - is there some notes I can
read about why this is?
Anyhow, here is the code I'm using to put chars in a buffer - I really
appreciate any thoughts, this feels so close to working, but I've
spend several hours trying to get it right. Very frustrating :)
Cheers
Brad
static void vim_append_char (buf_T target_buf, char c)
{
static char string[2] = {0};
buf_T *savebuf = curbuf;
curbuf = target_buf;
int start_line = curbuf-b_ml.ml_line_count;
int lines_changed = 0;
string[0] = c;
if (string[0] == '\n')
{
ml_append (start_line, , 0, FALSE);
lines_changed = 0;
}
else
ml_append_string (start_line, string, -1);
changed_lines (start_line, 0, start_line, lines_changed);
/* restore and return */
curbuf = savebuf;
}
Re: How to trigger a screen update from RealWaitForChar
I have gathered together the network_io patch as best I can. The author is Jim Bailey, not me - however Jim is currently roaming about in the world and was unable to respond to my requests for permission to distribute this code. Jim did include the standard Vim license comments in his files, and other members on the Slim-Vim list agree that he would want his code in the main Vim tree. Author : Jim Bailey Purpose : Add functions to Vim that enable callbacks on file descriptors. This is particularly useful for scripts that wish to listen to a socket and trigger some code to run when a socket gets data. When callbacks return true, they effectively send a flag up the stack, where update_screen is called from getchar.c Known bugs : Callback code can write text to buffers, when the callback triggers screen_update, the syntax colouring can go the wrong colour. I used dmalloc to track memory, and there is some broken memory in syntax.c - sorry I can't be more specific right now. Potential issues : - The update_screen call might be happening at the wrong place and the wrong time. - The callback occurs deep in RealWaitForChar, and script code can potentially call any exposed Vim function. This might be a Bad Thing, I'm not familiar enough with Vim internals to know. - Callbacks MUST drain data from the file descriptor, otherwise there will be a loop that hangs Vim, always calling the callback. Patch specific bugs: (These are all my fault) - I recorded the patch the wrong way around, it must be applied with patch -R ... - I am not familiar with autoconf stuff, so this patch does not integrate with the make system of Vim, somebody probably needs to add it to config.in. There were too many diffs in the various files that I thought to check, and I didn't want to over patch them. I think that's it. I'd love to hear any feedback about this patch. It would be fantastic if others found this useful. Cheers Brad network_io.patch.gz Description: GNU Zip compressed data
Re: Fastest way to append line or char to a buffer
On 22/08/06, Bram Moolenaar [EMAIL PROTECTED] wrote:
Brad Beveridge wrote:
SNIP
Thanks for making this patch.
No problem - it happened to be my particular itch :)
I wonder how you create a new line. Using the old mechanism you mostly
end up reallocating the line at the next insert. Allocating a bit of
room to anticipate further appends would be more efficient.
I'm sorry, I don't understand quite what you mean with that comment.
Do you mean if you only used ml_append_string, how do new lines ever
get created? If you do mean that, then the answer is: new lines need
to be created by an upper level function. I have an upper level
append_char function that is more or less:
void append_char (buf, char) {
static uchar string[2] = {0};
curbuf = buf;
if (char == '\n')
ml_append // create a new line
else
{
string[0] = char;
ml_append_string (string)
...
The reason that I didn't do new line creation in append_string is that
you would need to check all incoming strings for newlines, the upper
level layers can better do this check because they have more
information about the incoming string - they could know it will never
contain newlines.
Also, for appending to a different line than before you copy the line
twice. You can avoid that by using vim_strnsave() instead of
vim_strsave() and reserving space for new characters right away.
Yes, this is a slight inefficiency. The main reason that I did this
was that I wanted all the code to do with re-allocation in the same
place, ie the next if block. Looking at the code again now, the
solution is probably to not copy in the cache miss code, where we
flush - just do
curbuf-b_ml.ml_line_ptr = ml_get(lnum);
Cheers
Brad
Re: Fastest way to append line or char to a buffer
On 22/08/06, Bram Moolenaar [EMAIL PROTECTED] wrote: SNIP I understand. It would be a bit more efficient if the new, empty line that you add when a newline is encountered already has room for a certain number of characters. Ah, I see. True, but since I don't know the code base that well I didn't want to touch too much code all over the place. Also, for appending to a different line than before you copy the line twice. You can avoid that by using vim_strnsave() instead of vim_strsave() and reserving space for new characters right away. Yes, this is a slight inefficiency. The main reason that I did this was that I wanted all the code to do with re-allocation in the same place, ie the next if block. Looking at the code again now, the solution is probably to not copy in the cache miss code, where we flush - just do curbuf-b_ml.ml_line_ptr = ml_get(lnum); I suppose that should work (didn't look at it carefully). I might have a look later if this code can be used in certain situations. That should make some operations work faster, e.g. pasting a large chunk of text into a terminal emulator. I think it will work from the quick glance over it. I can test it more throughly tonight. I'd love it if I made an actually useful contrib to Vim - I love this editor, thanks for writing it! Brad
Re: Fastest way to append line or char to a buffer
The last code I posted was a little broken. Here is a patch that appears (after a reasonable amount of testing) to work quite well for very quickly appending chars to a vim line. As I said, scripting language authors that use Vim output buffers as STDOUT style streams could use this to good advantage. Cheers Brad append-char.patch.gz Description: GNU Zip compressed data
How to trigger a screen update from RealWaitForChar
Hi all, I'm working with a Vim that has been hacked so that you can get callbacks when data appears on a socket. Basically, our callback system works in pretty much the same way as the code in FEAT_SNIFF. We add the filedescriptors that we are interested in to the select/poll code in os_unix, if there is data pending on a socket then we callback a registered function, which should clear the data from the socket. In my particular case, the callback calls into Lisp code in our embedded Lisp interpreter, which writes data to buffers using ml_replace and ml_append. Q1) Is using ml_append from (effectively) within RealWaitForChar a bad thing? At the moment it appears to work properly. At the moment, after the callback has run we call screen_update(NOT_VALID). However, from what I can see, calling screen_update may (does?) re-call RealWaitForChar, which can retrigger the callback, etc. Q2) Is this nesting of RealWaitForChar a bad thing? (I thing it may be) Q3) What is the correct way/place to trigger screen_update(NOT_VALID)? How should I trigger it from within RealWaitForChar? One other point to consider, although we may do callbacks, RealWaitForChar may not have keyboard input - what should RealWaitForChar return in this case? If the list would like to see this code, I can post a patch, or you can use Darcs to get a Vim+Callbacks+ECL from http://theclapp.org/repos/vim70+async+ecl/. Q4) Would this callback mechanism be more generally useful for other scripting engines? Cheers Brad
Re: Fastest way to append line or char to a buffer
On 16/08/06, Brad Beveridge [EMAIL PROTECTED] wrote:
On 16/08/06, Bram Moolenaar [EMAIL PROTECTED] wrote:
Brad -
As a side note, if I wanted to get really fast char append, how does
this method sound:
1) In a structure (not sure if a mem_line or bufT struct), have three
new entries
uint current_line_num;
uchar *current_line_buffer;
uint current_line_alloced;
2) current_line_buffer points to the string data for the current line,
it starts off at a sane size (say 80, or screen_width), and doubles in
alloced size when it is too full.
3) ml_append_char (or whatever I call it), will check the target line
number against current_line_number, if the match then
- append the char, possibly reallocing the current_line_buffer if we
have run out of space
ELSE, the edit line number and our cached line don't match
- trim the size of the current_line_buffer to match the STRLEN of the line
- alloc a new oversized line buffer and copy the new edit line into it
- append the char as above
Vim caches the last changed line, you could do something with that.
Start looking at ml_flush_line(), you can find the rest from there.
Would this kind of optimisation be more widely useful?
Perhaps. Changing the same line several times is not unusual. It
mostly depends on how much code would need to be changed compared to
how much we would gain.
- Bram
Thanks for the tips, I'll test all this out and see what I can come up
with. I also accidentally posted just to Bram, so I've included the
list in this reply.
Thanks
Brad
I ended up writing a string append function, that has sped up my
output code by a couple of orders of magnitude :) Here it is, does it
look sane?
Cheers
Brad
/*
* Append string to line lnum, with buffering, in current buffer.
*
* Always makes a copy of the incoming string
*
* Check: The caller of this function should probably also call
* changed_lines(), unless update_screen(NOT_VALID) is used.
*
* return FAIL for failure, OK otherwise
*/
int
ml_append_string(lnum, string, slen)
linenr_Tlnum;
char_u *string;
int slen;
{
int append_len = 0;
int line_len = 0;
if (slen == -1)
append_len = STRLEN(string);
else
append_len = slen;
if (string == NULL) /* just checking... */
return FAIL;
if (lnum == 0)
ml_append (lnum, string, slen, 0);
/* When starting up, we might still need to create the memfile */
if (curbuf-b_ml.ml_mfp == NULL open_buffer(FALSE, NULL) == FAIL)
return FAIL;
#ifdef FEAT_NETBEANS_INTG
/*
#error I am 99% sure this is broken
if (usingNetbeans)
{
netbeans_removed(curbuf, lnum, 0, (long)STRLEN(ml_get(lnum)));
netbeans_inserted(curbuf, lnum, 0, string, (int)STRLEN(string));
}
*/
#endif
if (curbuf-b_ml.ml_line_lnum != lnum) /* other line buffered */
{
ml_flush_line(curbuf); /* flush it, this frees
ml_line_ptr */
/* Take a pointer to the existing line, will get re-alloced right
* away below because it is not big enough */
curbuf-b_ml.ml_line_ptr = vim_strsave(ml_get(lnum));
curbuf-b_ml.ml_line_len = STRLEN(curbuf-b_ml.ml_line_ptr);
curbuf-b_ml.ml_line_lnum = lnum;
}
/* By here,
* curbuf-b_ml.ml_line_ptr - is filled with the correct existing line
* curbuf-b_ml.ml_line_len - is the alloced size of the line
*/
line_len = STRLEN(curbuf-b_ml.ml_line_ptr);
if (curbuf-b_ml.ml_line_len line_len + append_len + 1)
{
int new_len = (curbuf-b_ml.ml_line_len + line_len + 1) * 2;
char_u *new_line;
if (new_len 80) new_len = 80;
new_line = vim_strnsave(curbuf-b_ml.ml_line_ptr, new_len);
vim_free(curbuf-b_ml.ml_line_ptr);
curbuf-b_ml.ml_line_ptr = new_line;
curbuf-b_ml.ml_line_len = new_len;
}
/* by here:
* ml_line_ptr is allocated to fit the current line + the append
* ml_line_len is the total alloced size of the line
*/
STRCPY (curbuf-b_ml.ml_line_ptr[line_len], string); /* Append
our string */
curbuf-b_ml.ml_line_ptr[line_len + append_len] = NUL; /* End the line */
curbuf-b_ml.ml_flags = (curbuf-b_ml.ml_flags | ML_LINE_DIRTY) ~ML_EMPTY;
return OK;
}
Re: Fastest way to append line or char to a buffer
On 16/08/06, Bram Moolenaar [EMAIL PROTECTED] wrote: Brad - As a side note, if I wanted to get really fast char append, how does this method sound: 1) In a structure (not sure if a mem_line or bufT struct), have three new entries uint current_line_num; uchar *current_line_buffer; uint current_line_alloced; 2) current_line_buffer points to the string data for the current line, it starts off at a sane size (say 80, or screen_width), and doubles in alloced size when it is too full. 3) ml_append_char (or whatever I call it), will check the target line number against current_line_number, if the match then - append the char, possibly reallocing the current_line_buffer if we have run out of space ELSE, the edit line number and our cached line don't match - trim the size of the current_line_buffer to match the STRLEN of the line - alloc a new oversized line buffer and copy the new edit line into it - append the char as above Vim caches the last changed line, you could do something with that. Start looking at ml_flush_line(), you can find the rest from there. Would this kind of optimisation be more widely useful? Perhaps. Changing the same line several times is not unusual. It mostly depends on how much code would need to be changed compared to how much we would gain. - Bram Thanks for the tips, I'll test all this out and see what I can come up with. I also accidentally posted just to Bram, so I've included the list in this reply. Thanks Brad
Fastest way to append line or char to a buffer
Hi all,
I'm working with a Vim that has been patched to support ECL Lisp
(http://wiki.alu.org/Vim_ECL). Part of our project involves
outputting potentially large amounts of data to a buffer in a
streaming fashion, ie always adding at the end.
I think that I have worked out a reasonable way to append lines[1],
but I can't see any obvious way to quickly append single chars to a
buffer.
Are there any functions like ml_append I can look to for this support?
How should I go about appending single chars very quickly?
Cheers
Brad
[1] Here is the append code that I hacked up tonight
static cl_object cl_vim_append_lines_int (cl_object buf, cl_object lines)
{
buf_T *savebuf = curbuf;
curbuf = ((buf_T *)ecl_foreign_data_pointer_safe(buf));
int start_line = curbuf-b_ml.ml_line_count;
while (lines != Cnil) {
char_u *line = string_to_line (cl_car (lines));
ml_append (curbuf-b_ml.ml_line_count-1, line, 0, FALSE);
vim_free (line);
lines = cl_cdr (lines);
}
changed_lines(start_line, 0, curbuf-b_ml.ml_line_count,
(long)(curbuf-b_ml.ml_line_count - start_line));
/* restore and return */
curbuf = savebuf;
return Ct;
}
Fwd: Extra file descriptor callbacks
I accidentally posted to individuals, not the list - damn Gmail! Brad On 16/06/06, John (Eljay) Love-Jensen [EMAIL PROTECTED] wrote: Emacs can do everything: it's an editor, an environment, an operating system (in the same sense that JVM is an operating system), and a religion. The only thing emacs lacks is a good editor. And now, emacs would have a fantastic editor: Vim! I'm not sure if I am reading you right, but we are not aiming for any sort of Emacs compatibility. Slime is a debugger for Lisp, it has two parts a backend that runs in a Common Lisp image called Swank, and a frontend that runs in Emacs and is written in ELisp (Emacs' Lisp dialect). Our project is attempting to re-implement the frontend only, we will crib code heavily from the existing Slime frontend, but Elisp is not the same as CLisp so some stuff will need to be different. And of course, Slime relies heavily on Emacs specific drawing functions that won't port at all. Really the only relationship we have to Emacs is there is an existing code base with the functionality we need, and that existing base happens to run in Emacs. Cheers Brad
Fwd: Extra file descriptor callbacks
Repost: to the actual list this time: On 16/06/06, Yakov Lerner [EMAIL PROTECTED] wrote: On 6/16/06, Bram Moolenaar [EMAIL PROTECTED] wrote: Brad Beveridge wrote: Hello all, I am involved in a project that has embedded ECL (http://ecls.sourceforge.net/), a Lisp interpreter into Vim. Is the next goal of this project to have emacs runnnig inside List interpreter inside Vim :-) ? Heaven forbid!!! :) The main goal of the project is to get Slime (http://common-lisp.net/project/slime/) running in Vim. Slime is a Lisp debugger that uses Emacs as the frontend, and the Emacs side is written in ELisp. Just trying to make it so Emacs isn't the only decent Lisp development environment :) Cheers Brad
Extra file descriptor callbacks
Hello all, I am involved in a project that has embedded ECL (http://ecls.sourceforge.net/), a Lisp interpreter into Vim. Our end goal for the project involves having Lisp code inside of Vim respond to incoming data from a socket, so we also have a patch that allows us to register callbacks that will be triggered when we receive data on a file descriptor. The code for this is here http://theclapp.org/repos/. The async callback patch only really works for GTK and console I think. Basically all the patch does is add a few more file descriptors to poll/select on during Vim's read_char, if the appropriate descriptor is triggered then we get a callback to Lisp. The callbacks all appear to work fine, for example you can setup a listening socket in Vim and have data output to a Vim buffer without any problems. But there is a bug. Lets assume that we have a server that simply echoes back data, and a client that sends data. Data flow is something like: (CLIENT) Embedded Lisp script sends data (SERVER) Echoes data back (CLIENT) Still sending data, there is also pending data coming back from the server (CLIENT) Finishes sending data, next time through the read_char loop should trigger call back which will read data out from the socket. However, the CLIENT side crashes in this scenario, it appears to crash inside the Lisp interpreter. I suspect that somehow the last two phases above are overlapping, and the interpreter is being re-entered, which I don't think it supports. However, I don't see how this can be happening - as far as I am aware Vim is totally single threaded (at least *nix console builds), so the behaviour should be deterministic, the data will finish sending, continue out to the main loop and then trigger the callback. I know it is a lot to ask for help on such a modified Vim, but this is really starting to get me down. I would really appreciate any thoughts about what might be going on, or how to debug this. Cheers Brad
