This patch implements the concept suggested in this
post: https://groups.google.com/forum/#!topic/vim_dev/lylNU4Na2Nk
Heres a simple explanation of whats going on:
- A frozen 'background' thread is started at the beginning of the program.
- At every iteration, the main_loop function will wake that thread to collect
user input, and then will wait for a message
- The background thread will block until the user interacts with vim, then it
will put a message in the queue, and block until it receives another signal
from the main thread.
- When a message is put in the queue, the main_loop function unblocks,
processing
the message and continuing the loop.
The reason for going through all this message queue trouble is simple: It
enables vim to work and synchronize with multiple threads, which is a feature
missed by many vim users and plugin authors :)
For now the message queue only understands two types of messages: User input
and deferred execution of functions.
The simplest case of deferred execution is implemented by the 'defer' function,
which accepts a function name to execute on the next message loop iteration. All
this function does is put a message in the queue, and since the queue is
thread-safe, this function can be used by other threads, started by python for
example. I've written a simple python plugin which illustrates usage:
http://pastebin.com/V9VzH0zU
If youre gonna try this patch be sure to regenerate the configure script:
(cd src && make autoconf).
This is just a proof-of-concept and I have posted it only for review by
Bram and other devs. I'm sure there are better options to some of the
decisions took and I'm still very unfamiliar with the codebase.
Here are the problems I could identify and could use help fixing:
- Test 94 fails
- gVim freezes at startup. I didnt dig much into this problem but its probably
easy to fix since vim treats input from the gui in the same manner as input
from the terminal.
- I couldn't get the screen to update when running the function set by 'defer'.
In the vimscript demo you will notice that keys must be pressed to see the
updates done by the background python thread. I've tried calling
'update_screen(0)' but it just didnt work, probably theres some
check/validation to see if the user typed something before doing the actual
update.
I've done my best to modify as little as possible of the existing code. Anyone
who reads the patch will notice that while this patch introduces a new module,
not many changes were done. In fact the message_loop is implemented as a
compile-time feature, to ensure that vim will continue to work on systems
without pthreads.
I'd like to finish this post by exposing some of the benefits of refactoring
vim's design around a message-loop.
- Plugins written in extension languages(python, ruby, perl...) could make
use of threading for long running tasks and call vim back when done.
- Introduce async versions of the builtin functions that can block the UI.
For example, we could easily create the 'globAsync' and 'systemAsync'
functions that would both accept a function name as last argument. These
'async' versions of the builtin functions would do their work in non-ui
threads, which would periodically post messages with partially calculated
data to be passed to the callbacks.
Two widely used plugins could greatly benefit from this: Syntastic and CtrlP
could collect compiler output and globbed filenames in a very efficient
manner.
- Better code organization. All kinds of events(signals, mouse or keyboard)
could be modeled as specialized messages in the loop
- Bring another level of extensibility to vim:
- Terminal emulators in buffers
- IDE-like capabilities
- Write whole applications on top of vim
Some may look down upon having these features in a text editor, but notice
that these are provided by plugins, and wont affect the size or speed of
a vim default installation.
Vim may not provide an operating system but it certainly can provide a tiny
framework to build one!
- Bigger community. If vim can do more then it will attract more developers.
Thats it for now, anyone is welcome to work on this patch, just comment, send
patches or raise issues at: https://github.com/tarruda/vim/tree/event-loop
---
Filelist | 2 +
runtime/doc/eval.txt | 3 +
runtime/doc/various.txt | 1 +
src/Makefile | 5 ++
src/config.h.in | 3 +
src/configure.in | 17 +++-
src/eval.c | 25 ++++++
src/getchar.c | 8 +-
src/globals.h | 5 ++
src/main.c | 51 +++++++++++
src/message_queue.c | 226 +++++++++++++++++++++++++++++++++++++++++++++++
src/message_queue.h | 34 +++++++
src/version.c | 5 ++
src/vim.h | 4 +
14 files changed, 384 insertions(+), 5 deletions(-)
create mode 100644 src/message_queue.c
create mode 100644 src/message_queue.h
--
--
You received this message from the "vim_dev" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php
---
You received this message because you are subscribed to the Google Groups
"vim_dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
diff --git a/Filelist b/Filelist
index b324933..a50f2b4 100644
--- a/Filelist
+++ b/Filelist
@@ -43,6 +43,8 @@ SRC_ALL = \
src/memline.c \
src/menu.c \
src/message.c \
+ src/message_queue.c \
+ src/message_queue.h \
src/misc1.c \
src/misc2.c \
src/move.c \
diff --git a/runtime/doc/eval.txt b/runtime/doc/eval.txt
index 08edd10..5c16563 100644
--- a/runtime/doc/eval.txt
+++ b/runtime/doc/eval.txt
@@ -6446,6 +6446,9 @@ lua Compiled with Lua interface |Lua|.
mac Macintosh version of Vim.
macunix Macintosh version of Vim, using Unix files (OS-X).
menu Compiled with support for |:menu|.
+messagequeue Compiled with a message queue that scripting languages
+ languages (python, ruby...) can use to safely call
+ vim from other threads.
mksession Compiled with support for |:mksession|.
modify_fname Compiled with file name modifiers. |filename-modifiers|
mouse Compiled with support mouse.
diff --git a/runtime/doc/various.txt b/runtime/doc/various.txt
index 4a1c64a..e78279f 100644
--- a/runtime/doc/various.txt
+++ b/runtime/doc/various.txt
@@ -357,6 +357,7 @@ N *+localmap* Support for mappings local to a buffer |:map-local|
m *+lua* |Lua| interface
m *+lua/dyn* |Lua| interface |/dyn|
N *+menu* |:menu|
+ *+messagequeue* Compiled with thread-safe message queue.
N *+mksession* |:mksession|
N *+modify_fname* |filename-modifiers|
N *+mouse* Mouse handling |mouse-using|
diff --git a/src/Makefile b/src/Makefile
index bceb65c..aa9f445 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -1449,6 +1449,7 @@ BASIC_SRC = \
memline.c \
menu.c \
message.c \
+ message_queue.c \
misc1.c \
misc2.c \
move.c \
@@ -1537,6 +1538,7 @@ OBJ_COMMON = \
objects/memline.o \
objects/menu.o \
objects/message.o \
+ objects/message_queue.o \
objects/misc1.o \
objects/misc2.o \
objects/move.o \
@@ -2646,6 +2648,9 @@ objects/menu.o: menu.c
objects/message.o: message.c
$(CCC) -o $@ message.c
+objects/message_queue.o: message_queue.c
+ $(CCC) -o $@ message_queue.c
+
objects/misc1.o: misc1.c
$(CCC) -o $@ misc1.c
diff --git a/src/config.h.in b/src/config.h.in
index a4e216b..e473d08 100644
--- a/src/config.h.in
+++ b/src/config.h.in
@@ -323,6 +323,9 @@
/* Define for linking via dlopen() or LoadLibrary() */
#undef DYNAMIC_LUA
+/* Define if you want to use thread-safe message queue. */
+#undef FEAT_MESSAGEQUEUE
+
/* Define if you want to include the MzScheme interpreter. */
#undef FEAT_MZSCHEME
diff --git a/src/configure.in b/src/configure.in
index 2718d31..8e52d21 100644
--- a/src/configure.in
+++ b/src/configure.in
@@ -2801,7 +2801,6 @@ AC_CHECK_HEADERS(sys/sysctl.h, [], [],
# include <sys/param.h>
#endif])
-
dnl pthread_np.h may exist but can only be used after including pthread.h
AC_MSG_CHECKING([for pthread_np.h])
AC_TRY_COMPILE([
@@ -2812,6 +2811,22 @@ AC_TRY_COMPILE([
AC_DEFINE(HAVE_PTHREAD_NP_H),
AC_MSG_RESULT(no))
+AC_ARG_ENABLE(messagequeue,
+ [ --enable-messagequeue[=OPTS] Configure vim to use a message queue for synchronization with multi-threaded code. [default=yes] [OPTS=no/yes]], ,
+ [enable_messagequeue="yes"])
+AC_MSG_RESULT($enable_messagequeue)
+
+if test "$enable_messagequeue" = "yes"; then
+ # pthread is needed for the message queue
+ AC_MSG_CHECKING([for pthread.h])
+ AC_TRY_COMPILE([
+ #include <pthread.h>],
+ [int i; i = 0;],
+ AC_MSG_RESULT(yes)
+ AC_DEFINE(FEAT_MESSAGEQUEUE),
+ AC_MSG_RESULT(no))
+fi
+
AC_CHECK_HEADERS(strings.h)
if test "x$MACOSX" = "xyes"; then
dnl The strings.h file on OS/X contains a warning and nothing useful.
diff --git a/src/eval.c b/src/eval.c
index 5d8c7c8..d619a76 100644
--- a/src/eval.c
+++ b/src/eval.c
@@ -31,6 +31,8 @@
# include <math.h>
#endif
+#include <string.h>
+
#define DICT_MAXNEST 100 /* maximum nesting of lists and dicts */
#define DO_NOT_FREE_CNT 99999 /* refcount for dict or list that should not
@@ -502,6 +504,9 @@ static void f_count __ARGS((typval_T *argvars, typval_T *rettv));
static void f_cscope_connection __ARGS((typval_T *argvars, typval_T *rettv));
static void f_cursor __ARGS((typval_T *argsvars, typval_T *rettv));
static void f_deepcopy __ARGS((typval_T *argvars, typval_T *rettv));
+#ifdef FEAT_MESSAGEQUEUE
+static void f_defer __ARGS((typval_T *argvars, typval_T *rettv));
+#endif
static void f_delete __ARGS((typval_T *argvars, typval_T *rettv));
static void f_did_filetype __ARGS((typval_T *argvars, typval_T *rettv));
static void f_diff_filler __ARGS((typval_T *argvars, typval_T *rettv));
@@ -7890,6 +7895,9 @@ static struct fst
{"cscope_connection",0,3, f_cscope_connection},
{"cursor", 1, 3, f_cursor},
{"deepcopy", 1, 2, f_deepcopy},
+#ifdef FEAT_MESSAGEQUEUE
+ {"defer", 1, 1, f_defer},
+#endif
{"delete", 1, 1, f_delete},
{"did_filetype", 0, 0, f_did_filetype},
{"diff_filler", 1, 1, f_diff_filler},
@@ -9825,6 +9833,20 @@ f_deepcopy(argvars, rettv)
}
}
+#ifdef FEAT_MESSAGEQUEUE
+/*
+ * "defer()" function
+ */
+ static void
+f_defer(argvars, rettv)
+ typval_T *argvars;
+ typval_T *rettv;
+{
+ rettv->v_type = VAR_UNKNOWN;
+ queue_push(DeferredCall, strdup(get_tv_string(&argvars[0])));
+}
+#endif
+
/*
* "delete()" function
*/
@@ -12354,6 +12376,9 @@ f_has(argvars, rettv)
#ifdef FEAT_MENU
"menu",
#endif
+#ifdef FEAT_MESSAGEQUEUE
+ "messagequeue",
+#endif
#ifdef FEAT_SESSION
"mksession",
#endif
diff --git a/src/getchar.c b/src/getchar.c
index fe6423d..dc42929 100644
--- a/src/getchar.c
+++ b/src/getchar.c
@@ -1335,11 +1335,11 @@ save_typebuf()
return OK;
}
-static int old_char = -1; /* character put back by vungetc() */
-static int old_mod_mask; /* mod_mask for ungotten character */
+int old_char = -1; /* character put back by vungetc() */
+int old_mod_mask; /* mod_mask for ungotten character */
#ifdef FEAT_MOUSE
-static int old_mouse_row; /* mouse_row related to old_char */
-static int old_mouse_col; /* mouse_col related to old_char */
+int old_mouse_row; /* mouse_row related to old_char */
+int old_mouse_col; /* mouse_col related to old_char */
#endif
#if defined(FEAT_EVAL) || defined(FEAT_EX_EXTRA) || defined(PROTO)
diff --git a/src/globals.h b/src/globals.h
index 916f7e3..8d7e6d3 100644
--- a/src/globals.h
+++ b/src/globals.h
@@ -82,6 +82,8 @@ EXTERN int screen_Columns INIT(= 0); /* actual size of ScreenLines[] */
* held down based on the MOD_MASK_* symbols that are read first.
*/
EXTERN int mod_mask INIT(= 0x0); /* current key modifiers */
+EXTERN int old_mod_mask; /* mod_mask for ungotten character */
+EXTERN int old_char; /* ungotten character */
/*
* Cmdline_row is the row where the command line starts, just below the
@@ -394,6 +396,9 @@ EXTERN buf_T *au_new_curbuf INIT(= NULL);
*/
EXTERN int mouse_row;
EXTERN int mouse_col;
+EXTERN int old_mouse_row; /* mouse_row related to old_char */
+EXTERN int old_mouse_col; /* mouse_col related to old_char */
+
EXTERN int mouse_past_bottom INIT(= FALSE);/* mouse below last line */
EXTERN int mouse_past_eol INIT(= FALSE); /* mouse right of line */
EXTERN int mouse_dragging INIT(= 0); /* extending Visual area with
diff --git a/src/main.c b/src/main.c
index 0407795..286a1a2 100644
--- a/src/main.c
+++ b/src/main.c
@@ -597,6 +597,11 @@ vim_main2(int argc UNUSED, char **argv UNUSED)
# endif
#endif
+#ifdef FEAT_MESSAGEQUEUE
+ /* Initialize the message queue */
+ queue_init();
+#endif
+
#ifndef NO_VIM_MAIN
/* Execute --cmd arguments. */
exe_pre_commands(¶ms);
@@ -1043,6 +1048,10 @@ main_loop(cmdwin, noexmode)
linenr_T conceal_new_cursor_line = 0;
int conceal_update_lines = FALSE;
#endif
+#ifdef FEAT_MESSAGEQUEUE
+ input_data_T *id; /* Input data read from the other thread */
+ message_T *msg; /* next message */
+#endif
#if defined(FEAT_X11) && defined(FEAT_XCLIPBOARD)
/* Setup to catch a terminating error from the X server. Just ignore
@@ -1326,7 +1335,49 @@ main_loop(cmdwin, noexmode)
do_exmode(exmode_active == EXMODE_VIM);
}
else
+ {
+#ifdef FEAT_MESSAGEQUEUE
+ /* Notify the background thread that it should read some input */
+ input_notify();
+
+ /*
+ * Wait for a message, which can be an 'UserInput' message
+ * set by the background thread or a 'DeferredCall' message
+ * indirectly set by vimscript.
+ */
+ msg = queue_shift();
+
+ switch (msg->type)
+ {
+ case UserInput:
+ id = (input_data_T *)msg->data;
+
+ /*
+ * Trick vgetc into thinking this char was returned by
+ * vungetc. Its hacky but avoids messing with that
+ * function for now.
+ */
+ old_char = id->character;
+ old_mod_mask = id->mod_mask;
+ old_mouse_row = id->mouse_row;
+ old_mouse_col = id->mouse_col;
+
+ /* Run the normal command */
+ normal_cmd(&oa, TRUE);
+ break;
+ case DeferredCall:
+ /* Call the defered function */
+ (void)call_func_retnr((char_u *)msg->data, 0, 0, FALSE);
+ break;
+ }
+ /* Free memory we no longer need */
+ vim_free(msg->data);
+ vim_free(msg);
+#else
+ /* Run the normal command */
normal_cmd(&oa, TRUE);
+#endif
+ }
}
}
diff --git a/src/message_queue.c b/src/message_queue.c
new file mode 100644
index 0000000..4c7bc8c
--- /dev/null
+++ b/src/message_queue.c
@@ -0,0 +1,226 @@
+#include "vim.h"
+
+#ifdef FEAT_MESSAGEQUEUE
+
+#include <pthread.h>
+
+#include "message_queue.h"
+
+typedef struct message_queue_T
+{
+ pthread_mutex_t mutex;
+ pthread_cond_t cond;
+ message_T *head;
+ message_T *tail;
+} message_queue_T;
+
+message_queue_T message_queue;
+
+int waiting_for_input;
+pthread_t input_thread;
+pthread_mutex_t input_mutex;
+pthread_cond_t input_cond;
+
+/*
+ * FIXME: Figure out the right way to deal with such errors by asking
+ * on the list
+ */
+ void
+pthread_error(const char *msg)
+{
+ fprintf(stderr, "%s\n", msg);
+ mch_exit(EXIT_FAILURE);
+}
+
+/*
+ * Private helpers to used to lock/unlock the input mutex
+ */
+ static void
+lock(pthread_mutex_t *mutex)
+{
+ if (pthread_mutex_lock(mutex) != 0)
+ pthread_error("Error acquiring user input lock");
+}
+
+ static void
+unlock(pthread_mutex_t *mutex)
+{
+ if (pthread_mutex_unlock(mutex) != 0)
+ pthread_error("Error releasing user input lock");
+}
+
+/*
+ * Function used by the background thread to wait for a signal to read
+ * input from the main thread
+ */
+ void
+input_wait()
+{
+ lock(&input_mutex);
+
+ if (pthread_cond_wait(&input_cond, &input_mutex) != 0)
+ pthread_error("Failed to wait for condition");
+}
+
+
+/*
+ * Function used by the main thread to notify that it should read something
+ */
+ void
+input_notify()
+{
+ if (waiting_for_input)
+ return;
+
+ lock(&input_mutex);
+
+ if (pthread_cond_broadcast(&input_cond) != 0)
+ pthread_error("Failed to acquire lock");
+
+ unlock(&input_mutex);
+}
+
+/*
+ * This function will listen for user input in a separate thread, but only
+ * when asked by the main thead
+ */
+ void *
+vgetcs(arg)
+ void *arg UNUSED; /* Unsused thread start argument */
+{
+
+ input_data_T *data;
+
+ while (TRUE)
+ {
+ waiting_for_input = FALSE;
+
+ // Only try to read input when asked by the main thread
+ input_wait();
+
+ // Dont let the main thread call 'input_notify' or else it would block
+ waiting_for_input = TRUE;
+
+ // Allocate space to hold input data
+ data = (input_data_T *)alloc(sizeof(input_data_T));
+
+ /* The input mutex was configured to be reentrant, so we lock */
+ /* it before entering vgetc. This way we can safely */
+ /* retrieve other global variables set by it (mod_mask, mouse{row,
+ * col}) without risking the main thread overriding them */
+ data->character = vgetc();
+ data->mod_mask = mod_mask;
+ data->mouse_row = mouse_row;
+ data->mouse_col = mouse_col;
+
+ unlock(&input_mutex);
+ queue_push(UserInput, data);
+ }
+}
+
+/*
+ * Initialize the message queue and start listening for user input in a
+ * separate thread.
+ */
+ void
+queue_init()
+{
+ pthread_attr_t attr;
+
+ if (pthread_mutex_init(&input_mutex, NULL) != 0)
+ pthread_error("Failed to init the mutex");
+
+ if (pthread_cond_init(&input_cond, NULL) != 0)
+ pthread_error("Failed to init the condition");
+
+ if (pthread_mutex_init(&message_queue.mutex, NULL) != 0)
+ pthread_error("Failed to init the mutex");
+
+ if (pthread_cond_init(&message_queue.cond, NULL) != 0)
+ pthread_error("Failed to init the condition");
+
+
+ message_queue.head = NULL;
+ message_queue.tail = NULL;
+
+ if (pthread_attr_init(&attr) != 0)
+ pthread_error("Failed to initialize the thread attribute");
+
+ if (pthread_create(&input_thread, &attr, &vgetcs, NULL) != 0)
+ pthread_error("Failed to initialize the user input thread");
+}
+
+/*
+ * Insert a message at the end of the queue.
+ */
+ void
+queue_push(type, data)
+ MessageType type; /* Type of message */
+ void *data; /* Data associated with the message */
+{
+ int empty;
+ message_T *msg = (message_T *)alloc(sizeof(message_T));
+ msg->type = type;
+ msg->data = data;
+ msg->next = NULL;
+
+ /* Acquire queue lock */
+ lock(&message_queue.mutex);
+
+ empty = message_queue.head == NULL;
+
+ if (empty) {
+ /* Put the message at the beginning for immediate consumption */
+ msg->next = message_queue.head;
+ message_queue.head = msg;
+
+ /*
+ * Queue was empty and consequently the main thread was waiting,
+ * so wake it up to continue after the lock is released
+ */
+ if (empty && pthread_cond_broadcast(&message_queue.cond) != 0)
+ pthread_error("Failed to wake the main thread");
+
+ } else {
+ /*
+ * There are pending messages, put this one at the end, adjusting the
+ * next pointer.
+ */
+ if (message_queue.tail == NULL) {
+ message_queue.head->next = msg;
+ } else {
+ message_queue.tail->next = msg;
+ }
+ message_queue.tail = msg;
+ }
+
+ unlock(&message_queue.mutex);
+}
+
+/* Take a message from the beginning of the queue */
+ message_T *
+queue_shift()
+{
+ message_T *rv;
+
+ lock(&message_queue.mutex);
+
+ if (message_queue.head == NULL) {
+ /*
+ * Queue is empty, temporarily release the lock and wait for
+ * more messages
+ */
+ if (pthread_cond_wait(&message_queue.cond,
+ &message_queue.mutex) != 0)
+ pthread_error("Failed to wait for condition");
+ }
+
+ rv = message_queue.head;
+ message_queue.head = rv->next;
+
+ unlock(&message_queue.mutex);
+
+ return rv;
+}
+
+#endif
diff --git a/src/message_queue.h b/src/message_queue.h
new file mode 100644
index 0000000..3b25d03
--- /dev/null
+++ b/src/message_queue.h
@@ -0,0 +1,34 @@
+/* vi:set ts=8 sts=4 sw=4:
+ *
+ * VIM - Vi IMproved by Bram Moolenaar
+ *
+ * Do ":help uganda" in Vim to read copying and usage conditions.
+ * Do ":help credits" in Vim to see a list of people who contributed.
+ */
+
+#ifndef MESSAGE_QUEUE_H
+#define MESSAGE_QUEUE_H
+
+typedef enum { UserInput, DeferredCall } MessageType;
+
+typedef struct message_T
+{
+ struct message_T * next;
+ MessageType type;
+ void *data;
+} message_T;
+
+typedef struct input_data_T
+{
+ int character;
+ int mod_mask;
+ int mouse_row;
+ int mouse_col;
+} input_data_T;
+
+void input_notify();
+void queue_init();
+void queue_push(MessageType, void *);
+message_T * queue_shift();
+
+#endif
diff --git a/src/version.c b/src/version.c
index 4a4bcd0..bcce586 100644
--- a/src/version.c
+++ b/src/version.c
@@ -335,6 +335,11 @@ static char *(features[]) =
#else
"-menu",
#endif
+#ifdef FEAT_MESSAGEQUEUE
+ "+messagequeue",
+#else
+ "-messagequeue",
+#endif
#ifdef FEAT_SESSION
"+mksession",
#else
diff --git a/src/vim.h b/src/vim.h
index 88f3dc2..678eaa0 100644
--- a/src/vim.h
+++ b/src/vim.h
@@ -1978,6 +1978,10 @@ typedef int VimClipboard; /* This is required for the prototypes. */
#include "globals.h" /* global variables and messages */
+#ifdef FEAT_MESSAGEQUEUE
+# include "message_queue.h" /* message queue API */
+#endif
+
#ifdef FEAT_SNIFF
# include "if_sniff.h"
#endif
