Hi,
So I'll try to describe here what I'm thinking about the core to UI
communication protocol that may be used by the shell interface and other
eventual UI frontends to gtkg.
The following, of course, reprensents only my views and are not meant to
be inflicted on you guys in any way. Comments, suggestions, and
improvements are always welcome. If you think something is missing or
plain stupid, please let me know.
Raphael, I know about your dislike of XML, and I understand as sometimes
I don't recommend it either, so please read on and tell me what you
think :)
Let's get a little structure from now on.
** Contents **
1. Intent
2. Notification Model
3. Subscription Model
4. Data Representation
5. Sample : struct dl_queued
6. Data Structure
7. Sample : struct download
8. Query Format
9. Sample Queries
10. Implementation Details
11. Final Words
** 1. Intent **
In the lengthy process of seperating core from UI in gtkg, some way must
be designed to take orders from the UI, perform actions in the core, and
notify the UI of changes that occurred since last update.
Richard proposed a subscription/notification model : the UI would
request updates on a given topic and the core would provide it from then
on. This idea interested me a lot because of its inherent portability to
different kinds of frontends (including raw shells or handcrafted
scripts if needed).
Taking from here, Richard is designing the remote shell module of gtkg,
which now provides a way for the UI to issue orders to the core. This
document's intent is to extend it by providing a generic XML-based
subscription/notification protocol we may use to implement the tricky
part of Richard's model.
** 2. Notification Model **
When we eventually complete the core/UI split, we may end up with a
daemon that supports talking to multiple UIs at the same time, and/or
multiple daemons listening on different ports each talking to its own
UI. This should not be a concern to the protocol.
The core needs to notify the UI when something that was subscribed has
changed since the last update. Notifications should be asynchronous, ie
not needing a command from the UI for each request. The computation time
and bandwidth use for the notification process should be as efficient as
possible.
I propose that the notification core queues events and fire them all
from time to time to the UI. In addition, only events that are
subscribed by the UI have to be queued. And if there are multiple UIs
running, only the subscribed data items have to be sent on a per-UI
basis.
So we need a data structure that is network-practical, aggregatable, can
be processed and searched efficiently, and can be easily matched against
some subscription token.
There are only a few structs in gtkg that participate in UI updating. We
can translate them into XML nodes, aggregate them (if they are
subscribed) as the notification events fire, and send the whole XML
packet to the UI when we decide to update. It would be even better to
send only the requested elements in the XML packet, but this may take
unnecessary processing time, so we'll have to compromise. More about
this later on.
** 3. Subscription Model **
We can use Richard's remote shell to send update queries to the core.
The real question here is to find a way for the UI to request only a
subset of all the possible changes, such as :
* Hi, I'm a multiview local GUI running under X, please notify me
about everything happening to you at all times. Bandwidth is not
a concern. Thanks.
* Hi, I'm a remote ncurses UI relying on fullscreen panes. Right
now I'd like the search stats and nothing more, but when my user
presses PgUp I will require the gnet nodes' state and nothing
more. Please help me.
* HELO, I'm a remote human-operated telnet and I'm here for
maintenance/debugging. I'll require exactly those and those
pieces of data, and please don't die on me.
* H3L0, I'm a custom script designed to filter searches based on a
whitelist database. Only search hits and filters interest me, so
don't lose time computing anything else.
Yes, a little humor doesn't hurt these days. All those situations have
to be taken to our advantage in the core, spending only the resources
necessary to provide requested data, maybe slightly more if needed but
by all means we don't have to spend time computing XML packets for gnet
nodes when the only things the UI cares about is the current state of
download 0xff7e3c80.
This is actually the main reason I wish to use XML on the problem. If we
use XPath expressions for update queries, not only can we handle all
those crazy situations and more, but we can also optimize and build only
the XML packets matching the queries. In a multiple UI context we can
aggregate the whole changes in a single XML tree and match it against
requested XPaths to provide each UI with only requested content, thus
saving bandwidth (maybe too CPU intensive though, see implementation
details below).
Finally, the update request may specify its own refresh rate, or the
core may decide to update any UI on a given interval. I still don't know
whether is better, but I don't think it matters much.
** 4. Data Representation **
The current gtkg UI frontends use the same structures as the core to
perform updating. We want to have a generic notification system to gtkg,
so sticking to the core structures is the right way. We have to
translate them to XML, but we don't lose flexibility. Some struct
members, however, are useless for UI updates and should not be
translated.
We can use the standard method of using an element for each struct and a
attribute for each member (the pointer members are part of the document
structure and discussed below).
The transformation process from gtkg structs to XML packets must handle
atom resolving, and probably enum translation (int to string) as well.
There is a wall here that needs to be set on what UI updating must be
performed by the core (such as download state string formatting) or by
the UI (such as counting the active gnet nodes ratio).
** 5. Sample : struct dl_queued **
Struct dl_queued is used to store the queue state of a download. I'll
use it here as a sample because of its flat nature (no pointers).
It is currently declared as :
struct dl_queued {
guint position; /* Current position in the queue */
guint length; /* Current queue length */
time_t ETA; /* Estimated time till upload slot retrieved */
guint lifetime; /* Max interval before loosing queue position
*/
guint retry_delay; /* Interval between new attempt */
gchar ID[PARQ_MAX_ID_LENGTH+1]; /* PARQ Queue ID, +1 for trailing NUL */
};
Here are two XML representations (among many more) suitable for such
data. Data types enclosed within parenthesis are substituted for values
or struct members.
First the "intuitive" one :
<dl_queued position="(uint)" length="(uint)" ETA="(ulong)"
lifetime="(uint)" retry_delay="(uint)" ID="(string)"
/>
Then the "awfully flexible" one :
<struct name="dl_queued">
<prop name="position">(uint)</prop>
<prop name="length">(uint)</prop>
<prop name="ETA">(ulong)</prop>
<prop name="lifetime">(uint)</prop>
<prop name="retry_delay">(uint)</prop>
<prop name="ID">(string)</prop>
</struct>
I like the first approach much more than the second one, since XPath
query expressions are much simpler, structuring is a lot easier (see
below) and the added flexibility is not really needed in the first
place.
** 6. Data Structure **
We need two more attributes in order to give structure to the XML
packets. One of them is a unique identifier that can be used to refer to
this particular object from the UI (the adress of the object can be
used, but may cause problems with memory allocators reusing addresses
between updates).
The other one is the object "state", or relation to the UI. There are
three states : NEW means it's the first time the UI sees the object,
MODIFIED means an existing object has been changed in some (subscribed)
way, and REMOVED means it's the last time the UI sees this particular
object (though the identifier may be reused later for another object).
Not every structure we need to pass to the UI is as trivial as
dl_queued. Many contain pointers to other structures as well as lists
storing subdata. XML handles lists quite well, pointers being another
thing (and XPointer is not finished yet AFAIK).
I'd handle lists using container elements, and since they contain
pointers, the problem remains the same, and depends on whether the
pointer is shared or not :
* If the pointer is not shared, we should render its XML
representation inline within the parent structure element.
* If the pointer is shared, we should render its XML
representation in a shared packet node and insert an XPath
attribute in its parent. We should only render the XML once.
* We don't do anything if the pointer indirection is not requested
by a subscription.
** 7. Sample : struct download **
Struct download is a good example of a structure used in UI update
notifications. And is contains both shared and unique pointers.
It is currently declared as :
struct download {
gchar error_str[256]; /* Used to sprintf() error strings with vars */
guint32 status; /* Current status of the download */
gpointer io_opaque; /* Opaque I/O callback information */
bio_source_t *bio; /* Bandwidth-limited source */
struct dl_server *server; /* Remote server description */
enum dl_list list_idx; /* List to which download belongs in server */
struct dl_file_info *file_info;
guint32 record_index; /* Index of the file on the Gnutella server */
gchar *file_name; /* Name of the file on the Gnutella server */
guint32 file_size; /* Total size of the file, in bytes */
guint32 size; /* Total size of the next request, in bytes */
guint32 skip; /* Number of bytes for file we had before
start */
guint32 pos; /* Number of bytes of the file we currently
have */
guint32 range_end; /* First byte offset AFTER the requested range
*/
struct gnutella_socket *socket;
gint file_desc; /* FD for writing into downloaded file */
guint32 overlap_size; /* Size of the overlapping window on resume */
time_t start_date; /* Download start date */
time_t last_update; /* Last status update or I/O */
time_t last_gui_update; /* Last stats update on the GUI */
time_t record_stamp; /* Stamp of the query hit that launched us */
guint32 retries;
guint32 timeout_delay;
const gchar *remove_msg;
guchar *sha1; /* Known SHA1 (binary atom), NULL if none */
guint32 last_dmesh; /* Time when last download mesh was sent */
GSList *ranges; /* PFSP -- known list of ranges, NULL if none
*/
guint32 ranges_size; /* PFSP -- size of remotely available data */
guint32 sinkleft; /* Amount of data left to sink */
guint32 flags;
gboolean keep_alive; /* Keep HTTP connection? */
gboolean visible; /* The download is visible in the GUI */
gboolean push; /* Currently in push mode */
gboolean always_push; /* Always use the push method for this
download */
struct dl_queued queue_status; /* Queuing status */
};
This is a huge structure, and many members don't need to be sent to the
UI. In addition queue_status is an embedded structure, and file_info is
a shared pointer.
So here is an excerpt of what we would send to the client, if it
requested the whole content of the download (let's suppose it's a new
download) :
<downloads>
<download object-id="(adress of download object or other identifier)"
object-state="NEW" status="(enum)"
<!-- this is a shared pointer, store a reference -->
file_info="/refs/[EMAIL PROTECTED]'(file_info identifier)']"
record_index="(uint32)" file_name="(string)" file_size="(uint32)"
<!-- this is an embedded structure (or a unique pointer), store it inline -->
<dl_queued position="(uint)" length="(uint)" ETA="(ulong)"
lifetime="(uint)" retry_delay="(uint)" ID="(string)"
/>
</download>
</downloads>
<refs>
<!-- here is the unique definition for the file_info shared pointer -->
<dl_file_info object-id="(file_info identifier)"
flags="(uint32)" file_name="(string)" path ="(string)"
<!-- blah, blah, you get the idea -->
<alias-list>
<alias object-id="(alias identifier)" value="(string)"
/>
<alias object-id="(alias identifier 2)" value="(string)"
/>
</alias-list>
<!-- and so on -->
/>
</refs>
** 8. Query Format **
XPath query strings should be used to subscribe to some part of the
queued XML tree generated during changes. The XML tree should be rooted
with an element called say <ui-update>.
The query string should contain a valid XPath expression refrerring to
the XML subtree the UI is interested in. Multiple subscriptions should
be allowed and cumulated, either by "merging" the XPaths or matching
them in order.
When the core sees a change occurring, it should match the queried
XPaths to check if the change should be translated into XML and
aggregated to the queue tree. When time has come to send the update to
an UI, we can optimize the XML elements we send by matching against
queried XPaths (but see below).
** 9. Sample Queries **
Here are some sample queries that can be achieved using XPath
expressions :
/ui-update/downloads/download
= give me everything about all downloads.
/ui-update/downloads/download/@*
= give me all the "inline" properties of all downloads.
/ui-update/gnet-nodes/[EMAIL PROTECTED]'ff75b080']
= tell me more about the gnet node 0xff75b080 from now on.
/ui-update/gnet-nodes/[EMAIL PROTECTED]'ff75b080']/@ip
= I only want the IP of this particular node.
/ui-update
= I want it all.
And of course, there are many more, given the flexibility of XPath.
** 10. Implementation Details **
First, helper functions and macros are needed to "easily" build a proper
XML representation from every involved structure. These should make the
whole "UI change->xpath query check->create only required XML" step as
transparent as possible.
If we want to optimize data sent back to the UI, we have to match the
XPath queries and send the cumulated results. Properly cumulating two
DOM trees is not cheap, so I'd rather cumulate all the XPath queries
from a given UI into an optimized-but-not-too-much superset of all that
was queried. But we still have to use the original queries to check for
XML content creation or we'll end up creating too much most of the time.
Actually the UI change functions in gtkg are quite identifiable and not
that numerous, so we can build the whole thing without disturbing much
code (yet). I suggest we create a kind of "third frontend" configure
option and branch from the same places in the code the gtk1 and gtk2
frontend branch into. If they can react differently to changes, so can
we :)
** 11. Final Words **
All feedback is welcome, of course. I hope I was clear enough, but don't
hesitate to email me for additional info.
That's all for today. Whew.
Cheers,
ko [junkpile at free dot fr on this platform]
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Gtk-gnutella-devel mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/gtk-gnutella-devel
