This is an excellent idea!
There are three candidate mechanisms for GridFTP servers to publish
data on recent transfer performance:
1) The local log file
2) A WS-ResourceProperty-based mechanism (you refer to this as MDS,
though it's actually more general)
3) The built-in usage reporting mechanism (aka Metrics mechanism)
The first option (local log file) would not be very useful for the
system you describe because it doesn't provide client access to the
data.
The second option would be useful but I would suggest a few
modifications to the implementation that you described. First,
GridFTP isn't a web service, so you'd need to have a Web service
container running in parallel with the GridFTP server *or* (and this
would be a lot of work) use the C WS Core code to add a WS-
ResourceProperty interface to the C code. In either case, you would
need to model a Web service with state so that there could be WS-
Resources and associated ResourceProperties. An elegant way to do
this would be to have a Web service that represents the GridFTP
server as a resource, and then publish ResourceProperties that
provide access to the recent transfer history. You could then use an
MDS4 Index service to aggregate the data from multiple GridFTP
services. None of this exists today, so it would all have to be
developed from scratch, but it seems to me to be a very useful line
of work. (I could imagine this being the core for a more ambitious
effort to provide a WS management interface for GridFTP servers that
allows remote users with appropriate authorization to manage the
GridFTP server as a resource: create a new service, reconfigure the
service, shut down the service, see server status, etc.)
The third option (Metrics mechanism) already exists: the GridFTP
server reports data for each transfer already using the Metrics
mechanism (http://www.globus.org/toolkit/docs/4.0/Usage_Stats.html).
The server can be configured to report this data to multiple listener
services. (I.e., you could run your own local listener service as
well as reporting to the world-wide Globus listener service.) Once
the data is in a database that you can access, you can pull the data
from the database to make predictions about expected performance.
The primary caveat with this line of attack is that the usage reports
currently do not include the location of the remote service or
client, so you'd only know what one end of the transfer was (the one
with the local server). However, unless I'm much mistaken, the
GridFTP team has already written the code to provide the info on the
remote end of the transfer and this code is either available in the
current development release (4.1.2) or else it will be available in
the next development release (and then later in GT 4.2).
-- Lee
On Oct 29, 2007, at 12:02 PM, gaurav khanna wrote:
Hi all,
I am trying to use the MDS framework to set up a customized
information
service as follows.
For every globus-url-copy call, the globus-url-copy client should
send the
following information to a centralized MDS. The information should
include
a) For a two-party transfer, the client url, server url, file size,
time
taken to transfer
b) For a three-party transfer, the source server url, the destination
server url, file size, time taken to transfer
Every subsequent globus-url-copy should be able to then access this
MDS
and perform some informed decisions.
To do this, I am trying to write a ExternalElementProducer
Information provider. This requires specifying the location of a
script
in the file rp-provider-config.xml and to specify the periodicity with
which it will run and populate MDS. However, for my purpose,
a globus-url-copy call should send the data to MDS as and when it is
called. So, this is more like an on-demand push based data publishing
model instead of a pull based model which seems to be used when a
script
runs periodically and MDS gets the
data from it.
Essentially what I need is some C API which when used within
globus-url-copy, would transfer or push the above mentioned data to
the
MDS. Is it possible to do that in Globus ?
Regards
Gaurav
Gaurav Khanna
Phd Student
CSE Department,OSU
Phone (office):614-292-7036
