Hi Arek,
Thanks for the positive reply. It is particularly important to us that
you you have the policy of maintaining backward compatibility.
You say of counts: "For large datasets this frequently ends up taking
longer than the preview of the results for the actual query" - I don't
see this as a big problem, as the queries seem to run fast anyway. It is
more important to be able to get the answer, even if if it takes a bit
longer.
You answered preliminary ETA is April for BioMart 0.7, but didn't give
any comment about a published list of enhancements and bug fixes for
that release. Such a list would be really helpful - a list of known or
suspected bugs would save users time if they hit an already known bug,
as the first assumption of a user is that it is something he or she has
done wrong, and sometimes only after much time spent investigating will
it get reported to the mart-dev list.
For enhancements, I am particularly interested in extension of your
queries to cover the standard types of query supported by SQL, for
example "GROUP BY" (in conjunction with counts) and WHERE clauses that
have an expression involving more than one column (attribute). These
constructions often provide quick answers to common questions about the
datasets, like "how many proteins are there of each of the following
types...".
I await with interest a further response from Syed.
Regards,
Roger
Arek Kasprzyk wrote:
On 12-Mar-08, at 3:44 PM, Roger Hull wrote:
Hi Syed,
Hi Roger,
I'll let Syed reply to you in detail to your questions but let me put
this into the context of our future developments.
Thanks for the advice. I think we see quite big risks in doing
changes which depend on our understanding too much of how your code
works. This could present problems in maintaining the code we have
written, or changed, in the future, when BioMart is upgraded or we
need to add new features.
I would like to ask some questions about BioMart upgrades:
(A) Will you maintain backward compatibility between a BioMart 0.6
installation which gets its data from a remote BioMart MartService,
when the remote BioMart is upgraded to 0.7,..., 1.0, etc ?
We do currently maintain the compatibility between our 0.6 service and
all other marts which run earlier versions. The central sever is a
'translation point' which maintains backward compatibility.
Not all of the BioMart servers accessible from our central server are
0.6 but you can query them in uniform fashion. It is our intention to
maintain this compatibility in a similar way in the future.
(B) Do you have a date for BioMart 0.7, and have you published a
there a list of enhancements and bug fixes for that release?
the preliminary ETA is april
(C) There seem to be a number of BioMart installations where people
have modified your code. As I said, I'm reluctant to follow this
path, but I wonder if you could consider adding functions, hooks, or
similar mechanisms in your code so that the BioMart behaviour can be
changed in various ways without modifying your code? (The files
header.tt and footer.tt are useful, but it is limited what can be
done by adding code to these files.)
yes, the web code will undergo a major re-organization for 0.8 which
will coincide with the release of the new configuration system. One of
the main goals of this 're-organization' is to provide a flexible
framework by
which people can extend the code both in terms of web GUI but also
things like visualization etc. This is still at a very early stage so
any suggestions are very welcome.
A couple of suggestions, based on what I have been wanting to do:
(C1) I would like to call your AJAX functions for my own
purposes. But your function doAjaxMagic(toDo) only supports the two
values of toDo = 'countByAjax' and 'resultsByAjax'. Could you provide
a general purpose function which anyone could use? - with an argument
to specify the URL which will handle the request and another to
supply a function to handle the results (preferably supporting POST
as well as GET). Then use this function internally to implement
doAjaxMagic.
If I modify your current function, or make a modified copy, then I
have to maintain this code in the future if you change the function
internally (e.g. to support new browser versions).
(C2) Could you implement, and document, a callback function in
perl, which has as input arguments the results of a query in a
parsable format (maybe XML), so that by default this function returns
the input results unchanged. Then if the user wants to filter or
otherwise modify the result data, this can be done by adding code to
this function, and modifying the data before returning it to the
caller. [OK, it's not quite as simple as this, because you batch the
results data and return a certain number of result rows at a time,
and if some are filtered out, some more have to be processed to make
up the number, but I'm sure a solution can be found.]
When I looked at your code to see how and where I might do this type
of result modification, so that various formats (HTML, CSV, etc) and
various output methods (display in MartView, download to a file), are
all supported, it needed quite a lot of study of your code, and might
well present difficulties to maintain. You have a perl API based upon
BioMart::QueryRunner, which you use internally in your code, but the
only way I can see to get at the results using this API is
$query_runner->printResults(), which gives already formatted results.
I want to get at the results before formatting, so I don't have to
parse data formatted in various ways, and reformat the data after
modification.
Apart from filtering the results, another common requirement would be
to add links to fields when the results are to be formatted as HTML,
or to add another column of fields fetched from a local non-BioMart
database, using one of the BioMart attributes in the results as an
index to retrieve data from this database.
(D) As I mentioned earlier, the Count returned by BioMart is "how many
rows in the main table of the dataset match your filters so far". In
the case of PRIDE, this is the number of experiments. But often I need
to know the number of rows returned from the query, not the number of
rows from the main table - for example I want to know the number of
distinct proteins or peptides which match my filters. Will you support
a more usual "number of rows returned" count in the future?
There are two problems here: one the counts for multiple mains, the
second the count of the actual number of rows returned. As far as the
mains are concerned (which I think would solve your protein problem
for PRIDE) we do intend to provide a proper count for each main table
ei, your query has selected this many experiments, proteins etc ...
how many mains you happen to have there as oppose to what we support
now which is a top main. As far as the number of rows is concerned the
implementation is really trivial for single datasets and the only
concern there was the unpredictable performance. For large datasets
this frequently ends up taking longer than the preview of the resutls
for the actual query so it is not always works nice in an interactive
environment. A bigger challenge is to provide a row count for
federated queries. We are currently thinking of all possible scenarios
for all of them
a.
Regards,
Roger.
