Hi All,
This has been a low-level fantasy of mine for a while and I may soon have
the funding to pursue it fulltime (at least for a few months).
The big picture idea is to integrate Perl into DX so that DX can make use
of Perl's DBI database connectivity and text-parsing capabilities. This is
in support of a large gene expression project and to organize some of the
data requires considerable text-parsing of annotations and supporting data.
The data will be stored in a PostgreSQL database that has a considerable
amount of support code written in Perl and Perl's DBI (DataBase
Independent) module. That storage mechanism and supporting code makes it
very attractive to be able access it from DX (used because of DX's very
nice numeric and visualization &rendering capabilities).
So some reality checks before we begin:
I've seen the Oracle-DX interface in the DX showcase (Western Wildfire
datasets), but since we're not using Oracle and we do need Perl, I'm not
keen about porting it to Postgres.
This whole approach seems very similar to the inclusion of Perl in Apache
as mod_perl, and that approach is pretty well described and documented.
Are there other things that will have to be specially watched for in DX?
There are 2 things especially that concern me:
1) As I understand the module approach, I could link the module C code with
the Perl lib or I could try to link the lib with the main executive. Is
there an advantage to either approach?
2) The data generated by the Perl calls to the DBI to query the database
can return data in a number of Perl variables (lists, arrays, hashes).
What is the best way to return data from those structures to that required
for further processing by DX? The simplest way is of course to structure
the data and write them to disk as netCDF files and then explicitly have DX
read them, but this has a horrible I/O downside (and obviates the whole
integration effort).
The 'standard' way (if there is such a thing) is to create the structure
in-memory and then convert it to a C structure via Perl's scalar conversion
utilities which could then format (?) the Perl creation into a
DX-compatible struct. There is a Perl (PDL) netCDF writer so I COULD
maybe use that to create an in-memory netCDF-like structure then pass it to
DX via one of these conversion utility functions.
Comments about these particular points or critiques of this approach in
general are welcome.
--
Cheers, Harry
Harry J Mangalam - 949 856 2847 (v&f) - [EMAIL PROTECTED] (primary)
[EMAIL PROTECTED] (big attachments) <<plain text much appreciated>>
