Comment #2 on issue 1961 by [email protected]: Export routine (MARC) for
Archives Canada
http://code.google.com/p/qubit-toolkit/issues/detail?id=1961
Hide
Permalink Edit Delete
Hutchinson, Timothy added a comment - 10/Feb/11 2:05 PM
Response from David, 15 Dec 2010:
Yes, it would be best to use a separate plugin for MARC as we are doing for
the Dublin core (plugins/sfDcPlugin) or EAD (plugins/sfEadPlugin)
templates, or any of our other meta-data standards based templates. EAD is
probably the closest analogue to MARC 21 in that it is a text-based
representation of an archival description, though the content is very
different.
I don't know MARC 21 at all well, but my intuition is that it would be best
to use one "sfMarcPlugin" for all MARC implementations (including XML) and
then use a different template file (e.g.
plugins/sfMarcPlugin/module/template/cain.marc21.php) to differentiate
between the different standards. The sfDcPlugin provides one example of how
to handle displaying the same standard (DC) in two different contexts (HTML
and XML).
As for your CURL implementation and how to make it more integrated with the
system, I would use the symfony task system [1]. Unfortunately the symfony
documentation is not up-to-date with symfony 1.4, but most of the
documentation for 1.2 is still relevant. We use tasks for migrating data
(lib/task/migrate) as well as the built in symfony tasks for data-load from
a yaml file, building table schema from the config files, etc [2] (Note:
many of the "generate" tasks and form builders do NOT work with ICA-AtoM
due to changes we've made to the framework).
The advantages I see to using the symfony task interface are:
* Avoid web application overhead while allowing direct access to the ORM
and other assets
* One access/failure point as opposed to using PERL + CURL + PHP
* No problems with client/server timeouts that may occur with the HTTP
transport, so scripts can run for many hours (we've had data-loads run for
days)
* A robust and simple interface for run-time configuration (e.g.
--turn-on-function)
* Can write an export file directly or through stdout
* Is CLI, so it can be called via crontab
The main hurdle with the using tasks for your export is you may still have
memory problems due to the way the Propel ORM works (specifically using
circular references) and how PHP does garbage collection [3]. However,
version 5.3 of PHP has an "experimental" new garbage collector [4] that is
supposed to work for cleaning up circular referenced objects. We haven't
had a chance to try out the new garbage collector yet, but you may want to
give it a try.
[1] => http://www.symfony-project.org/cookbook/1_2/en/cli
[2] => http://www.symfony-project.org/reference/1_4/en/16-Tasks
[3] => http://bugs.php.net/bug.php?id=33595
[4] => http://ca.php.net/gc_enable
--
You received this message because you are subscribed to the Google Groups "Qubit
Toolkit Issues" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/qubit-issues?hl=en.
