I hope this isn't too far off topic...
I am designing and plan to write yet another "spider" based on libwww. (Yes I
know there are many already.) This one is intended for internal management of
a large complicated web presence spanning many hosts and domains. Thus it
needs to know affiliated domains, hostnames, and related information, such as
a way to group hosts. (Intranet, Public, Partner site, the-rest-of-the-world,
etc.)
With the thought that some of this code may make its way to CPAN some day, I
would like to store this data in a form that offers maximum portability and
standardization. I know there are many Perl modules to interface with a
specific database, but databases aren't free and you can't count on everybody
having access to the same brand.
My current thinking is to write the data to pipe-delimited tables in flat
files, since they're free and exist on most every platform. I know this will
have drawbacks (how do you store data containing pipes, performance
considerations if the files get large, no record locking...) but at least one
could import the pipe delimited file into a database someday if desired. And,
there's always grep, cut, and sort for ad-hoc reporting. Well almost always.
;)
So I am soliciting comments on the wisdom of this course from more experienced
CPAN developers. Is there a better or Perl-standard way that would still be
fully portable and free? Thanks in advance for your thoughts.
