Hi Russell (and Ray and Robyn),
I was just speaking with Ray and I think we concluded that I would be
creating a tab-delimited text file of N number of rows. Each row could
be the same as the next. And that I would not provide an ID field as
that would be handled via autoincrement. My script would then be
callable from Robyn's python code. She would provide a value for N, and
I would create the text file, and be done.
But from the information you provide below, it looks like we're not
exactly on the same page, or are we? Thanks Russell. :-)
DF
Russell E Owen wrote:
The information to write policy files for dbingest is contained the help
for the command-line scripts. But they cannot be installed and run until
MySQLdb is available, so here is a copy of the information. This may
change over time, so please always try to get the help directly if
possible.
-- Russell
$ dbingestinit.py --help
usage: dbingestinit.py [options] policyFile
The policy must include these keywords:
- dbHosts: a list of database hosts
- dbUser: username for database
and may include the following keywords (defaults listed in []):
- dbPasswd ['']: password for database
- dbPort [3306]: port for database
- dbPrefix ['dc1_']: prefix for database table name;
the full name is <dbPrefix>YYYYMMDD (gmt date)
- dbSocket [None]: path to socket file
- dropIfExists [False]: if True, drop the database if it exists;
otherwise fail
options:
--version show program's version number and exit
-h, --help show this help message and exit
-t, --test run in test mode (fake all database communication)
$ dbingestworker.py --help
usage: dbingestworker.py [options] policyFile
The policy must include these keywords:
- slices: a list of slices to process
- rootDir: root directory of slice directories
- dbHost: IP address of database server
- dbUser: username for database
It must also include one of the following two parameters:
- schemaFile: path to text file consisting of the table schema
where the table name is coded as "%(tableName)s"
or
- templateTable: name of existing template table
It may also include the following parameters (defaults listed in []):
- dbPasswd ['']: password for database
- dbPrefix ['dc1_']: prefix for database table name;
the full name is <dbPrefix>YYYYMMDD (gmt date)
- dbPort [3306]: port for database
- dbSocket [None]: path to socket file
- dirPrefix ['']: prefix for name of each slice directory
- dirSuffix ['']: suffix for name of each slice directory
- createTables [True]: create database tables?
- tablePrefix ['']: prefix for name of each slice table
- tableSuffix ['']: suffix for name of each slice table
- endFileName ['done']: name of file indicating processing is done
- sleepTime [1.00]: time to sleep if no files found (sec)
- loadCmdFile: file containing LOAD DATA INFILE command where:
- the data file path is coded as "%(filePath)s"
- the table name is coded as "%(tableName)s"
The following command is used if no file specified:
'LOAD DATA INFILE "%(filePath)s" INTO TABLE "%(tableName)s"'
which means MySQL's defaults are used, including:
- all columns must be specified, and in order
- values are tab-separated ASCII
- lines are terminated with \n
- strings are not delimited
- NULL is represented by \N
Note: if your schema file or load command file includes a %
(other than for the special names listed above) you must double it.
For example 100% must be listed as 100%%
options:
--version show program's version number and exit
-h, --help show this help message and exit
-t, --test run in test mode (fake all database communication)
rowen:~/LSST/NCSADC1Repos/src/mw/das/ingest/dbingest rowen$
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
