The information to write policy files for dbingest is contained the
help for the command-line scripts. But they cannot be installed and
run until MySQLdb is available, so here is a copy of the information.
This may change over time, so please always try to get the help
directly if possible.
-- Russell
$ dbingestinit.py --help
usage: dbingestinit.py [options] policyFile
The policy must include these keywords:
- dbHosts: a list of database hosts
- dbUser: username for database
and may include the following keywords (defaults listed in []):
- dbPasswd ['']: password for database
- dbPort [3306]: port for database
- dbPrefix ['dc1_']: prefix for database table name;
the full name is <dbPrefix>YYYYMMDD (gmt date)
- dbSocket [None]: path to socket file
- dropIfExists [False]: if True, drop the database if it exists; otherwise fail
options:
--version show program's version number and exit
-h, --help show this help message and exit
-t, --test run in test mode (fake all database communication)
$ dbingestworker.py --help
usage: dbingestworker.py [options] policyFile
The policy must include these keywords:
- slices: a list of slices to process
- rootDir: root directory of slice directories
- dbHost: IP address of database server
- dbUser: username for database
It must also include one of the following two parameters:
- schemaFile: path to text file consisting of the table schema
where the table name is coded as "%(tableName)s"
or
- templateTable: name of existing template table
It may also include the following parameters (defaults listed in []):
- dbPasswd ['']: password for database
- dbPrefix ['dc1_']: prefix for database table name;
the full name is <dbPrefix>YYYYMMDD (gmt date)
- dbPort [3306]: port for database
- dbSocket [None]: path to socket file
- dirPrefix ['']: prefix for name of each slice directory
- dirSuffix ['']: suffix for name of each slice directory
- createTables [True]: create database tables?
- tablePrefix ['']: prefix for name of each slice table
- tableSuffix ['']: suffix for name of each slice table
- endFileName ['done']: name of file indicating processing is done
- sleepTime [1.00]: time to sleep if no files found (sec)
- loadCmdFile: file containing LOAD DATA INFILE command where:
- the data file path is coded as "%(filePath)s"
- the table name is coded as "%(tableName)s"
The following command is used if no file specified:
'LOAD DATA INFILE "%(filePath)s" INTO TABLE "%(tableName)s"'
which means MySQL's defaults are used, including:
- all columns must be specified, and in order
- values are tab-separated ASCII
- lines are terminated with \n
- strings are not delimited
- NULL is represented by \N
Note: if your schema file or load command file includes a %
(other than for the special names listed above) you must double it.
For example 100% must be listed as 100%%
options:
--version show program's version number and exit
-h, --help show this help message and exit
-t, --test run in test mode (fake all database communication)
rowen:~/LSST/NCSADC1Repos/src/mw/das/ingest/dbingest rowen$
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
