I've committed two files to support dbingest's of source data for DC1:
src/mw/das/ingest/schema/sourcetable.sql
- Schema for the source data table for DC1
in the format used by dbingestworker.py
- This is simply the Source table from
src/mw/das/ingest/schema/schema.sql
with changes to simplify import of fake data:
- sourceId changed to auto_increment
- no foriegn key relations
src/mw/das/ingest/schema/loadsourcecmd.sql
- LOAD DATA command for loading source data for DC1
in the format used by dbingestworker.py
This includes all columns except:
- sourceId, because it is auto-incremented
- position because it is of type POINT and I have no idea how
(or if) it is possible to specify data for that in a data file
These are not tested, though I hope to do so later today or tomorrow
(locally, not on teragrid or lsstarchive).
I also improved the help in dbingestworker.py to get the quoting
right for schema and load data files. The updated help is as follows:
-------
$ dbingestworker.py --help
usage: dbingestworker.py [options] policyFile
The policy must include these keywords:
- slices: a list of slices to process
- rootDir: root directory of slice directories
- dbHost: IP address of database server
- dbUser: username for database
It must also include one of the following two parameters:
- schemaFile: path to text file consisting of the table schema
where the table name is coded as %(tableName)s
or
- templateTable: name of existing template table
It may also include the following parameters (defaults listed in []):
- dbPasswd ['']: password for database
- dbPrefix ['dc1_']: prefix for database table name;
the full name is <dbPrefix>YYYYMMDD (gmt date)
- dbPort [3306]: port for database
- dbSocket [None]: path to socket file
- dirPrefix ['']: prefix for name of each slice directory
- dirSuffix ['']: suffix for name of each slice directory
- createTables [True]: create database tables?
- tablePrefix ['']: prefix for name of each slice table
- tableSuffix ['']: suffix for name of each slice table
- endFileName ['done']: name of file indicating processing is done
- sleepTime [1.00]: time to sleep if no files found (sec)
- loadCmdFile: file containing LOAD DATA INFILE command where:
- the data file path is coded as '%(filePath)s'
- the table name is coded as %(tableName)s
The following command is used if no file specified:
LOAD DATA INFILE '%(filePath)s' INTO TABLE %(tableName)s
which means MySQL's defaults are used, including:
- all columns must be specified, and in order
- values are tab-separated ASCII
- lines are terminated with \n
- strings are not delimited
- NULL is represented by \N
Note: if your schema file or load command file includes a %
(other than for the special names listed above) you must double it.
For example 100% must be listed as 100%%
options:
--version show program's version number and exit
-h, --help show this help message and exit
-t, --test run in test mode (fake all database communication)
-------
I'm not entirely sure how dbingest will handle the auto-incremented
primary key because it only uses the schema file to create the first
table; it then duplicates the schema of this table to create the
other tables. But at worst I suppose the primary keys may start at
odd values for the tables.
To answer David Fleming's question (in case this doesn't do the job
already), I think I've done what was requested at the phone con. If
you now produce data files that contain tab-separated values that
match the columns in the schema (omitting sourceId) your data files
should be fine for testing ingest.
Here is an example:
1 1 1 1 0.0 0.0 1 0.1 0.1
100.0 120.0 0.1 0.1 1.0 0.0 0.0 1.1
1.1 0.0 1000.0 1.0 1.0 0.1 1.0 0.1
5.0 1.0 0.1 0.0 0.1 0.0 0.1 0.0
0.1 1.0 0.1 0
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
