Keywords: DataAccWG
Attendees: Nicole, Russell, Ray, Sergei, Ani, Maria, Kem, Jacek
schemas
=======
Sergei has preliminary version, working on the remaining issues,
need couple of more days.
1) issue of linking detections to algorithm runs:
should we link sources to images or to algorithms
that produce sources?
--> we need the links to both: images and algorithms
2) working on adjusting schema to conform to sql coding standards
recently published in Docushare
3) adding support for Policy (just the hooks in the schema
at this point. Related to Ray's policy doc circulated recently)
4) Dangling references
Issue brought up by Jim Gray via emails, many tables not linked.
Need to clean that: review how we use foreign keys
5) It was brought up that Zeljko expressed some concerns
about different types of object tables
- examples of such tables: Supernova, VarStar, CV, AGN.
- at the moment:
1) only objects confirmed with confidence level
greater than x% will make it to these special tables
2) these tables keep copies of objects, the "big" table
with everything keeps all these objects too
3) created because it might be convenient for researchers
interested in for instance supernova only
(much smaller that 'everything')
- but classification may change
- also, it means more "housekeeping"
- how fine do we want to be?
--> better to rely on the big table with all objects,
and provide means to extract a subset if someone wants
to take a subset of data and work offline
--> a database 'views' on top of the big table might help here
- not clear how efficient/inefficient views are, comparing
to separate tables. Views are convenient, but likely
inefficient, since data is still fetched from the big table
--> action item (not urgent): understand the trade-offs
between view-based approach and separate-tables approach
--> for now: replace these Object-type tables with views
- can EA model views ??? ( <--action item)
What is the difference between policy and provenance?
- policy: class that contains parameters that will be used by pipeline
- provenance: info about parameters that have been used.
Provenance is a superset of policy: it includes other things
(like start/stop time)
Issue of primary index for AmpImageTable
(see email discussion after the last db telecon)
- the bigint does not have to be 'unsigned',
'signed bigint' sufficient
- the main issue: should we use composite key
and get rid of the "bigint auto_increment"
--> yes
ingest tests
============
JHU:
- redid float/decimal tests
- decimal less efficient than floats/double
- need to do similar test with mysql
- Maria planning to try it at jhu, installing
mysql now
SLAC:
- restaged data from hpss (disks reformatted during tuning)
- not much progress with ingest tests beyond that
DC1
- planned throughput: 2% of 1 night, 1 night = ~1 TB of
catalog data, so we are talking about ~2GB. If we divide
it into 200 streams (1/ccd) it is not much.
Likely will need to readjust DC1 to stress ingest better
Need to discuss the usecase of pre-staging object data
for the base
- how do we "glue" this with the 200 tables used for ingest?
- prestaged data in a single database? On a single server
or partitioned? If partitioned, then how to partition?
--> need to understand amount of object data that needs
to be cached at the base per night
- we know the schedule of the camera for given night,
we know which ccd will be looking at which piece of
the sky throughout the night, so we could partition
object data
- that is inflexible, the schedule may change
in the last minute..
AOB
===
write access in Docushare
- Maria managed to change ingest doc Jacek put in Docushare,
others should be able to do that too
Next DataAccWG (not db) telecon
- Tim/Ray/Jacek not available
- maybe move to Wednesday next week
- Nicole, Ani can't make it next Wed
Database telecons through video?
- once we try DataAccWG telecon
- JHU: inconvenient, will use phone lines
- UW: can join (for Database telecon, not DataAccWG telecon
- NCSA, SLAC, LLNL: can join
- don't always have participants from lsstc, but could
use the fact that MiddlewareWG telecon sets things up
just before us
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
Reminder: we have our regular Database telecon
tomorrow (Wednesday) at 11:00 PST.
Phone number: 866 330 1200
passcode: 300 2363
Fairly light agenda:
- update / status:
- schemas
- ingest tests
- adopting sql coding standards
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
