Keywords: DataAccWG
Attendees:
Jeff Kantor
Tim Axelrod
Maria Nieto-Santisteban
Ani Thakar
Sergei Nikolaev
Kem Cook
Serge Monkewitz
Dave Monet
Leif Wilden
Deborah Levine
Jacek Becla
revised nightly processing use case
===================================
synchronizing catalogs:
- full synchronization every time we run deep detection
- deep detection will run for long time, so might start
the transfer while deep detection running
[after-meeting thought: doesn't it mean we need to
plan on having extra space for this incoming data?
We will not be able to wipe out the old catalog
until the entire new catalog is transfered (days)]
- might need a special day when things are down
to install/validate new version of catalogs
- during full sync, archive data is treated as master
- every night:
- transfer changes to main archive, use it to QA
data, then transfer changes (if any) back to base camp
- still need to better understand what to do if
there are differences
For completeness, usecase needs to mention
- synchronization of Object catalog in "prepare phase"
- interaction between IPP and database
(yes, it is lightweight activity)
Deep detection cycle: twice more frequently than release
cycle. Why? Because every few weeks we will have enough
new epochs to make it worthwhile
- deep detection is the largest ingredient of making
data release. Why not call each deep detection run
a 'release'?
- have not worked through main archive in details yet,
so for now accept the frequency, and we will come back
to this issue in the future
The use case assumes we are doing cross matching against
entire object catalog, not var object first
mops:
- current baseline: mops will run at the base.
- concern: cpu intensive, very slow (at least at the moment)
Assume everything from the Object catalog might be
needed by Association Pipeline
- info in all filters, all attributes... for all 10 million
objects per field
- 10 million objects per field is the worst case
- yes, some fields will have 10 million objects/field
even in DR1
- worst cases will be "clustered" e.g. in July when
nights are long
- need to look at db load from 2 perspectives:
worst and average
- aver: 4 million objects per field
The use case needs to have more quantitative numbers,
e.g. number of sources database will need to deal with
- Tim/Kem will produce a table (external to uml),
- use case will reference it
Keep DIASource at Base for ~ 6 months, it is < 10TB
- [discussed at the last schema meeting, see notes]
--> yes, make it a baseline
For every detected source, there will be an entry
in Object table, or in Junk Collection
Need to do pruning of some objects after mops done,
- turnaround: 24 h is sufficient, so could do it
in main archive, but since our current baseline assumes
we will run mops at the base camp, do pruning at the base.
- it is only few K objects per field
association pipeline assumes there is 1-to-1 (object-to-source)
mapping
- yes, the current algorithm is: take the best match
(closest distance)
--> need to make it clear in user case
- currently in the schema: Source table allows a single source
to point to one Object. Sufficient.
revised database estimates
==========================
- not finished, Kem/Tim still working on input numbers
- current latest version based on rough assumptions:
- 50% less objects/sources,
- 10% of measurements stored individually
plus
- 1 year release cycle
- don't keep data that is on disk in deep storage
- don't store indexes for old releases
- numbers look encouraging so far:
- comparable db size to what we used to have prior
to schema meeting
- twice more disk needed (because releases are longer)
- twice less deep storage needed
- less spindles to handle query load (i/o)
- pending changes
- square deg 15,000 --> 24,000
- precise number of stars and galaxies per image
- % of high signal-to-noise for stars and
for galaxies (now using a guess 10% for both types)
- model galactic plane region (less epochs, but higher density)
What signal-to-noise is worth saving in Source table?
(as individual measurements)
- 2 sigmas is too aggressive
--> 3 sigmas
At the moment we are assuming 10% of measurements saved in db
- out of the remaining 90%, ~50% will be very faint,
difficult to decide whether it is galaxy or star,
so we would have to keep extended attributes for that 50%
- if we can afford keeping extended attributes for
all measurements, we should do this (currently it looks
like we might be able to afford it)
real time updates of Object table at the Base Camp
==================================================
discussion around proposal outlined in:
http://www.slac.stanford.edu/~becla/tmp/ObjectTableUpdatesAtBase.doc
- can't split per filter, association pipeline will
need information related to all filters
- so we have to read info for all filters, but
will write back info for a single filter
- need to use new number of fields: 2500
--> update the doc: redo the calculations
providing efficient query access to the Source table
====================================================
discussion around proposal outlined in:
http://www.slac.stanford.edu/~becla/tmp/SourceTableAccess.doc
- we all agree the primary clustering should be
"all measurements from all epochs related to a single
object stored together".
- Should not call it "temporal clustering" though
- such clustering comes naturally for Source table (Source
table is filled by Deep Detection Pipeline), but
much more difficult to implement for DIASource
- no major objections to proposal, some suggested
possible modifications:
- could try to assign sourceIDs based on position,
then we would not need to go through object table
- perhaps the mapping table could be replaced by
index
- mapping table is good because different sources
corresponding to the same object might have very
different positions, and mapping table will allow
them to be clustered together
- if we cluster sources based on objectID + time
we can achieve the same
- the approach has some valid merits, but it is too
early to use it for baselining
- it is for user queries, not pipelines, we should
focus on pipelines and come back to this issue later
other topics
============
- dates of future "big" meetings
- Dec 1 and Jan 12, 9-12 PST
- will try to do one smaller meeting before
Dec 1 (regular DataAccWG / DB telecon style,
1 hour, audio only)
- also will be trying to make progress by email
- query matrix
- keep the queries generic (no specific numbers),
but will ask people to provide reasonable ranges of values
thanks,
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
We will have a DataAccWG telecon this Friday at 8:30 - 10:30 PST
(note, the unusual time!)
We will continue discussion started at the
schema/x-match meeting in Tucson on Oct 18-19.
Agenda:
- revised nightly processing use case
- revised database estimates
- real time updates of Object table at the Base Camp
- providing efficient query access to the Source table
- dates of future meetings
Relevant reading:
- email "nightly processing usecase" sent by Tim on 10/26 to lsst-data
- http://www.slac.stanford.edu/~becla/tmp/ObjectTableUpdatesAtBase.doc
- http://www.slac.stanford.edu/~becla/tmp/SourceTableAccess.doc
call-in ip address for video: 140.252.1.34
dial-in: 866 330 1200 passcode 300 2363
We expect most of the participants of the Tucson meeting
to use video.
Everybody who is interested in these topics is welcome to join!
Jacek
p.s. In case of problems with video, you can try to talk to
Iain: direct line: 520.322.8732, cell: 520.401.0362.
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
