Keywords: DataAccWG
Attendees:
Tim Axelrod
Kem Cook
Ani Thakar
Nicole Silvestri
Deborah Levine
Serge Monkevitz
Maria Nieto-Santisteban
Ray Plante
Jacek Becla
debrief from mysqlcamp
======================
- "unconference"
- attendees:
- MySQL: both cofounders, director of architecture (our liaison),
key developers, ceo...
- user community: google, amazon, youtube, flicker, many others...
mysql @ google talk
- google seriously uses mysql (but not for bulk searches)
- 6 DBAs, 3 db developers
- they did not reveal any numbers how big there db is, or
how many servers they have
- we could learn a lot from google, especially things related
to running large dbms setup. They have hands-on experience with
running hundreds (thousands?) of mysql servers on cheap boxes.
- they deal in production with issues like fault tolerance,
hot fail over, redundancy, replication, automatic client redirection,
auto-load balancing, schema migration
- they built some useful tools on top of mysql (in python)
- offered some of them as open source
- they don't use foreign keys, allow dangling references
- google's dba comment about database vs bigtable:
"In bigtable you manage schema, there is no sql interface.
If you need to keep your options open: use database.
If you understand what you need really well, look into bigtable"
- I'm planning to meet with google DBAs over lunch to talk more
few other notes picked from discussions with user community
- mentioned by several people independently:
- very bad experience with expensive fancy hardware:
- high expectations, breaks more often then expected,
cheap hardware catches up quickly (the fancy hardware
no longer fancy after 2 years)
- much better in long term to keep it simple:
cheap hardware, build in redundancy, fault tolerance...
- v. difficult to get to work a mixture of long and short queries
(true for other systems, not mysql only).
--> better to separate them
- youtube offered script for parallel replica synchronization
notes related to discussions with mysql people
- mysql working on new storage engine, very good fit for us
(next generation myisam, transactional, bitmap indexes,
fully support of foreign keys...)
- this is still unofficial, will be publicized / announced
next year
- mysql will fix problems with precision (introduce "near" operator)
- mysql offering to write storage engine to bind mysql
with google's bigtable
- mysql director of architecture will be at slac early
December to talk to us about lsst db
debrief from prototyping platform / DC2 meeting
===============================================
- DC2: not throw away code, will evolve
- official dates: 9/30-11/30 2007, might be pushed 1 month earlier
- covers nightly processing
- no provenance
- no archive processing
- but we need to do some of it to prepare data for
nightly processing
- also, produced data need to be made available for users
- details of that to be understood...
- expected manpower:
- Maria/Serge: mostly on association pipeline / xmatch
- Jacek, Ani, Russell, new slac postdoc: prototype db
- prototype db: will be populated with real data from selected
existing experiments and sky-mapper
- ~night worth of data, target db: few times 10^9 objects,
10x more (likely somewhat less than 10x) detections
- that translates to ~ 8TB objects, 11 TB sources + indexes
- issue to think about: how to separate data from different surveys
(maybe in separate databases?)
- will need dedicated hardware for db servers, getting random
nodes assigned in the last minute through teragrid in not an option
discussion around some technical points brought during DC2 meeting
==================================================================
- maybe we should leave link from DIASource to Object when we
detect that it is a moving object, and just flag that it is
a moving object?
- fast moving objects may quickly move outside of region we
prepare for image processing
- moving objects should not be in object catalog
- scientifically it makes more sense to have moving objects
in separate table, not in Object table
- want some sort of special object catalog for things that are
uncertain (e.g. things that have single measurement)
- or keep these only in DIASource, and do not generate corresponding
object until we collect several measurements
- keep in Object catalog only objects that have multiple detections
--> Tim will think more about it...
We definitely will NOT deal at all with Sources at the base.
Only DIASource
Is most recent history of an object sufficient to generate alert?
No sources, no DIASources
This will happen immediately after we do deep detection and
resync Archive Center --> Base
[issues with telecon line here], email response sent by Tim:
"I have been making the assumption that the very recent history of an
object is important to making the alert decision. So, one needs the
compressed representation in the Object table + the most recent history.
This may not be right! We need to do some science-oriented
simulations to investigate this, I think."
Kem, Ani, Maria, Serge, Deborah and Jacek continued the discussion:
Maybe we should allow fussy edges between partitions/chunks rather
than reallocate rows between partitions
- the extra "row of chunks" addresses this
---> need to understand what is the good size for this safety "buffer".
don't try to force vertical partitioning now, assume all needed
- we could internally vertically split object table if there
are good reasons, but given that it helps with writes only,
and writes are <1% of what we read, not worth
what to do with one object becoming two objects?
Input from Robert at the DC2 meeting:
- introduce Object catalog
- column: "number of children"
- column: "parent object id". -1 if no parents
--> need to discuss
We need to look at schema: at the moment DIASource table has one
column for pointers to Object catalog. But it will sometimes
point to MovingObject table
- could we split DIASource into 2 tables: one for objects,
one for moving objects?
- no, leave it together, MOPS will be shuffling different
detections between different moving objects, at least
initially
--> need to come back to this issue
object table partitioning
=========================
chunking vs zoneId+ra index. The tradeoff:
- chunking: assigning chunkIds to objects not obvious
- zoneId+ra: probably reads more data than necessary,
because clustering for 2 of the 4 sides non-optimal
--> Jacek will think about assigning chunkIds to objects
--> then need to do tests and compare
- we don't have to find the perfect solution, at least not now
number of updates?
==================
How many updates will we have per visit?
During normal (steady) processing after DR1 or so, there will not
be that many updates. In most cases detected measurements will
carry exactly the information that was expected, in which case
there will not be anything to update (maybe 5-10% of detections
will need an update).
But initially, before DR1, "it will be a mess", everything will
be updated, re-shuffled, re-classified etc...db has to handle
that.
--> need to think how to design DB that will smoothly transform
from the initial-mess mode to steady-processing mode
We will have the next "big meeting" through video next Friday
Dec 01 9:00-12:00 PST. We will try to talk during DB telecon
slot next Wed prior to that meeting.
Happy Thanksgiving!
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hello,
Here is a brand new suggestion how to partition Object table
for the nightly processing use case:
http://www.slac.stanford.edu/~becla/tmp/objectTablePartitioningAtBase.doc
Hope we will have time to talk about it during the telecon tomorrow.
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi all,
We will have a Database telecon tomorrow (Wed) at 11:00 AM PST.
Agenda:
- debrief from the lsst-prototype-platform/DC2 meeting
- debrief from mysqlcamp unconference
- database architecture for nightly processing use case
Phone number: 866 330 1200
Pass code: 300 2363
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
