Keywords: DataAccWG
Attendees:
Robin Allsman
Tim Axelrod
Jeff Kantor
Maria Nieto-Santisteban
Ani Thakar
Marcus Miller
Sergei Nikolaev
Ray Plante
Cristina Beldica
Nicole Silvestri
Jacek Becla
Proposed change to the data-ingest model:
run Catalog Ingest Service continuously,
do not start/stop every night
- useful in particular at the Main Archive, where
there will be some data to ingest most of the time
- Implications:
- a new event needed: "end of nightly data"
- garbage collection more complicated
- not sure yet whether Data Ingest Converters should
run continuously as well
- their lifetime should match the lifetime of
Processing Slices
Data Ingest Converter can run as a standard "stage" of pipeline
====
Merged the use cases Catalog Ingest Service for Base and
Catalog Ingest Service for Main Archive into one.
- ended up with only one different piece: taking care of the
ingest tables after the nightly ingest finished
- introduced a generic use case "Finish Nightly Ingest"
with 2 realizations: one for Base, one for Main Archive
--> look at the uml, think about it, will re-ask whether
this looks ok at the next telecon
What to do with loaded data after nightly ingest ends:
- at Base Camp:
- could just delete (but first need to make sure that
all the info related to alerts was transferred ok)
--> better to keep for several more nights, then purge.
- use different database name for each night
- at Main Archive:
- merge the data from ingest tables into the final,
large production tables, then delete
- need to add to uml an event "successfully ingested data",
with some stats what was ingested
How to get/return nodes?
- this is not an issue for DC1, in DC1 will statically
assign nodes
- ingest nodes are not part of the "mpi world"
- it is ok to assume that Ingest Service will have a pool
of ingest nodes assigned permanently
- still need to have resource management
returning nodes:
- more generic issue, not ingest specific
- need bigger discussion
- need resource management for each class of nodes
(ingest servers is an example of such class)
- one solution: each process should issue an event when
it starts and stops, system can react to that
Implementation of communication buffers is likely DC1 specific,
but the overall Data Ingest architecture hopefully more permanent
AOB
===
Should we continue with status reports?
- all info in mailing list these days (at least
on Andromeda side)
- will come back to that question when David is at the telecon
We will have a Database telecon next Wednesday
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
We have a DataAccWG telecon this Friday at 11:00 PST.
Phone number: 866 330 1200
passcode: 300 2363
Agenda:
- first 30 min: recent updates to Data Ingest use case
(see uml)
- second 30 min: skunk-meeting related discussion
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
