Keywords: DataAccWG
Attendees:
Nicole Silvestri
Robyn Allsman
Ani Thakar
Maria Nieto-Santisteban
Ray Plante
Ghaleb Abdulla
Sergei Nikolaev
David Fleming
Jacek Becla
videocons
=========
- Will try to use video starting next time
- Robyn will be our LSSTC participant responsible
for patching in the phone line; on days when she
is not there, we will revert completely to the phone.
- should have at least 3 participants through video:
- ncsa, slac, lsstc, possibly uw, llnl and jhu too
- need to warn other WGs about patching in the phone line
passing data from pipeline to ingest etc...
==========================================
slicing the data
- one stream per ccd?
- one stream per amplifier? ("section of sky")
- what is natural for pipeline?
--> for DC1 most practical would be to parallelize
on CCD level: one processor/ingest stream per ccd
- so one Pipeline Processing Step per ccd, and
one ingest stream per ccd
- if we partition based on amplifier, the overhead of
managing all these files will start to dominate
- need to determine amount of data per ingested chunk
for DC1
- the baseline requirements mention average
#objects per detection
- Tim had that number, Robyn will find out
How the data will be "reduced" for DC1 (the 2% of production
throughput)
- smaller images? or full images but less frequently?
- Jacek, and Robyn always assumed it is "smaller images"
- Robyn heard Tim talking about "reduced images"
- need to clarify
Where a pipeline should put files for ingest?
How these files are transfered from machine to machine?
- for DC1 could write data to a shared file system (gpfs),
- have 2 directories, write to one, then mv file to
the other directory, consumer then grabs it
- alternative: through xrootd, Chris is testing xrootd now,
if he finds setting up xrootd is easy we might try it
for DC1
Clustering merged data at the archive?
- the partitioning will be done "strategically" during merging
data from temporary tables. That is for the archive,
at the base no need to merge things
- we do not really need to worry about "strategic partitioning"
for DC1. Main focus is on keeping up for alerts
- merging: can be included in DC1 if we have a coherent plan
- we need to merge data at the main archive before we
deliver it to consumers
- so do we have more than 24h to merge?
- data related to alerts will be transfered with high priority,
if we can ingest it first then we don't have the pressure,
could merge the rest less frequently than once every 24h
What is the lifetime of data in temporary databases used for ingest?
- the tmp databases will be scratched every night,
need to at least transfer data to the node doing merging.
policy class
============
We will have different "strategies" for staging data,
for instance "distributing data from ccds to different nodes"
- similar idea to what "policy" does for pipeline
- can we reuse Tim/Robyn's work on pipeline policies?
Ray working on a document discussing what he learned
so far about policies, etc
- should have the draft by the end of today
System should be able to adjust some parameters while
pipeline is running (human tweaks something through GUI,
e.g. when seeing goes bad).
- adjustment allowed and applied in real time?
- to be decided to what extend we are going to support it
(because it complicated things a lot)
- normally would just deal with static files that live
in policy library, but the run-time adjustments
complicate that, e.g. provenance
- provenance would now have to recode all the changes
of individual parameters. Then when we re-process,
we can't use just static files, have to go to provenance
--> we have to be restrictive when to allow things to change
- only allow changes at certain break-points
- for instance apply the requested changes only starting
from the next image, don't do any adjustments for
currently processed image
- at the moment the uml model does not allow operator
to change params in real time
- need usecases flashed better
- all what Ray described consistent with Robyn's plans,
no controversies identified
AOB
===
Last telecon: spatial/temporal issues
- Probably most things nailed down, and would not
make much progress if we come back to this topic now
Maria's ingest test results sent to mailing list
- they are now in the Ingest Docushare document
Next meeting in 2 weeks
- Tim, Ray and Jacek all at a meeting in Santa Monica
- maybe reschedule. Perhaps the Friday after?
- this slot is used by DataProdWG, but they did not
have meetings recently, so need to check with Kem
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
We have a DataAccWG telecon tomorrow (Friday) at 11:00 PST.
Phone number: 866 330 1200
passcode: 300 2363
Main topic: issues raised in the email thread
"problem statement for focusing further modeling"
(passing data from pipeline to data ingest service,
chunking data, clustering...)
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
