Hello all, I think in the DM Camera DAQ interface, they are planning to deliver the data to memory by amplifier output, so we can use up to 6400 processors in parallel in image processing. However, that is input to the pipelines, I assume output from the pipelines can be aggregated by CCD or however we wish for ingest.
Jeff > From: Jacek Becla <[EMAIL PROTECTED]> > Organization: Stanford Linear Accelerator Center > Reply-To: LSST Data Management <[email protected]> > Date: Fri, 10 Mar 2006 15:31:59 -0800 > To: LSST Data Management <[email protected]> > Subject: [LSST-data] Notes from DataAccWG telecon Fri Mar 10 > > Keywords: DataAccWG > > Attendees: > Nicole Silvestri > Robyn Allsman > Ani Thakar > Maria Nieto-Santisteban > Ray Plante > Ghaleb Abdulla > Sergei Nikolaev > David Fleming > Jacek Becla > > > videocons > ========= > - Will try to use video starting next time > - Robyn will be our LSSTC participant responsible > for patching in the phone line; on days when she > is not there, we will revert completely to the phone. > - should have at least 3 participants through video: > - ncsa, slac, lsstc, possibly uw, llnl and jhu too > - need to warn other WGs about patching in the phone line > > > > passing data from pipeline to ingest etc... > ========================================== > > > slicing the data > - one stream per ccd? > - one stream per amplifier? ("section of sky") > - what is natural for pipeline? > > --> for DC1 most practical would be to parallelize > on CCD level: one processor/ingest stream per ccd > - so one Pipeline Processing Step per ccd, and > one ingest stream per ccd > > - if we partition based on amplifier, the overhead of > managing all these files will start to dominate > > > - need to determine amount of data per ingested chunk > for DC1 > - the baseline requirements mention average > #objects per detection > - Tim had that number, Robyn will find out > > > How the data will be "reduced" for DC1 (the 2% of production > throughput) > - smaller images? or full images but less frequently? > - Jacek, and Robyn always assumed it is "smaller images" > - Robyn heard Tim talking about "reduced images" > - need to clarify > > > Where a pipeline should put files for ingest? > How these files are transfered from machine to machine? > - for DC1 could write data to a shared file system (gpfs), > - have 2 directories, write to one, then mv file to > the other directory, consumer then grabs it > - alternative: through xrootd, Chris is testing xrootd now, > if he finds setting up xrootd is easy we might try it > for DC1 > > > Clustering merged data at the archive? > - the partitioning will be done "strategically" during merging > data from temporary tables. That is for the archive, > at the base no need to merge things > > - we do not really need to worry about "strategic partitioning" > for DC1. Main focus is on keeping up for alerts > - merging: can be included in DC1 if we have a coherent plan > > > - we need to merge data at the main archive before we > deliver it to consumers > - so do we have more than 24h to merge? > - data related to alerts will be transfered with high priority, > if we can ingest it first then we don't have the pressure, > could merge the rest less frequently than once every 24h > > What is the lifetime of data in temporary databases used for ingest? > - the tmp databases will be scratched every night, > need to at least transfer data to the node doing merging. > > > > policy class > ============ > > We will have different "strategies" for staging data, > for instance "distributing data from ccds to different nodes" > - similar idea to what "policy" does for pipeline > - can we reuse Tim/Robyn's work on pipeline policies? > > > Ray working on a document discussing what he learned > so far about policies, etc > - should have the draft by the end of today > > > > System should be able to adjust some parameters while > pipeline is running (human tweaks something through GUI, > e.g. when seeing goes bad). > - adjustment allowed and applied in real time? > - to be decided to what extend we are going to support it > (because it complicated things a lot) > - normally would just deal with static files that live > in policy library, but the run-time adjustments > complicate that, e.g. provenance > - provenance would now have to recode all the changes > of individual parameters. Then when we re-process, > we can't use just static files, have to go to provenance > > --> we have to be restrictive when to allow things to change > - only allow changes at certain break-points > - for instance apply the requested changes only starting > from the next image, don't do any adjustments for > currently processed image > > - at the moment the uml model does not allow operator > to change params in real time > > - need usecases flashed better > > > - all what Ray described consistent with Robyn's plans, > no controversies identified > > > AOB > === > > Last telecon: spatial/temporal issues > - Probably most things nailed down, and would not > make much progress if we come back to this topic now > > > Maria's ingest test results sent to mailing list > - they are now in the Ingest Docushare document > > > Next meeting in 2 weeks > - Tim, Ray and Jacek all at a meeting in Santa Monica > - maybe reschedule. Perhaps the Friday after? > - this slot is used by DataProdWG, but they did not > have meetings recently, so need to check with Kem > > > Jacek > > > > > Jacek Becla wrote: >> Keywords: DataAccWG >> >> Hi, >> >> We have a DataAccWG telecon tomorrow (Friday) at 11:00 PST. >> >> Phone number: 866 330 1200 >> passcode: 300 2363 >> >> >> Main topic: issues raised in the email thread >> "problem statement for focusing further modeling" >> (passing data from pipeline to data ingest service, >> chunking data, clustering...) >> >> Jacek >> _______________________________________________ >> LSST-data mailing list >> [email protected] >> http://www.lsstmail.org/mailman/listinfo/lsst-data > > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
