we use this checksum file also as a semaphore to indicate that the data
is ready to be ingested, since it is the last file created for each
chunk of data.
ani
On Tue, 30 May 2006, Alex Szalay wrote:
> >From the SDESS experience I would recommend that we write
> a checksum file into each directory, which contains the name
> of each file and its checksum. Then we have a way to verify
> if anything happened to any of the contents. These entries
> should be generated immediately after the file was written.
>
> --Alex
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
> On Behalf Of Russell E. Owen
> Sent: Tuesday, May 30, 2006 5:45 PM
> To: [email protected]
> Subject: [LSST-data] Re: catalog ingest service [Data Challenge]
>
> In article
> <[EMAIL PROTECTED]>,
> Ray Plante <[EMAIL PROTECTED]> wrote:
>
> > On Fri, 26 May 2006, Russell E. Owen wrote:
> > > The plan is for the Catalog Ingest Service (CIS) to have a higher level
> > > component that will itself decide such things as:
> > > - how many worker processes are best to handle the slices
> > > - which slices are handled by which process
> > > - perhaps is smart enough to realize that a process is stalled and
> > > restart it on a different node (not likely for DC1)
> >
> > Would it be simpler to just replicate the ingest service on different
> > nodes, each one monitoring a different input directory? The configuration
>
> > of the converter pipeline stage would just indicate which directory to
> > write its output to. Since we only need 2 (rather than 200) ingest
> > nodes, half of the converters would write to one directory, and the other
> > half to the other directory.
>
> Having talked to Ray on the phone, we agreed to have each slice write
> data files to a separate subdirectory, one whose name is trivially
> derived from the name of the slice. In other words, we'll stick to
> Jacek's original plan in this regards.
>
> That is easier for the catalog ingest service (CIS) because it knows
> that all files in a given directory are from a given slice. If data from
> multiple slices is written into one directory then the CIS would have to
> parse each data file name to figure out which slice it came from.
>
> Also, to summarize other conversations with Ray and Jacek:
> - The CIS is naturally divided up into two levels.
> * At the lower level, a set of 2 or so CIS workers will run, each as a
> separate process on a separate node, each ingesting data from a
> specified set of slices into *one* database server.
> * At the higher level, somebody needs to launch these CIS workers.
> Exactly who is still in question:
>
> The CIS workers are smart enough that it appears to make sense for the
> pipeline manager to directly run them. Each CIS worker needs no more
> than a policy listing which slices to listen for and info for connecting
> to the appropriate database. The pipeline manager can detect when each
> CIS worker finishes (or fails, depending on its exit code).
>
> But I can write a higher level CIS manager if that is desirable. Its job
> would be to apportion the slices and databases among the CIS workers.
> Nearly trivial (the only unknown at all is learning how best to launch
> the CIS workers on the appropriate nodes).
>
> My impression from the phone con was that the pipeline manager was a
> natural choice to launch the CIS workers, but that it wasn't a done deal
> and we'd talk more about it next week. Meanwhile I will concentrate on
> finishing the CIS worker code.
>
> -- Russell
>
> _______________________________________________
> LSST-data mailing list
> [email protected]
> http://www.lsstmail.org/mailman/listinfo/lsst-data
>
>
> _______________________________________________
> LSST-data mailing list
> [email protected]
> http://www.lsstmail.org/mailman/listinfo/lsst-data
>
--
Aniruddha R. Thakar, Research Scientist
Center for Astrophysical Sciences, JHU, Bloomberg 375
3701 San Martin Drive, Baltimore MD 21218-2695
410-516-4850, Fax: 410-516-5096
[EMAIL PROTECTED], http://www.sdss.jhu.edu/~thakar
-----------------------------------------------------------------------
Conscience is what hurts when everything else feels so good.
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
