On Wed, 8 Mar 2006, Jacek Becla wrote: > I am not sure what the best forum to discuss these details is > - we could try to make progress at Friday's DataAccWG telecon > (if we have time) > - or at the next MiddlewareWG telecon > - or through emails > - or simply wait until March 23/24
Actually, I think talking about the issues you raise would be more productive at this time than talking about xrootd. While I expect an xrootd will be valueable to us here at NCSA, I don't expect that xrootd will play a role in our DC1 work at this late date. Email discussion would also be a good use of bandwidth. Let's try to hammer through some of these before the skunk works. > Database servers should run on dedicated machines, > not together with pipelines ...so that the database operations do not interfere with the pipeline components? or would having 200 dbs (one for each CCD) be overkill (or have too much overhead when it comes to merging)? I don't disagree, but I want to make sure I understand your reasoning. BTW, are you still talking about these "dedicated servers" as holding temporary databases until they can be merged into the master database? > Data will be passed in the form of ascii files > - we need to decide exactly what the format is > (mysql requirements will drive this) > - these files will be created by pipeline, > and deleted by ingest service Note (perhaps just to myself): The sources coming out of the Detection pipeline will be used by the Association pipeline which all happens within the context of nightly (real-time) processing. Thus, the "ingest" needs to happen in an "on-line" way as part of the pipeline management (as opposed to some ingest that doesn't need to happen in real time as is the case, I think, for image ingest). Is there any way to stream this (ascii) data into database, as opposed to reading from a file? > Data will be arriving in multiple streams > - one stream per CCD? My plan for DC1 is to do 1 process per CCD, so yes. > - in "chunks", a chunk corresponds to a section of sky > from a single image. if #chunks > #dbservers and we assign each server to a differnt part of the sky, will the order that they are ingested be important--that is, once we get the right chunks to the right servers? It occurs to me that if we keep these databases around all night and simple partition them by sky--i.e. the entire sky observed that night--then not all servers will be ingesting data during the processing of a given frame. I would think that we would want to keep all databases ingesting data as much as possible. If so, then partitioning can't be that simple. (Perhaps you've already thought through all this.) > - should we pass each "chunk" in a separate file (?) That seems most efficient to me. > How do we "buffer" the incoming files on database server? > - One directory per stream? One for all? > - need to decide on file name convention > - we should probably allow for buffering many files for given stream > (in case ingest falls behind for whatever reason) Again, is order important during ingest? Can the databases run completely independently? That is, is there any reason they, for example, need a shared file system across the servers? If not, I would recommend one directory per stream; this maximizes our flexibility in terms of filesystems. They can be on a shared fs or not. > Once the file appears in the buffer, it can be grabbed by ingest, > loaded to db, then deleted > - do we need mutexes/locks around files? Generally, we need some sort guard against reading files that are still being written to. There are simple tricks for doing this. > Do we want to preload data to memory before loading to db, > or just read from disk? > - if preload, then how? > - ram disk? (be aware: Redhat has a limit for RAM disks: 0.5GB, > we are working with them on this issue) > - xrootd memory mapped files? When we are talking about ingest that is part of the real-time processing, then "preloading" may not really help us (unless the ingest requires multiple passes over the same data). > If ingest determines during validation step that > the input file is corrupt/data is formatted incorrectly, > then it will send signal to pipeline. Then what? > - should ingest stop? > - should ingest ignore the file and continue? Let's assume an ingest service is ingesting many chunk file asynchronously from the processing step that wrote the file. Then the service should signal an error via an event. That processing step could be listening out for that error, but more likely the Pipeline Manager will have to react to the error and reschedule the processing step. > How serious should we be about recovering from errors, like > - full-disk failure > - server crash > - machine crash For DC1 or in general? If the latter, then the answer is very. For DC1, not so much. We should definitely have these concerns influence our design but we don't have to implement the actual detection and handling. cheers, Ray _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
