Hi Ray,
I am not sure what the best forum to discuss these details is
- we could try to make progress at Friday's DataAccWG telecon
(if we have time)
- or at the next MiddlewareWG telecon
- or through emails
- or simply wait until March 23/24
Actually, I think talking about the issues you raise would be more
productive at this time than talking about xrootd. While I expect an
xrootd will be valueable to us here at NCSA, I don't expect that xrootd
will play a role in our DC1 work at this late date.
We could do that: we can make it the main topic of the DataAccWG
telecon on Friday
Database servers should run on dedicated machines,
not together with pipelines
...so that the database operations do not interfere with the pipeline
components?
Yes, in order to best tune the database we need to understand:
- best balance between number of cpus, disks and memory
- how to stripe disks
- and so on.
Having some other activities (like pipeline components) will
make that tuning almost impossible.
or would having 200 dbs (one for each CCD) be overkill (or
have too much overhead when it comes to merging)?
[Actually, it might be more than one per ccd, see below
for more details]
200 database servers would be an overkill.
200 databases is not a problem. Notice that no matter how
we slice it, the merging has to deal with the same
amount of data. So it is only a matter of 'bookkeeping',
and few hundred or few thousand is easy to track.
We need to estimate how many simultaneous ingest streams
(databases) a single database server can handle, I imagine
it will be much more than 1. My guess is that
for DC1 we will need 1-2 (maybe 3) servers for ingest,
plus one for merging. I will be able to tell more precisely
after we do some performance tests and after you/Tim tell me
the scale of DC1 (MB/sec, or rows/sec)
BTW, are you still talking about these "dedicated servers" as holding
temporary databases until they can be merged into the master database?
yes, remember that for base camp (real time alerts) there will be
no master database, these dedicated servers will hold the data
for given night.
At the main archive these "dedicated server" will continuously
load data into db, and then the tables will immediately be moved
to the server merging the data.
Note (perhaps just to myself): The sources coming out of the Detection
pipeline will be used by the Association pipeline which all happens within
the context of nightly (real-time) processing. Thus, the "ingest" needs
to happen in an "on-line" way as part of the pipeline management (as
opposed to some ingest that doesn't need to happen in real time as is the
case, I think, for image ingest).
This means we need to send a signal from Ingest Service to
Association pipeline indicating that data is "loaded and available"
Is there any way to stream this (ascii) data into database, as opposed to
reading from a file?
Nope. We can request it, they were taking seriously our requests
so far.
My plan for DC1 is to do 1 process per CCD, so yes.
- in "chunks", a chunk corresponds to a section of sky
from a single image.
if #chunks > #dbservers and we assign each server to a differnt part of
the sky, will the order that they are ingested be important--that is,
once we get the right chunks to the right servers?
No, we should not rely on the order in which we write.
We should always use ids/indices to locate data.
It occurs to me that if we keep these databases around all night and
simple partition them by sky--i.e. the entire sky observed that
night--then not all servers will be ingesting data during the processing
of a given frame. I would think that we would want to keep all databases
ingesting data as much as possible. If so, then partitioning can't be
that simple. (Perhaps you've already thought through all this.)
I am glad you brought that up. I looked back at the notes
from our older discussions. We talked about it at the
DataAccWG telecon on Jan 13:
"How to split the image?
- based on sensor readouts (~6400)
- the 6400 images cover separate regions, no overlap"
so we thought a "chunk" would be a piece of data corresponding
to one output amplifier, which means 32 chunks per ccd,
x~200 ccds/image --> 6400 "chunks".
I believe we can handle ~6K ingest streams if we wanted.
The question is what is the natural chunk from pipeline
perspective. If a pipeline processes data for the whole
ccd together, then we should setup 1 ingest stream per ccd,
but if a pipeline processes data for output amplifiers
in parallel, it would make sense to be able to have
one ingest stream per amplifier.
How do we "buffer" the incoming files on database server?
- One directory per stream? One for all?
- need to decide on file name convention
- we should probably allow for buffering many files for given stream
(in case ingest falls behind for whatever reason)
Again, is order important during ingest?
No, only to the extend that at the base you probably don't
want delays, so if data from the next image arrived and
we did not ingest data from the previous one we have a problem...
Can the databases run completely independently?
yes
That is, is there any
reason they, for example, need a shared file system across the servers?
no
If not, I would recommend one directory per stream; this maximizes our
flexibility in terms of filesystems. They can be on a shared fs or not.
yes, it simplifies many things (like recovery)
Once the file appears in the buffer, it can be grabbed by ingest,
loaded to db, then deleted
- do we need mutexes/locks around files?
Generally, we need some sort guard against reading files that are still
being written to. There are simple tricks for doing this.
Right, that is what I was after.
Do we want to preload data to memory before loading to db,
or just read from disk?
- if preload, then how?
- ram disk? (be aware: Redhat has a limit for RAM disks: 0.5GB,
we are working with them on this issue)
- xrootd memory mapped files?
When we are talking about ingest that is part of the real-time processing,
then "preloading" may not really help us (unless the ingest requires
multiple passes over the same data).
no it does not.
If ingest determines during validation step that
the input file is corrupt/data is formatted incorrectly,
then it will send signal to pipeline. Then what?
- should ingest stop?
- should ingest ignore the file and continue?
Let's assume an ingest service is ingesting many chunk file asynchronously
from the processing step that wrote the file. Then the service should
signal an error via an event. That processing step could be listening out
for that error, but more likely the Pipeline Manager will have to react to
the error and reschedule the processing step.
so the processing step will re-generate this file. That means
ingest should signal an error, ignore the file and continue,
because this file will likely be re-generated. Right?
Hey, so what if that file is re-generated and it is still wrong?
Will Pipeline Manager eventually recognize that?
How serious should we be about recovering from errors, like
- full-disk failure
- server crash
- machine crash
For DC1 or in general? If the latter, then the answer is very. For DC1,
not so much. We should definitely have these concerns influence our
design but we don't have to implement the actual detection and handling.
For DC1. Of course!
thanks,
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
