Keywords: DataAccWG
-----Original Message----- From: Maria A. Nieto-Santisteban [mailto:[EMAIL PROTECTED] Sent: Thursday, November 09, 2006 6:29 AM To: Becla, Jacek Cc: Serge Monkewitz; Ani Thakar; Tim Axelrod; Jeffrey P Kantor Subject: RE: indexing/partitioning action item Hi, > - "Update Object table: This might be very ugly if we try to use a > traditional SQL update as it would have to be a row by row based on > objid". I think things should not look too bad, let me highlight why: > 1) we load FOV partition + "whatever is needed" from > neighbor-partitions, all into one in-memory "temporary" table. In > prepare stage > 2) we update 100K objects as needed during processing, all in the > in-memory table, no disk access needed (that will help!) The idea is that if all you need fits in memory then the DBMS will keep it in memory. The DBMS doesn't like I/O either. > 3) in the post-processing stage, we flush the updated rows from the > in-memory table into appropriate disk-based partitions. Flushing 100K > rows to disk (in sequential fashion) should not be too bad Flushing 100 K rows to disk doesn't mean you have updated the original tables. One alternative to avoid the updates(deletes/inserts) is reading the 4 neighbor regions, do whatever you have to do in the memory and then replace the old partitions with the new ones. That is a common technique. > - you are not saying that we should keep these "temporary tables" in > memory, but I am guessing you do want it, right? Not sure whether you refer to 1) keeping them in memory in the sense of work with them in memory or 2) keeping them on memory the whole night? 1) As I said above: The idea is that if all you need fits in memory then the DBMS will keep it in memory. The DBMS doesn't like I/O either. So if the temporary tables fits in memory work will be done in memory but by the DBMS not us implementing a new indexing system. I would go on the path of implementing my own custom/smart indexing system only if the DBMS cannot do the job. But if works, don't touch it! 2) temporary tables last whatever takes to process a FOV > - you assume we will need to read DIASource row size (300 bytes) x > number of DIASources (10 million): is that sufficient given that 1 > DIASource represents one measurement in one filter? (I don't know, > will check with Tim and others at "prototype platform" meeting next > week) > > - "M = something like the number of times (epochs) that an object has > been detected by Difference Image processing? [I really have no idea > what is this number is]". > By the time we reach DR1, each field will be observed ~60 times. Also, > recall that only ~10% of all measurements will have high > signal-to-noise. I am not too sure whether your assessment "assuming M > = 1 DIA source per object" is correct, my feeling is it will be more > than 1 (again, I will try to check) The problem I have trying to figure out this number M is .. I don't fully understand what is the content of the DIA Source table. As I understand it: Using Difference Image Processing helps you to detect those things that change. So my presumption is that the DIA source table contains different epochs for one object, but mainly only for variable objects. We have 5% of variable objects in one FOV in the worst case but we cannot detect all of them becouse as you say we will be detecting only high signal to noise because in a single exposure you cannot go very deep. However as I recall from the meeting, during (or after) deep processing we were going to store more DIA sources which I assume corresponds to those thing you could not detect in a single exposure. Any way, so in the worst case lets assume I can detect all existing variable objects in the Galactic Plane => 5% of 10 000 000 = 500 000 M = 500 000 * 60 epochs = 3 000 000 (+ flashes, moving objetcs ~ tiny) So my 1 DIA source per object is a very overshooting but if the DIA source tables contains information about No Variable objects then the 1 DIA source per object can be very underestimated. > > - DIASources are produced by Detection Pipeline, and read by AP. A > thought: could we write all new DIASources to a separate in-memory > table, I guess only if you can do all in memmory by your self. I don't know if the DBMS will let you create the table only in memory and work with that. Is there any DBMS that let you say i want this table to be in memory only? > then after processing is done append it to the big DIASource table > (that avoids writing to disk by DP and reading from disk by AP during > the most critical time). All new DIASources for 1 visit = 30 MB, that > is nothing! Again, I am trying to avoid disk IO as much as possible. yeah ... I thought of just appending the new DIA sources table nad insert a new row in the partitions table. The problem I have with it is it breaks the symmetry and things are not clusterd any more. > > I started earlier today to work on a database model for the nightly > processing usecase, I will make sure your comments are in there, and > will clarify whatever needs clarification next week. > good. > We should try to talk on Monday, I am flying to Tucson Monday late > afternoon, then in Tucson the rest of the week, practically unavailable. How about 2:00 ET? > We are making great progress =) yeap :-) cheers Maria _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
