Keywords: DataAccWG
-----Original Message----- From: Becla, Jacek Sent: Wednesday, November 08, 2006 9:59 PM To: 'Maria A. Nieto-Santisteban' Cc: Serge Monkewitz; Ani Thakar; Tim Axelrod; Jeffrey P Kantor Subject: RE: indexing/partitioning action item Hi Maria Thanks - I am very glad to see this: we agree almost 100%, (although we are using slightly different language/approach to describe our thoughts). Some comments: - there is not such thing as Ingest Pipeline. It is called Ingest Service =) - I wasn't planning on extracting "whatever is needed" from neighbor-partitions, instead I was going to load entire partitions into memory (your approach saves ~70 GB of RAM, 2x if we want redundancy, this is at the expense of extracting "whatever is needed" from neighbor-partitions). Not sure which approach is more efficient, might be yours. - "Update Object table: This might be very ugly if we try to use a traditional SQL update as it would have to be a row by row based on objid". I think things should not look too bad, let me highlight why: 1) we load FOV partition + "whatever is needed" from neighbor-partitions, all into one in-memory "temporary" table. In prepare stage 2) we update 100K objects as needed during processing, all in the in-memory table, no disk access needed (that will help!) 3) in the post-processing stage, we flush the updated rows from the in-memory table into appropriate disk-based partitions. Flushing 100K rows to disk (in sequential fashion) should not be too bad - you are not saying that we should keep these "temporary tables" in memory, but I am guessing you do want it, right? - you assume we will need to read DIASource row size (300 bytes) x number of DIASources (10 million): is that sufficient given that 1 DIASource represents one measurement in one filter? (I don't know, will check with Tim and others at "prototype platform" meeting next week) - "M = something like the number of times (epochs) that an object has been detected by Difference Image processing? [I really have no idea what is this number is]". By the time we reach DR1, each field will be observed ~60 times. Also, recall that only ~10% of all measurements will have high signal-to-noise. I am not too sure whether your assessment "assuming M = 1 DIA source per object" is correct, my feeling is it will be more than 1 (again, I will try to check) - DIASources are produced by Detection Pipeline, and read by AP. A thought: could we write all new DIASources to a separate in-memory table, then after processing is done append it to the big DIASource table (that avoids writing to disk by DP and reading from disk by AP during the most critical time). All new DIASources for 1 visit = 30 MB, that is nothing! Again, I am trying to avoid disk IO as much as possible. I started earlier today to work on a database model for the nightly processing usecase, I will make sure your comments are in there, and will clarify whatever needs clarification next week. We should try to talk on Monday, I am flying to Tucson Monday late afternoon, then in Tucson the rest of the week, practically unavailable. We are making great progress =) Thanks! Jacek _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
