Keywords: DataAccWG
-----Original Message----- From: Becla, Jacek Sent: Thursday, November 09, 2006 9:47 AM To: 'Maria A. Nieto-Santisteban' Cc: Serge Monkewitz; Ani Thakar; Tim Axelrod; Jeffrey P Kantor Subject: RE: indexing/partitioning action item Hi Maria > > 3) in the post-processing stage, we flush the updated rows > from the > > in-memory table into appropriate disk-based partitions. > Flushing 100K > > rows to disk (in sequential fashion) should not be too bad > > Flushing 100 K rows to disk doesn't mean you have updated the original > tables. Yes, sure, poor choice of words on my side. I meant replacing affected rows in the original table > One alternative to avoid the updates(deletes/inserts) is reading the 4 > neighbor regions, do whatever you have to do in the memory and then > replace the old partitions with the new ones. That is a common > technique. This is very disk IO heavy. 1 entire partition = 10 million rows, and we update ~1 % of rows (100K), in neighbor-partitions this fraction will be even smaller. We should not attempt to write back all 10 million rows if we can get away with writing << 1% (especially if these <<1% will be clustered in a "good way") > > - you are not saying that we should keep these "temporary > tables" in > > memory, but I am guessing you do want it, right? > > Not sure whether you refer to 1) keeping them in memory in the sense > of work with them in memory or 2) keeping them on memory the whole > night? The former, per visit, then discard > I guess only if you can do all in memmory by your self. I don't know > if the DBMS will let you create the table only in memory and work with > that. > Is there any DBMS that let you say i want this table to be in memory > only? Oh YES, in-memory database are very hot these days, the whole telecom rides on them!, and now defense it starting. One example (recently in the news): http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9004516&source=NLT_AM&nlid=1 Oracle and mysql have good support for in-memory databases, postgres: no but they are working on it. Don't know about db2. In mysql you just say that the table type should be "in memory". You can trivially covert regular table to in memory. Indexes are optimized for memory access, but there are so restrictions too, e.g. var-length fields are converted to non-var. > > We should try to talk on Monday, I am flying to Tucson Monday late > > afternoon, then in Tucson the rest of the week, practically > unavailable. > > How about 2:00 ET? 11 PST? That would be during Data Mgmt telecon. 1 hour later might work better (earlier there is techassess telecon, so we will not get the telecon line) Thanks, Jacek _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
