Dave, I think it will clarify the context if you read the "nightly processing at base" usecase I sent out to the list a week or so ago. It addresses a number of your questions.
To short circuit things a bit: Yes, there is understanding that the detection (which we're calling "source") to object mapping is a difficult problem. At present, we are addressing it only for the "nightly processing at base" usecase. This runs off of difference images, and is only for the purpose of generating alerts. The association task required for producing a data release, which will be done at the archive center, is more difficult and has not yet been addressed. Your advice on how to do it will certainly be welcome! And, yes, creating a catalog a second time will make a different catalog because both the input data, and probably the algorithms, will be different. That's life. We'd like to retain as much consistency in object id's as practical, but for many obvious reasons this can only be partly successful. Any concrete suggestions? Cheers, Tim > Sorry to drop in out of ether, and perhaps in the middle of a discussion, > but I am quite interested in the issues raised by Serge and Jacek. If > the comments are clueless, please point me in the right direction and > I will be quiet for a while. > > 1) My first concern is whether the base camp catalog is the final version > of the catalog, or just an aide to other processing being done at > the base camp. I still have this serious mental issue when it comes > to the distinction between "detections" and "objects". To me, a > detection is a group of pixels that pass some sort of significance > test and thereby trigger additional processing for various bulk > parameters such as position, total flux, shape, etc. To me, an > object belongs to the physical universe and can be interpreted using > the laws of physics and having attributes like temperature, size, etc. > > The mapping between detections and objects is an amazingly difficult > problem, and my feeling is that doing it more than once will result > in different catalogs. Not all detections are objects (airplanes > and Earth satellites are obvious cases), and not all objects can > be detected (otherwise we would have solved the Dark Matter problem). > There are additional issues such as variable seeing that would cause > two or more objects to appear as a single detection in bad seeing, > but as separate detections in good seeing. A similar confusion arises > for extended objects such as galaxies, variable stars, variable stars > in galaxies, etc., etc. > > Again in my jaundiced view, the mapping of detections into > new/existing/ > spurious objects is quite subtle, and may require extensive processing > and database access that may be unreasonable to do at the base camp. > So to repeat my first sentence, what is the role of the proposed > in-memory catalog at base camp? > > 2) As is usual for an astrometrist, I am frightened of IDs based on > the measurement of position. Serge's note shows a sensitivity to > this issue in noting that the IDs need to remain fixed even if > subsequent processing changes the coordinates, but I still worry. > Many clever people have proposed single integers that are useful > indices for the 2-dimensional sky. Chunking of (RA,Dec) is certainly > one way to go, but nets and meshes of funny-shaped things (triangles > for instance) may be more useful. > > I am a bit surprised that one needs to store the whole catalog in > memory no matter how it is organized. The small chunk needed > for a single LSST observation should be a relatively small and quick > read-only access to a database, and I would have thought that this > could be done in milliseconds, not seconds. The subtle issues > raised in (1) above would not preclude finding and matching enough > stars to compute a provisional WCS transformation, for example, > and perhaps this would be a useful thing to do for down-stream > processing. > This doesn't address the update issue, the what-to-do-with-extra- > detections issue, the is-this-interesting issue, and others. > > So I think that I am getting back to my previous concern about what is > to be accomplished by the base camp correlation to an existing catalog. > Who is the consumer of this correlation, and what are the needs? > > -Dave Monet is [EMAIL PROTECTED] > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
