Actually, the issue is somewhat complex as you suggest, but also not that hard to address. Let me try to explain things, although I might be too close to the problem. Before being involved with DX since nearly the beginning (10 years), I worked on visualization and data management among other things at GSFC (> a decade), including archives of satellite observations, the development of CDF, etc...
The problem is that tools like IDL or data structures like CDF, netCDF and HDF natively treat data as arrays, which represent regular grids. Obviously, this is simplistic. Any semantics associated with the interpretation of those arrays are an exercise left to the use, which are required for realistic treatment of the data. Some groups have established conventions and applications that use them to address some but not all of these limitations. This was the case for the space physics community and CDF over a decade ago. For atmospheric and oceanographic work, similar ideas have been independently implemented for netCDF in the mid-90s. The wheel was reinvented again with Eos-HDF and elsewhere. But there's also been a bit of self-propagation here. For example, the design of formats for archived data have often been based upon the assumption that visualization tools only know how to deal with regular grids. Thus, regular grids and arrays have become interchangable, which is wrong because information is lost in the exchange. So, these details haven't often been maintained and there are no apparent problems when these tools are used with such data, but they are there. There's not much you can do about this situation. But let's talk about data and how to use them the best way you can in DX. The way to think about the data independently of DX, is that you have points in space that represent where you sample data, whether this is a "real" sample from a sensor or a point at which you compute. If the points are NOT isolated as from an in situ observations or a Monte Carlo simulation, then there is a relationship between those sample points (i.e., a mesh). This relationship is the connectivity, which also describes the topology of the mesh. That information can also indicate how you determine values at locations between the sample points (i.e., interpolation, which is needed for visualization). The DX data model implements this notion. So "positions" become the sample points and "connections" becomes the relationship between them. Thus, a gridded field has at a minimum data. positions and connections. The data may be defined at the positions (sample points) or connections (the center of the cell defined by whatever number of points compose the cell depending on element type, i.e., 2 for lines, 3 for triangles, 4 for quads or tetrahedra and 8 for cubes). For the latter, it means the data value applies to the whole cell. For the former, it means only at that point. BTW, there is mathematical formalism for all of this, but I won't bore you with the details since it is its implications for visualization that is relevant here. Another source of confusion might be in the distribution of the sample points. An image or regular grid has samples that are equally spaced. DX has conventions for storing such a specification compactly. But this is independent of the data dependence (on vertices or cell centers). Thus, an image (bitmap) or a DEM on a cartesian (cylindrical equidistant projection) is on a regular grid with data dep connections. But for satellite data (e.g., remotely-sensed image) the sample points (cell centers) are irregularly spaced and the data are cell-centered as you have implied. DX can handle this, but to do it correctly, you'll need the data to be properly registered -- either the geographic or projected coordinates of each cell center or a definition of the projection so that you can warp a regular grid to the proper projection. A consequence is the coordinate system that operations take place, including rendering, are defined by those coordinates. A byproduct is that the realization of other data registered in that coordinate system can be easily combined in the same image. Amazingly, most tools don't both with such details. Admittedly, if everything is treated as pixel coordinates, which some tools do, than you can't address hits. Of course, if the number of samples is large compared to rendering resolution, then you can't see the difference, as you suggest. On the other hand, the calculations that treat the data more simply will be wrong. You mention about DX and IDL. This very issue was a problem when we built an interface between DX and IDL over 6 years ago. IDL doesn't have a data model in the sense that DX does. It only knows about arrays. One could pass data arrays between the packages, but IDL didn't know what do with the semantics that DX supports. So, for example, I might have read an image in IDL, which I could manipulate and display. I could do the same in DX. Instead, however, I could treat it as an irregular mesh of sample points which actually correspond to where the data where acquired in some geographic coordinate system. The data would be dep connections because it was an image. But the "pixels" are actually distorted quads that would correspond to the projected footprint from the "imaging array" on the spacecraft. Does this help? Yes, your problem might be an issue of dep positions vs dep connections. I've also seen cases where the definition of the coordinates or projection in archived data were not even correct. In some cases, the gap might be real (e.g., data from an instrument on a sun-synchronous platform on a polar orbiter), because a global grid represents an integration of approximately 24 hours in time). "Sharon Cady" <[EMAIL PROTECTED]>@opendx.watson.ibm.com on 12/14/2000 01:07:56 PM Please respond to [email protected] Sent by: [EMAIL PROTECTED] To: [email protected] cc: Subject: re: [opendx-users] How control layers Lloyd (or others familiar with remote sensing topics), please tell me more about this dep "positions" vs. dep "connections" issue. I have some knowledge of it, but never have gotten a complete handle on it. I have inherited one of those visualization tools in the atmospheric sciences which has "ignored the distinction". I have tried to set things straight in the past, but am still hindered by confusion. Currently I am reading in satellite data as dep "positions". I know that this isn't exactly correct because when the data is displayed in an image, the colors are interpolated across data points. This is an unwanted effect, because the data points do, in fact, correspond to an "area" of some (not easily obtained) size. But, following inquiries about this issue, I concluded that it's not as simple a solution as to merely convert my data to dep "connections" or cell-centered data, that this wouldn't be 100% "correct" either (I do know that it doesn't work well to do the conversion after the data has been read in, that if data truly does fit the definition of cell-centered, it must be defined that way up-front. But, I gather that there is more to it than that). So, my data remains dep "positions". With a huge data set, the eye can't detect the interpolation very well. But with a subset, there is a huge problem. I get around this by displaying the small number of points in the subset with a glyph of appropriate (data scaled) color. However, this is a flaky work-around. The difficulty that I've had with the concept of satellite data being treated as cell-centered is that any "grid" that a particular data set would "fit" into would be irregular and highly variable, depending on geolocation. I'm sure that the vertices of each grid cell for each data set could be calculated or determined some how, but the idea of specifying them within dxseems to me too big a bear to even think about handling. So, how do us misguided atmospheric/space science folks recover from a mess like this ? Lloyd, this topic came up briefly some time back when someone asked for a DX/IDL comparison. It was my opinion that IDL's treatment of data as arrays was a much more straight forward concept in terms of applications to satellite data, but you seemed to suggest that DX is perfectly suited to handle it, and well, or better. I wanted to query you more about that at the time, but didn't. There didn't seem to be much interest in remote sensing topics within the user's group. But now, there is some meteorologist out there apparently who is beginning to build a dx tool from scratch. I would highly recommend to him tackling these sorts of things at the onset. By the way, I've also had a similar problem to this "sectorial hole" that Paul describes. I don't know what kind of global projection he is using, but when I use rectangular (cylindrical equidistant), I get wrap-around problems (a line across the globe trying to "connect" fragmented satellite swath of data). Is this also a dep "positions" vs. dep "connections" issue ? Regards, Sharon
