Hi Andrew, The idea is for SIS to be data agnostic which means that it shouldn't matter what format the data are stored in. For SIS processing on Hadoop, I envision passing something like WKB or GeoJSON around rather than going through a RDBMS which tends to be a bottleneck anyway.
Adam On Nov 13, 2011, at 7:53 PM, Andrew Hart wrote: > Hey Adam, >> 1. Everything is run over the web now which means that a full REST API would >> be very cool. There is the WPS specification but I don't know anyone >> actually using it. http://www.opengeospatial.org/standards/wps Its based on >> older and heavier SOAP web requests. Yuck! I think that if you could make >> very light weight calls to a centralized service, you would inevitably be >> rolling your own geoprocessing service. Win! >> >> With that said, anything that is to be developed should adhere to OGC >> specifications first! This will ensure adaption across a lot of different >> platforms. > > I like this suggestion a lot! > >> 2. As you very well know...a major problem with geospatial data is its size. >> This translates to to SLOW processing times, right? What if you were to wire >> the entire thing to run on Hadoop. Whatever, you're the expert when it comes >> to this sort of thing but I would love to hear how you would address >> read/write on large data sets. Keep in mind that the major limitations are >> on the geospatial formats, not necessarily on the underlying infrastructure. > Agreed. In previous experience at least, Hadoop+Hive for back-end querying > works great provided the end-user isn't waiting for a real-time response. For > that, we've had good results with both a hybrid MySQL back end (a bit of a > hack, honestly) and with using MongoDB as the data store. Either way, I think > your 3rd point, below, about a pluggable architecture also applies to the > data store. > >> 3. A pluggable architecture that will allow the end users to swap out >> projection engines (Proj.4, GeoTrans, etc.) as well as format translators >> like GDAL. Those are the two major issues that I've heard folks raise in the >> past. All modern applications permit the end user to make that choice. >> >> 4. The key component with any solution like SIS is in the geometry engine. >> This project is going to be considered direct competition with JTS and GEOS. >> How do you plan on handing processing geometries? Roll your own? I suppose a >> good starting point would be to start porting the Shapely project >> http://gispython.org/shapely/docs/1.0/manual.html but I believe even it is >> relying on GEOS. >> >> Lemme know what you think about all this. >> Adam > > Best, > Andrew. > >> On Nov 13, 2011, at 2:36 PM, Mattmann, Chris A (388J) wrote: >> >>> Thanks Adam. >>> >>> You are contributing now through discussion and so forth. Please continue >>> to do so, we like >>> having you around! >>> >>> Cheers, >>> Chris >>> >>> On Nov 12, 2011, at 2:54 PM, Adam Estrada wrote: >>> >>>> Dr. Mattmann et al, >>>> >>>> I speak for spatial folks everywhere when I say that an ALv2 toolkit would >>>> be very widely used. I have ideas that I would like to contribute too so >>>> please let me know when and where I can do this. >>>> >>>> I really hope to see this project move forward and will contribute as much >>>> as I can. >>>> >>>> Regards, >>>> Adam >>>> >>>> "Mattmann, Chris A (388J)"<chris.a.mattm...@jpl.nasa.gov> wrote: >>>> >>>>> Hey Kevan, >>>>> >>>>> Totally agree. >>>>> >>>>> My thoughts are, as I have time to develop out spatial code, I want it to >>>>> go into SIS. I've talked with many people about this, >>>>> and I think there is a general consensus from the broader community that >>>>> an ALv2 licensed spatial toolkit is something >>>>> folks would want. It's just that a lot of the geospatial experts out >>>>> there I think are looking to develop their custom solutions >>>>> and to then leverage them to make money. An ALv2 licensed spatial toolkit >>>>> would probably do a lot to combat that, so >>>>> maybe that's why we don't see a lot of folks lending a hand, I dunno. >>>>> >>>>> Anyways I did a few commits in the past 3 months and did close out an >>>>> issue or 2, so I'll include that in the board report >>>>> and I'll whip it up today. I don't want this community to die, so I'll >>>>> try my best to keep it going, and to get more people >>>>> interested in it, and to attract new contributors. I'm kind of doing that >>>>> right now with some folks that we are working >>>>> on geospatial stuff with, so I hope it pans out. In the meanwhile, it's >>>>> not costing us much other than board reports >>>>> to keep the project going at the moment, so I think it's pretty low >>>>> overhead and worth it to keep trying. >>>>> >>>>> BTW, great to meet you in person at ApacheCon! :-) >>>>> >>>>> Cheers, >>>>> Chris >>>>> . >>>>> On Nov 12, 2011, at 1:33 PM, Kevan Miller wrote: >>>>> >>>>>> I've been having some mail client issues. I don't see a reminder to SIS >>>>>> for a November report. However, we are scheduled to report in November. >>>>>> We're a bit late, but still have time to get a report in, I think. Any >>>>>> volunteers? >>>>>> >>>>>> It's been kind of quiet the last few months. This may be a good time to >>>>>> evaluate where we see the community going. There's no big rush on this. >>>>>> But board reports are good reminders to think about these things… >>>>>> >>>>>> --kevan >>>>> >>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>> Chris Mattmann, Ph.D. >>>>> Senior Computer Scientist >>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>> Office: 171-266B, Mailstop: 171-246 >>>>> Email: chris.a.mattm...@nasa.gov >>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>> Adjunct Assistant Professor, Computer Science Department >>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>> >>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> Chris Mattmann, Ph.D. >>> Senior Computer Scientist >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>> Office: 171-266B, Mailstop: 171-246 >>> Email: chris.a.mattm...@nasa.gov >>> WWW: http://sunset.usc.edu/~mattmann/ >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> Adjunct Assistant Professor, Computer Science Department >>> University of Southern California, Los Angeles, CA 90089 USA >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> >
