Hey Adam,Gotcha - thanks for the clarification. For the record, I wasn't advocating for an RDBMS here, just mentioning an experience we had where Hadoop et.al. wasn't as responsive as we needed -- the use case was different though, so maybe not that relevant.
What you're suggesting with WKB and GeoJSON sounds like it would be a natural complement to SIS-10 [1].
-Andrew. [1] https://issues.apache.org/jira/browse/SIS-10 On 11/13/11 6:21 PM, Adam Estrada wrote:
Hi Andrew, The idea is for SIS to be data agnostic which means that it shouldn't matter what format the data are stored in. For SIS processing on Hadoop, I envision passing something like WKB or GeoJSON around rather than going through a RDBMS which tends to be a bottleneck anyway. Adam On Nov 13, 2011, at 7:53 PM, Andrew Hart wrote:Hey Adam,1. Everything is run over the web now which means that a full REST API would be very cool. There is the WPS specification but I don't know anyone actually using it. http://www.opengeospatial.org/standards/wps Its based on older and heavier SOAP web requests. Yuck! I think that if you could make very light weight calls to a centralized service, you would inevitably be rolling your own geoprocessing service. Win! With that said, anything that is to be developed should adhere to OGC specifications first! This will ensure adaption across a lot of different platforms.I like this suggestion a lot!2. As you very well know...a major problem with geospatial data is its size. This translates to to SLOW processing times, right? What if you were to wire the entire thing to run on Hadoop. Whatever, you're the expert when it comes to this sort of thing but I would love to hear how you would address read/write on large data sets. Keep in mind that the major limitations are on the geospatial formats, not necessarily on the underlying infrastructure.Agreed. In previous experience at least, Hadoop+Hive for back-end querying works great provided the end-user isn't waiting for a real-time response. For that, we've had good results with both a hybrid MySQL back end (a bit of a hack, honestly) and with using MongoDB as the data store. Either way, I think your 3rd point, below, about a pluggable architecture also applies to the data store.3. A pluggable architecture that will allow the end users to swap out projection engines (Proj.4, GeoTrans, etc.) as well as format translators like GDAL. Those are the two major issues that I've heard folks raise in the past. All modern applications permit the end user to make that choice. 4. The key component with any solution like SIS is in the geometry engine. This project is going to be considered direct competition with JTS and GEOS. How do you plan on handing processing geometries? Roll your own? I suppose a good starting point would be to start porting the Shapely project http://gispython.org/shapely/docs/1.0/manual.html but I believe even it is relying on GEOS. Lemme know what you think about all this. AdamBest, Andrew.On Nov 13, 2011, at 2:36 PM, Mattmann, Chris A (388J) wrote:Thanks Adam. You are contributing now through discussion and so forth. Please continue to do so, we like having you around! Cheers, Chris On Nov 12, 2011, at 2:54 PM, Adam Estrada wrote:Dr. Mattmann et al, I speak for spatial folks everywhere when I say that an ALv2 toolkit would be very widely used. I have ideas that I would like to contribute too so please let me know when and where I can do this. I really hope to see this project move forward and will contribute as much as I can. Regards, Adam "Mattmann, Chris A (388J)"<chris.a.mattm...@jpl.nasa.gov> wrote:Hey Kevan, Totally agree. My thoughts are, as I have time to develop out spatial code, I want it to go into SIS. I've talked with many people about this, and I think there is a general consensus from the broader community that an ALv2 licensed spatial toolkit is something folks would want. It's just that a lot of the geospatial experts out there I think are looking to develop their custom solutions and to then leverage them to make money. An ALv2 licensed spatial toolkit would probably do a lot to combat that, so maybe that's why we don't see a lot of folks lending a hand, I dunno. Anyways I did a few commits in the past 3 months and did close out an issue or 2, so I'll include that in the board report and I'll whip it up today. I don't want this community to die, so I'll try my best to keep it going, and to get more people interested in it, and to attract new contributors. I'm kind of doing that right now with some folks that we are working on geospatial stuff with, so I hope it pans out. In the meanwhile, it's not costing us much other than board reports to keep the project going at the moment, so I think it's pretty low overhead and worth it to keep trying. BTW, great to meet you in person at ApacheCon! :-) Cheers, Chris . On Nov 12, 2011, at 1:33 PM, Kevan Miller wrote:I've been having some mail client issues. I don't see a reminder to SIS for a November report. However, we are scheduled to report in November. We're a bit late, but still have time to get a report in, I think. Any volunteers? It's been kind of quiet the last few months. This may be a good time to evaluate where we see the community going. There's no big rush on this. But board reports are good reminders to think about these things… --kevan++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
