Indeed. However, I guess the important load and stress is in the processing of the 3D data (DEM or alike) into geometries/shades/whatever. Hence you can use spark (geotrellis can be tricky for 3D, poke @lossyrob for more info) to perform these operations then keep an RDD of only the resulting geometries. Those geometries won't probably that heavy, hence it might be possible to coalesce(1, true) to have to whole thing on one node (or if your driver is more beefy, do a collect/foreach) to create the index. You could also create a GeoJSON of the geometries and create the r-tree on it (not sure about this one).
On Mon Dec 01 2014 at 3:38:00 PM Stadin, Benjamin < benjamin.sta...@heidelberg-mobil.com> wrote: > Thank you for mentioning GeoTrellis. I haven’t heard of this before. We > have many custom tools and steps, I’ll check our tools fit in. The end > result after is actually a 3D map for native OpenGL based rendering on iOS > / Android [1]. > > I’m using GeoPackage which is basically SQLite with R-Tree and a small > library around it (more lightweight than SpatialLite). I want to avoid > accessing the SQLite db from any other machine or task, that’s where I > thought I can use a long running task which is the only process responsible > to update a local-only stored SQLite db file. As you also said SQLite (or > mostly any other file based db) won’t work well over network. This isn’t > only limited to R-Tree but expected limitation because of file locking > issues as documented also by SQLite. > > I also thought to do the same thing when rendering the (web) maps. In > combination with the db handler which does the actual changes, I thought to > run a map server instance on each node, configure it to add the database > location as map source once the task starts. > > Cheers > Ben > > [1] http://www.deep-map.com > > Von: andy petrella <andy.petre...@gmail.com> > Datum: Montag, 1. Dezember 2014 15:07 > An: Benjamin Stadin <benjamin.sta...@heidelberg-mobil.com>, " > user@spark.apache.org" <user@spark.apache.org> > Betreff: Re: Is Spark the right tool for me? > > Not quite sure which geo processing you're doing are they raster, vector? More > info will be appreciated for me to help you further. > > Meanwhile I can try to give some hints, for instance, did you considered > GeoMesa <http://www.geomesa.org/2014/08/05/spark/>? > Since you need a WMS (or alike), did you considered GeoTrellis > <http://geotrellis.io/> (go to the batch processing)? > > When you say SQLite, you mean that you're using Spatialite? Or your db is > not a geo one, and it's simple SQLite. In case you need an r-tree (or > related) index, you're headaches will come from congestion within your > database transaction... unless you go to a dedicated database like Vertica > (just mentioning) > > kr, > andy > > > > On Mon Dec 01 2014 at 2:49:44 PM Stadin, Benjamin < > benjamin.sta...@heidelberg-mobil.com> wrote: > >> Hi all, >> >> I need some advise whether Spark is the right tool for my zoo. My >> requirements share commonalities with „big data“, workflow coordination and >> „reactive“ event driven data processing (as in for example Haskell Arrows), >> which doesn’t make it any easier to decide on a tool set. >> >> NB: I have asked a similar question on the Storm mailing list, but have >> been deferred to Spark. I previously thought Storm was closer to my needs – >> but maybe neither is. >> >> To explain my needs it’s probably best to give an example scenario: >> >> - A user uploads small files (typically 1-200 files, file size >> typically 2-10MB per file) >> - Files should be converted in parallel and on available nodes. The >> conversion is actually done via native tools, so there is not so much big >> data processing required, but dynamic parallelization (so for example to >> split the conversion step into as many conversion tasks as files are >> available). The conversion typically takes between several minutes and a >> few hours. >> - The converted files gathered and are stored in a single database >> (containing geometries for rendering) >> - Once the db is ready, a web map server is (re-)configured and the >> user can make small updates to the data set via a web UI. >> - … Some other data processing steps which I leave away for brevity … >> - There will be initially only a few concurrent users, but the system >> shall be able to scale if needed >> >> My current thoughts: >> >> - I should avoid to upload files into the distributed storage during >> conversion, but probably should rather have each conversion filter >> download >> the file it is actually converting from a shared place. Other wise it’s >> bad >> for scalability reasons (too many redundant copies of same temporary files >> if there are many concurrent users and many cluster nodes). >> - Apache Oozie seems an option to chain together my pipes into a >> workflow. But is it a good fit with Spark? What options do I have with >> Spark to chain a workflow from pipes? >> - Apache Crunch seems to make it easy to dynamically parallelize >> tasks (Oozie itself can’t do this). But I may not need crunch after all if >> I have Spark, and it also doesn’t seem to fit to my last problem >> following. >> - The part that causes me the most headache is the user interactive >> db update: I consider to use Kafka as message bus to broker between the >> web >> UI and a custom db handler (nb, the db is a SQLite file). But how >> about update responsiveness, isn’t it that Spark will cause some lags (as >> opposed to Storm)? >> - The db handler probably has to be implemented as a long running >> continuing task, so when a user sends some changes the handler writes >> these >> to the db file. However, I want this to be decoupled from the job. So file >> these updates should be done locally only on the machine that started the >> job for the whole lifetime of this user interaction. Does Spark allow to >> create such long running tasks dynamically, so that when another (web) >> user >> starts a new task a new long–running task is created and run on the same >> node, which eventually ends and triggers the next task? Also, is it >> possible to identify a running task, so that a long running task can be >> bound to a session (db handler working on local db updates, until task >> done), and eventually restarted / recreated on failure? >> >> >> ~Ben >> >
