MultiThreaded processing is always a fun game. You may wish to look into what
is already available and come up with some best practices for use with
GeoSpatial data. Such a direction could benefit all the Java projects.
There are a couple of approaches:
1) Thread First
You should also have a look at the Java Concurrency API
(http://docs.oracle.com/javase/tutorial/essential/concurrency/) (this was an
open source project that was eventually folded into Java after it repeatedly
showed Sun up each JDK release).
In particular the Java futures concept which allows processes to be defined
independently of a thread pool(s) available for execution. This forms the core
of our GeoTools process
(http://docs.geotools.org/latest/userguide/tutorial/process.html) data
structure. Process is basically a Callable allowing for multiple inputs and
outputs and a progress bar, it is wrapped up as an normal Java Executor for
dispatch to a thread pool.
2) Data First
This is probably the correct approach, and one enjoyed automatically by Scala
programmers (with the cost of switching to functional programming). The idea is
to make the collection class responsible for the multi-threading, and throw
processing operations at it. The collection class can best decide what indexes
are available, how the results can be collected (when the work is split between
threads) and so on.
This separation of concerns to enable multithreading is the main advantage of
the visitor design pattern, one we have not really taken advantage of in
GeoTools. Modern developers sometimes refer to this as an "Internal iterator".
Note for strict data traversal A large portion of the benefit can be obtained
by allowing the collection to return different "External Iterators" in response
to different queries. Still an Iterator dooms you to a single threaded process.
For a really nice clear take on the split:
a) Review Scala Collections for excellent accessible implementation of map
reduce
b) Review the Sun JVN Language Fortress
(http://en.wikipedia.org/wiki/Fortress_(programming_language))
- the for operator resulted in a parallel operation
- the Runnable sent to the for operations could be marked with an interface
describing: how the operation could be multithreaded and; how each result
could be combined from different threads.
You can kind of view MapReduce as an optimisation of the data first approach,
with spatial data it should just been one arrow in your quiver to address
different workloads.
c) Have a look at a library like Guava
(http://code.google.com/p/guava-libraries/) which Gabriel likes for this kind
of work
For a sensible illustration of how to split
--
Jody Garnett
On Wednesday, 12 June 2013 at 8:49 AM, Landon Blake wrote:
> I've been chewing over a plan to extract some of the most beneficial parts of
> OpenJUMP into a stand-alone library that can be used outside of the OpenJUMP
> GUI. I've decided to focus this project, at least for the short term, on
> supporting multi-threaded geospatial processing using OpenJUMP's simple
> feature model.
>
> I'll need to do some refactoring to support threading. This will include
> making some classes immutable and designing some thread safe data structures
> for the feature classes.
>
> I can package this as a project for OSGeo Incubation Labs, but I wanted to
> check here first. Is this project something that can be packaged as a set of
> Geotools modules? I don't want to muddy the water by introducing modules that
> use OpenJUMP's simple feature model, but it makes sense to work with Geotools
> as much as possible, instead of starting yet another project.
>
> Please share your thoughts with me. I'll consider the input from the Geotools
> community before I move forward. If hosting this code at Geotools is a bad
> idea, tell me. I won't get my feelings hurt. :]
>
> Landon
>
>
> ------------------------------------------------------------------------------
> This SF.net (http://SF.net) email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> (mailto:GeoTools-Devel@lists.sourceforge.net)
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
>
------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:
Build for Windows Store.
http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel
