[
https://issues.apache.org/jira/browse/USERGRID-324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
George Reyes updated USERGRID-324:
----------------------------------
Sprint: (was: Usergrid 34)
> [SPIKE] Prototype a few distributed realtime parallel processing systems
> ------------------------------------------------------------------------
>
> Key: USERGRID-324
> URL: https://issues.apache.org/jira/browse/USERGRID-324
> Project: Usergrid
> Issue Type: Story
> Reporter: Todd Nine
> Priority: Minor
>
> We need a system to allow us to build distributed/parallel data process
> flows. Ultimately, this system must have the following requirements at a
> component level.
> h1. Software Requirements
> # Work well within Reactive/Streams logic
> # Allow easy operationalization. No additional external systems should be
> required
> # Allow failover during node failure, and automatic recovery. Processing
> should not stop/restart because of a failure, it should resume.
> # Not additional state administration should need to be built. Distributed
> futures should be an inherent part of the framework.
> Some examples are the following.
> # Migrations
> # Import/Export
> # Distributed Indexing on heavily connected entities
> # Post processing deletes
> # Collection deletes
> # Application deletes
> I have the following requirements.
> # You can define a deployment topology and limit the number of sub
> processes/workers in the workflow
> # Ability to reject requests when there is no capacity
> # Preferably, do not introduce another dependency (like Zookeeper) and deploy
> it in the stack war file
> # An easy intuitive interface for programming flows which will work in a
> single node, or clustered environment
> # Administrative status, giving the ability to poll operational status via
> REST for automated monitoring
> h2. Examples
> h3. Guaranteed post processing
> # A write occurs. It is written to Cassandra and a post processing message
> is triggered.
> ## This message must be persistent, and allow for recovery during complete
> failure.
> ## If there is no capacity for post processing in the system, we should
> reject this request via back pressure
> ## The post processing should return a future (or similar) that could be
> blocked on if the caller chooses to.
> # The HTTP call is returned to the user
> # Post process indexing occurs asynchronously
> h3. Reindex all entities in the system.
> # Launch a root process. This process emits all application ids within the
> system.
> # Child processes receive the application id. For each app, create the index
> in elastic search, then emit all collections.
> # Child processes receive the collections and app ids. For each collection,
> emit the entity id
> # Child process receives the app, collection, and id. For each entity, get
> it's edges, and re-index the documents within elastic search.
> h3. Delete a collection
> Realtime -> Update collection alias to point to a new internal collection
> name. Fire delete collection task.
> Job process
> # Launch root process. Load previous collection name and emit to 2 child
> tasks.
> # Child task 1: Remove every entity of the previous type from elastic search
> using bulk delete until empty.
> # Child task 2: Iterate every entity and remove it from Cassandra, as well as
> it's graph edge.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
