Robert Kanter created OOZIE-2335:
------------------------------------
Summary: Create a Shard node
Key: OOZIE-2335
URL: https://issues.apache.org/jira/browse/OOZIE-2335
Project: Oozie
Issue Type: New Feature
Components: core
Affects Versions: trunk
Reporter: Robert Kanter
A Shard node has been discussed every so often, and it would be great to
actually make it. It's similar to a Fork node, but it multiplexes the same
path instead of different paths. For example, you can run the same action(s)
multiple times.
A config for concurrency would be good, to allow users to control how many
concurrent multiplexed instances could run at the same time, similar to the
concurrency control in a Coordinator. This would also allow users to write
loops (in a safe way, because the number of iterations is set to a number).
It would be good for a sharded path to support an EL function, say
{{$\{shard:count()\}}}, which would return a the path count (in other words,
each multiplexed instance of a sharded path would have it's own unique id).
We also probably need to have something similar to the ForkJoin checking that
{{LiteWorkflowAppParser}} does.
I'm sure there's other features and things that would be good that I'm not
thinking of right now.
It might be a good idea to take a look at [~tucu00]'s lite2 WiP patch at
OOZIE-592, where he actually has a Shard/Unshard node. I'm sure a lot of that
can be brought in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
