[
https://issues.apache.org/jira/browse/HIVE-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769449#action_12769449
]
Chaitanya Mishra commented on HIVE-549:
---------------------------------------
Thanks for your comments He.
Just to clarify: The way I'm thinking about it is: a TaskRunner will be
launched by the execute function in ql/driver.java only when it satisfies the
isRunnable() check which is currently implemented. Therefore, the workflow will
be like.
Task
- Scan through list of tasks. Launch taskrunners for each task that satisfies
the isRunnable call.
- Wait for any task to finish or fail.
If task finishes correctly
- Check its child tasks if isRunnable() is satisfied. If so, launch
TaskRunners for the taskrunners.
If task fails
- Kill all existing taskrunners, making sure to cleanup.
TaskRunner (extends thread)
- Takes a task, executes it in a separate thread, and returns either 0 or 1,
depending on success or failure.
The difference from your design (as I understand it) is that a TaskRunner will
be launched only when a task is runnable. All the notifications 2.1, 2.2 and
2.3 aren't necessary then. The existing logic in the execute() function should
handle this.
> Parallel Execution Mechanism
> ----------------------------
>
> Key: HIVE-549
> URL: https://issues.apache.org/jira/browse/HIVE-549
> Project: Hadoop Hive
> Issue Type: Wish
> Components: Query Processor
> Reporter: Adam Kramer
> Assignee: Chaitanya Mishra
>
> In a massively parallel database system, it would be awesome to also
> parallelize some of the mapreduce phases that our data needs to go through.
> One example that just occurred to me is UNION ALL: when you union two SELECT
> statements, effectively you could run those statements in parallel. There's
> no situation (that I can think of, but I don't have a formal proof) in which
> the left statement would rely on the right statement, or vice versa. So, they
> could be run at the same time...and perhaps they should be. Or, perhaps there
> should be a way to make this happen...PARALLEL UNION ALL? PUNION ALL?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
