[
https://issues.apache.org/jira/browse/OOZIE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145775#comment-13145775
]
Mohammad Kamrul Islam commented on OOZIE-588:
---------------------------------------------
Design choices:
=============
Background:
--------------------
Oozie will need 3 types of information:
1. Hadoop Job Counters for MR action: Currently supported. It is stores in the
WFInstances directly for faster access. It can be accessed through EL function.
No way to query by the end user.
2. Pig Stats: Stats related to only Pig job. Not currently implemented.
3. Pig generated Hadoop Ids: Currently stored in WF Instance. Same as Job
Counters.
New Proposed Design Choices:
------------------------------------------
Two requirements:
1. End user should retrieve the above 3 stats directly.
2. End User could access through EL functions. (Already supported : #1 and #3)
Design choices to determine where to store and how to retrieve.
Choice 1:
...............
A. Store into WFInstances as it is done currently.
B. Pros: Minimum effort in storing. Reuse the current design.
C. Cons: Any end user direct-query will need to get stats from WFInstance.
Since WF Instance is in-memory, there will be sufficient memory pressure.
Choice 2:
................
A. Store all stats into WF Action table.
B. Pros: User query will be straight forward. No extra memory.
C. Cons: Accessing through EL function will be tricky. There will be two
choices:
i. EL function evaluation: will get the Action record and evaluate. Not an
efficient process. But we do it in other cases too such as File System based EL
functions.
ii. When WFInstance is loaded, update the memory using the stats datat
from WF Action table. At the same time, any modification of these stats needs
to be reflected in DB and memory. Extra works and memory issue.
Please comment on those choices.
> Oozie to allow drill down to hadoop job's details
> ---------------------------------------------------
>
> Key: OOZIE-588
> URL: https://issues.apache.org/jira/browse/OOZIE-588
> Project: Oozie
> Issue Type: New Feature
> Reporter: Mohammad Kamrul Islam
>
> High-level Requirements:
> -----------------------------------
> Since Oozie is designed as the gateway to grid, we need to support WS API for
> most common hadoop commands through Oozie. User doesn't want to go to
> multiple system to get the required data. Based on these, we propose to
> implement the following requirements into Oozie.
>
> R1: Oozie will provide WS endpoints to get hadoop job details (including job
> counters).
> R2: It will support both types of hadoop jobs : MR job created for MR action,
> MR jobs created as part of pig script.
> R3: In addition, for pig action, oozie will provide a way to query the pig
> stats.
> Proposed design:
> ----------------------
> D1: Oozie will store the *summary* jobcounter /pigstats into oozie DB.
> The items in the summary stats will be determined by oozie to limit the size.
> However,the commonly used stats will be include into the summary. It is
> important to note that summary information will be collected *after* the job
> finished.
>
> D2: If the user asks for *details* hadoop job stats , the user needs to
> query using different WS API. In this query, a user will specify *a* hadoop
> job id. Oozie will directly query the hadoop JT/RM/HS. Since it is an
> external call with undetermined response time, Oozie will provide only one
> hadoop job id per-request to avoid the timeout in WS call. Caveats: If hadoop
> is down or the job is not in JT/RM/History Server, Oozie will fail to collect
> the details.
>
> D3: For pig, Oozie will store the pig-generated hadoop ids in it DB and
> will expose that to user throw the "verbose" query.
> D4: Oozie will need to collect those summary pig stats and corresponding
> job counters and store it in Oozie DB. PigStats has a way of getting job
> counter for each hadoop job that it submits. We could use that API to collect
> summary counters for pig-created jobs.
> D5: The complete/detail pigstats will be stored into Pig Launcher Mapper
> as job counter. So that if a user wants to get the detail pig stats, we could
> get it from the LM directly.
>
> Open questions:
> ----------------------
> * What should be in the summary counters/stats?
> * What is the max size of stats?
> Advanced planning: <Not in the scope of this task, but might required for
> design to support later>
> --------------------------
> * Some users are asking to query the job stats when the job is RUNNING. They
> need it to decide for subsequent job submissions.
> * By the above design , user could use D2, to get the counter when MR action
> is running.
> * However, for pig, it is not that straight forward. Because Pig submits the
> jobs during execution. But the new PigRunner provide a listener concept where
> user can get the notifications such as when a new MR job submitted and its ID.
> * By using this, Oozie could get the running hadoop job id instantly. In
> future, user might want this to query using D2.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
