[
https://issues.apache.org/jira/browse/FALCON-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shwetha G S updated FALCON-510:
-------------------------------
Description:
Currently there is no falcon context injected at MR job level. The job conf has
at most the oozie workflow / action ID either in the job name or sometimes in
the job conf.
Therefore there is no way for a tool like hraven, which relies completely on
jobconf and job history data, to identify that a particular job maps to a
particular falcon process or it's instance time, etc. Right now hraven does
regex-based job name surgery on a best effort basis before emitting metrics to
graphite
Request the following feature in falcon:
Add the following properties to the job conf (for all jobs - be it a pig action
or an MR action):
falcon.process.name
falcon.process.instancetime
while we're at it, might as well add any other falcon context as a jobconf
property (like whether it was a rerun or the input/output feeds, cluster,
validity, any process properties, etc.)
This will ofcourse inject at the first job level and cannot ensure that any
child jobs get the properties passed on (unless we can figure out a way to do
that too).
was:
Currently there is no falcon context injected at MR job level. The job conf has
at most the oozie workflow / action ID either in the job name or sometimes in
the job conf.
Therefore there is no way for a tool like hraven, which relies completely on
jobconf and job history data, to identify that a particular job maps to a
particular falcon process or it's instance time, etc. Right now hraven does
regex-based job name surgery on a best effort basis before emitting metrics to
graphite (can see the messy outcome at http://graphite.grid.ua2.inmobi.com)
Request the following feature in falcon:
Add the following properties to the job conf (for all jobs - be it a pig action
or an MR action):
falcon.process.name
falcon.process.instancetime
while we're at it, might as well add any other falcon context as a jobconf
property (like whether it was a rerun or the input/output feeds, cluster,
validity, any process properties, etc.)
This will ofcourse inject at the first job level and cannot ensure that any
child jobs get the properties passed on (unless we can figure out a way to do
that too).
> Inject falcon related properties to job conf
> --------------------------------------------
>
> Key: FALCON-510
> URL: https://issues.apache.org/jira/browse/FALCON-510
> Project: Falcon
> Issue Type: Improvement
> Reporter: Shwetha G S
>
> Currently there is no falcon context injected at MR job level. The job conf
> has at most the oozie workflow / action ID either in the job name or
> sometimes in the job conf.
> Therefore there is no way for a tool like hraven, which relies completely on
> jobconf and job history data, to identify that a particular job maps to a
> particular falcon process or it's instance time, etc. Right now hraven does
> regex-based job name surgery on a best effort basis before emitting metrics
> to graphite
> Request the following feature in falcon:
> Add the following properties to the job conf (for all jobs - be it a pig
> action or an MR action):
> falcon.process.name
> falcon.process.instancetime
> while we're at it, might as well add any other falcon context as a jobconf
> property (like whether it was a rerun or the input/output feeds, cluster,
> validity, any process properties, etc.)
> This will ofcourse inject at the first job level and cannot ensure that any
> child jobs get the properties passed on (unless we can figure out a way to do
> that too).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
