[
https://issues.apache.org/jira/browse/OOZIE-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13728375#comment-13728375
]
Mona Chitnis commented on OOZIE-1486:
-------------------------------------
As starters, I have analyzed the LauncherMapper flow, and I think the following
files can be combined into one, with a prefix for each different type of
property, enabling detection of whether that type of property has been recorded
or not (instead of checking each individual file and uploading to HDFS right
now)
# output.properties
# stats.properties
# externalChildIds.properties
# newId.properties
# error.properties
Other files - "recovery.id" and "action.conf.xml" are required in the early
configure() phase of a mapper and so should be kept separate from this combo
file, that is referred to only when job task ends.
> Cut down on number of small files created to track a running action
> --------------------------------------------------------------------
>
> Key: OOZIE-1486
> URL: https://issues.apache.org/jira/browse/OOZIE-1486
> Project: Oozie
> Issue Type: Improvement
> Affects Versions: trunk
> Reporter: Mona Chitnis
> Assignee: Mona Chitnis
> Fix For: trunk
>
>
> Oozie creates multiple files while running a action. This has been observed
> to be an overkill and can be consolidated (as applicable) into a lesser
> files. Advantages involve not only staying within user storage quotas but
> also reducing Namenode pressure in a large production environment.
> static final String ACTION_CONF_XML = "action.xml";
> public static final String ACTION_PREPARE_XML =
> "oozie.action.prepare.xml";
> private static final String ACTION_OUTPUT_PROPS = "output.properties";
> private static final String ACTION_STATS_PROPS = "stats.properties";
> private static final String ACTION_EXTERNAL_CHILD_IDS_PROPS =
> "externalChildIds.properties";
> private static final String ACTION_NEW_ID_PROPS = "newId.properties";
> private static final String ACTION_ERROR_PROPS = "error.properties";
> Consolidate and reduce the number of files required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
