[
https://issues.apache.org/jira/browse/OOZIE-555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274168#comment-13274168
]
Virag Kothari commented on OOZIE-555:
-------------------------------------
Duplicate of oozie-830
> (H | M) Support multiple versions of jar
> ----------------------------------------
>
> Key: OOZIE-555
> URL: https://issues.apache.org/jira/browse/OOZIE-555
> Project: Oozie
> Issue Type: Task
> Reporter: Virag Kothari
>
> Background:
> -------------
> Currently oozie supports 3 ways of including jar files for any workflow. All
> these cases are getting file from hdfs
> directory add those jar into distributed cache.
> 1. System supported jars.
> 2. User-specific common jars.
> 3. Workflow specific jar.
> Current Shortcomings:
> ---------------------
> However, if the system wants to support multiple version of the same product
> (such as pig or hive), it is not possible
> in current oozie. In addition, oozie user often includes very old or
> non-supported jars (mostly unintended). This
> creates a lot of support issues. Id oozie could restrict (in some extent) the
> usage of version, this frequent support
> overhead could be minimized.
> Purpose:
> -------
> This JIRA is created for the following purposes:
> 1. Support multiple version of jars. In reality, for instance, there are
> multiple active version of pig jar. One might
> be in 'stable' and another might be immediate next version.
> 2. Enforce the usage of supported jars. For example, user could configure to
> use specific version of pig. If user
> doesn't provide any configuration, oozie will pick up the most stable jar
> (system configurable).
> It is important to note, that this proposed feature will not fully remove the
> usage of unsupported version of jar.
> Design Details:
> ===================
> 1. Every product could have a product specific sub-directory in the system
> lib dir. In that sub-directory, there could
> be multiple versions of jars. For example: <SYSTEM_LIB>/pig/0.7/lib,
> <SYSTEM_LIB>/pig/0.8/lib and
> <SYSTEM_LIB>/pig/stable/lib.
> 2. Need to modify the current way of including jar from SYSTEM_LIB.
> 3. In addition of including SYSTEM_LIB jars, oozie need to include the jar
> from the user-selected (or default) version.
> For example, if a user configure to use pig 0.8, oozie should include jar
> from <SYSTEM_LIB>/pig/0.8/lib/.jar. And
> if user doesn't configure for any specific pig version, oozie should include
> <SYSTEM_LIB>/pig/stable/lib/.jar.
> 4. If user specified some unsupported version jar, Oozie should throw
> exception with appropriate error message.
> 5. Oozie should include the product specific jar when it asked for that
> product. For example, oozie should include the
> pig jar through PigActionExecutor and hive jars through HiveActionExecutor.
> As a side effect,it will reduce the number
> of jars included in the Distributed Cache by selectively including the
> appropriate jar.
> 6. It will be the SE/OPS responsibility to maintain the supported versions of
> lib directories for any product.
> Implementation Details:
> -------------------------
> How to implement it in current Oozie?
> Once we agreed on the conceptual part, we could do the following changes in
> the code.
> 1. In place of putting all APP_LIB_PATH into
> WorkflowAppService.APP_LIB_PATH_LIST, we could create two such list.
> WorkflowAppService.SYSTEM_LIB_PATH_LIST that will contain only
> <SYSTEM_LIB/*.jar>.
> WorkflowAppService.APP_LIB_PATH_LIST will hold the rest (user-specific and
> wf/lib/.)
> 2. Modify JavaActionExecutor:setLibFilesArchives()
> .....
> String[] paths = getLibPaths(); //New method
> if (paths != null) {
> for (String path : paths) { addToCache(conf, appPath, path, false); }
> }
> ....
> }
> //New method :Base implementation
> protected String[] getLibPaths(...) { String[] paths =
> proto.getStrings(WorkflowAppService.APP_LIB_PATH_LIST); return
> paths; }
> 3. For example, in PigActionExecutor getLibPaths could be overridden using
> the following pseudo-code:
> protected String[] getLibPaths(...) {
> {
> String paths[] = super.getLibPaths();
> String path = services.getConf().get(SYSTEM_LIB_PATH, " ");
> if (path.trim().length() > 0) { systemLibPath = new Path(path.trim()); } else
> { return ..}
> String pigHome = systemLibPath = "/pig/" + usedVersion;
> List<String> libPaths = getLibFiles(fs, systemLibPath);
> return paths + libPaths;
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
