[
https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308135#comment-14308135
]
Chris Nauroth commented on HADOOP-11485:
----------------------------------------
Hi, [~aw]. Thank you for doing this. This looks great.
I took it for a test run with a concrete use case that I mentioned on another
jira: adding hadoop-azure to the classpath at distro time. Everything worked
as advertised in the example. I read through that and then dropped the
following file at shellprofile.d/hadoop_azure:
{code}
hadoop_add_profile hadoop_azure
function _hadoop_azure_hadoop_classpath
{
hadoop_add_classpath
"${HADOOP_PREFIX}/share/hadoop/tools/lib/hadoop-azure-3.0.0-SNAPSHOT.jar" after
hadoop_add_classpath
"${HADOOP_PREFIX}/share/hadoop/tools/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar"
after
}
{code}
With that in place, I can run fs shell commands that access paths in Azure
Storage. This is easier than the existing options for trying to achieve this
kind of customization.
I have only 2 meta-comments, which don't necessarily need to be addressed in
the scope of this jira:
# With the introduction of shell profiles, we essentially have a new public API
contract. We'll need to maintain compatibility so that we don't break existing
shell profiles during routine code maintenance. (i.e. {{hadoop_add_classpath}}
can't be renamed within a major version). Maybe we need a commenting
convention, similar to the interface visibility and stability annotations that
we use in Java, to help guide developers.
# I think we'll also eventually need user-facing documentation on
hadoop.apache.org to describe the new functionality in the shell rewrite. For
shell profile implementors, something like API docs would be helpful to let
people know about useful things like {{hadoop_debug}}.
I think the current patch needs a small rebase. After that, +1.
> Pluggable shell integration
> ---------------------------
>
> Key: HADOOP-11485
> URL: https://issues.apache.org/jira/browse/HADOOP-11485
> Project: Hadoop Common
> Issue Type: New Feature
> Components: scripts
> Affects Versions: 3.0.0
> Reporter: Allen Wittenauer
> Assignee: Allen Wittenauer
> Labels: scripts, shell
> Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch,
> HADOOP-11485-02.patch
>
>
> It would be useful to provide a way for core and non-core Hadoop components
> to plug into the shell infrastructure. This would allow us to pull the HDFS,
> MapReduce, and YARN shell functions out of hadoop-functions.sh.
> Additionally, it should let 3rd parties such as HBase influence things like
> classpaths at runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
