[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227308#comment-16227308
 ] 

Yufei Gu edited comment on MAPREDUCE-6994 at 10/31/17 6:44 PM:
---------------------------------------------------------------

Thanks [[email protected]] for working on this! Looks good 
generally. Here are my comments:
# The current design requires users to understand what directory should be 
collected by providing multiple directories for "input". For example, users 
needs to input 
$HADOOP_HOME/share/hadoop/client/*:$HADOOP_HOME/share/hadoop/common/*, etc. The 
benefit of this solution is that it is flexible no matter how input directories 
organized. However, the directory hierarchy in $HADOOP_HOME is fixed especially 
in upstream output, how about providing an option to just input a $HADOOP_HOME 
and tool can figure out which sub-directories to get jars? Seems like to make 
method {{collectPackages}} recursively traverse the input directory would be 
enough since there is predefined whitelist.
# Class {{UploaderException}} should be private. The projects other than Hadoop 
shouldn't use this class.
# Three functions are not used in class {{UploaderException}}. They are not 
necessary to be there. We can always add them back when we need them.
# {{DefaultJars}} should be a final class with private constructor. Other 
options would be make it an enum or an interface.
# I think we need to log an error message(e.g. "too many parameters") under 
this condition.
{code} if (parser.getRemainingArgs().length > 0) { printHelp(); return false; } 
{code}
# The help message needs some formatting, e.g. "input" needs to be in line with 
"fs". "where" needs some work as well. Let's put a real port# in the example, 
hdfs://foo.com:8020. The "classpath" should be "input". 
{code} 
Usage: mapred frameworkuploader -classpath <input> -fs <filesystem> -target 
<framework> [-whitelist <whitelist>] [-blacklist <blacklist>]
     where:
        input               input class path
       fs                   target file system to upload to. Example: 
hdfs://foo.com:port
       target               target file to upload to with a reference name. 
Example: /usr/mr-framework.tar.gz#mr-framework
       whitelist            Regex specifying the full path of jars to include 
in the framework tarball. Default is a hardcoded set of jars considered 
necessary to include
       blacklist            Regex specifying the full path of jars to exclude 
in the framework tarball. Default is a hardcoded set of jars considered 
unnecessary to include
{code} 
# Method {{validateTargetPath}} could throw the UploaderException instead of 
return a boolean.
# Logically method {{buildPackage()}} should be part of method {{upload()}}. 
How about call {{buildPackage()}} inside of {{upload()}}? In that case, 
{{targetStream}} could be local variable instead of a class member.
# Method {{parseWhiteList}} handles both white list and blacklist. You may 
split the method to two methods, or even better, one method takes {{whitelist}} 
or {{blacklist}} as a parameter and return {{whitelistedFiles}} or 
{{blacklistedFiles}} respectively.
# Need a VisibleForTesting annotation for method {{expandEnvironmentVariables}}.
# "maching" -> "matching" in line 269 of class {{TestFrameworkUploader}}
# assertEquals lines in method {{testEnvironmentReplacement}} and 
{{testRecursiveEnvironmentReplacement}} aren't necessary to take two lines.
# Method {{expandEnvironmentVariables()}} seems better to be in class {{Shell}}.



was (Author: yufeigu):
Thanks [[email protected]] for working on this! Looks good 
generally. Here are my comments:
# The current design requires users to understand what directory should be 
collected by providing multiple directories for "input". For example, users 
needs to input 
$HADOOP_HOME/share/hadoop/client/*:$HADOOP_HOME/share/hadoop/common/*, etc. The 
benefit of this solution is that it is flexible no matter how input directories 
organized. However, the directory hierarchy in $HADOOP_HOME is fixed especially 
in upstream output, how about providing an option to just input a $HADOOP_HOME 
and tool can figure out which sub-directories to get jars? Seems like to make 
method {{collectPackages}} recursively traverse the input directory would be 
enough since there is predefined whitelist.
# Class {{UploaderException}} should be private. The projects other than Hadoop 
shouldn't use this class.
# Three functions are not used in class {{UploaderException}}. They are not 
necessary to be there. We can always add them back when we need them.
# {{DefaultJars}} should be a final class with private constructor. Other 
options would be make it an enum or an interface.
# I think we need to log an error message(e.g. "too many parameters") under 
this condition.
{code} if (parser.getRemainingArgs().length > 0) { printHelp(); return false; } 
{code}
# The help message needs some formatting, e.g. "input" needs to be in line with 
"fs". "where" needs some work as well. Let's put a real port# in the example, 
hdfs://foo.com:8020. The "classpath" should be "input". 
{quote} 
Usage: mapred frameworkuploader -classpath <input> -fs <filesystem> -target 
<framework> [-whitelist <whitelist>] [-blacklist <blacklist>]
     where:
        input               input class path
       fs                   target file system to upload to. Example: 
hdfs://foo.com:port
       target               target file to upload to with a reference name. 
Example: /usr/mr-framework.tar.gz#mr-framework
       whitelist            Regex specifying the full path of jars to include 
in the framework tarball. Default is a hardcoded set of jars considered 
necessary to include
       blacklist            Regex specifying the full path of jars to exclude 
in the framework tarball. Default is a hardcoded set of jars considered 
unnecessary to include
{quote} 
# Method {{validateTargetPath}} could throw the UploaderException instead of 
return a boolean.
# Logically method {{buildPackage()}} should be part of method {{upload()}}. 
How about call {{buildPackage()}} inside of {{upload()}}? In that case, 
{{targetStream}} could be local variable instead of a class member.
# Method {{parseWhiteList}} handles both white list and blacklist. You may 
split the method to two methods, or even better, one method takes {{whitelist}} 
or {{blacklist}} as a parameter and return {{whitelistedFiles}} or 
{{blacklistedFiles}} respectively.
# Need a VisibleForTesting annotation for method {{expandEnvironmentVariables}}.
# "maching" -> "matching" in line 269 of class {{TestFrameworkUploader}}
# assertEquals lines in method {{testEnvironmentReplacement}} and 
{{testRecursiveEnvironmentReplacement}} aren't necessary to take two lines.
# Method {{expandEnvironmentVariables()}} seems better to be in class {{Shell}}.


> Uploader tool for Distributed Cache Deploy code changes
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-6994
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6994
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>            Reporter: Miklos Szegedi
>            Assignee: Miklos Szegedi
>         Attachments: MAPREDUCE-6994.000.patch
>
>
> The proposal is to create a tool that collects all available jars in the 
> Hadoop classpath and adds them to a single tarball file. It then uploads the 
> resulting archive to an HDFS directory. This saves the cluster administrator 
> from having to set this up manually for Distributed Cache Deploy.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to