[
https://issues.apache.org/jira/browse/BIGTOP-852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583440#comment-13583440
]
Konstantin Boudnik commented on BIGTOP-852:
-------------------------------------------
Here are few points to start with:
- Do not create user 'jenkins' in the package: we shouldn't shape the file
system this way for a user without knowing the use case.
- lose shell-outs: use groovy script to directly call in file system API for
the meta and check operations. You will speed it up for about 300% easily
- the script in the current way doesn't solve the lack of YARN-required
directories. Hence, you won't be able to use HDFS shaped in this way to run MR
jobs. In fact, nodemanager won't even start. The original patch
https://issues.apache.org/jira/secure/attachment/12569655/BIGTOP-637.patch has
all the details.
> Improve HDFS helper script
> --------------------------
>
> Key: BIGTOP-852
> URL: https://issues.apache.org/jira/browse/BIGTOP-852
> Project: Bigtop
> Issue Type: Bug
> Components: Deployment
> Affects Versions: 0.6.0
> Reporter: Mark Grover
> Fix For: 0.6.0
>
>
> There are a few things that can be improved regarding the HDFS helper script
> introduced by BIGTOP-547. Some of these issues have been discussed in
> BIGTOP-637.
> 1. The script seems to create user directories for users "root" and "jenkins"
> but not the current user running the script. I think it will be a good idea
> to add in the script the commands to create the /user/$USER directory in
> HDFS. Of course, we should be careful in case the user running the command is
> root or jenkins, in that case a simple mkdir command would throw error given
> that those directories already exist.
> 2. The script uses sudo which requires login shell. However, login shells are
> not available for certain use cases like init scripts. Consequently, we
> should consider using su instead.
> 3. Look into how the helper script can be made faster
> Folks, please feel free to add/edit if there is something I missed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
