[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534149#comment-14534149 ] Sunil G commented on YARN-1019: --- I would like to take over this as part of bugBash. Please reassign the same if you are working in it. I am writing test case for this fix and will upload a patch soon. YarnConfiguration validation for local disk path and http addresses. Key: YARN-1019 URL: https://issues.apache.org/jira/browse/YARN-1019 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.5-alpha Reporter: Omkar Vinit Joshi Priority: Minor Labels: BB2015-05-TBR, newbie Attachments: YARN-1019.0.patch Today we are not validating certain configuration parameters set in yarn-site.xml. 1) Configurations related to paths... such as local-dirs, log-dirs.. Our NM crashes during startup if they are set to relative paths rather than absolute paths. To avoid such failures we can enforce checks (absolute paths) before startup . i.e. before we actually startup...( i.e. directory handler creating directories). 2) Also for all the parameters using hostname:port unless we are ok with default port. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534371#comment-14534371 ] Sunil G commented on YARN-1019: --- Looks like Configuration.java in windows machine has some UTF chars and causing file saving problem,. i will rebase from a unix machine. YarnConfiguration validation for local disk path and http addresses. Key: YARN-1019 URL: https://issues.apache.org/jira/browse/YARN-1019 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.5-alpha Reporter: Omkar Vinit Joshi Assignee: Sunil G Priority: Minor Labels: BB2015-05-TBR, newbie Attachments: YARN-1019.0.patch Today we are not validating certain configuration parameters set in yarn-site.xml. 1) Configurations related to paths... such as local-dirs, log-dirs.. Our NM crashes during startup if they are set to relative paths rather than absolute paths. To avoid such failures we can enforce checks (absolute paths) before startup . i.e. before we actually startup...( i.e. directory handler creating directories). 2) Also for all the parameters using hostname:port unless we are ok with default port. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[
https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524736#comment-14524736
]
Hadoop QA commented on YARN-1019:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12616524/YARN-1019.0.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7596/console |
This message was automatically generated.
YarnConfiguration validation for local disk path and http addresses.
Key: YARN-1019
URL: https://issues.apache.org/jira/browse/YARN-1019
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.0.5-alpha
Reporter: Omkar Vinit Joshi
Priority: Minor
Labels: newbie
Attachments: YARN-1019.0.patch
Today we are not validating certain configuration parameters set in
yarn-site.xml. 1) Configurations related to paths... such as local-dirs,
log-dirs.. Our NM crashes during startup if they are set to relative paths
rather than absolute paths. To avoid such failures we can enforce checks
(absolute paths) before startup . i.e. before we actually startup...( i.e.
directory handler creating directories).
2) Also for all the parameters using hostname:port unless we are ok with
default port.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[
https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13836397#comment-13836397
]
Hadoop QA commented on YARN-1019:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616524/YARN-1019.0.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2567//console
This message is automatically generated.
YarnConfiguration validation for local disk path and http addresses.
Key: YARN-1019
URL: https://issues.apache.org/jira/browse/YARN-1019
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.0.5-alpha
Reporter: Omkar Vinit Joshi
Priority: Minor
Labels: newbie
Attachments: YARN-1019.0.patch
Today we are not validating certain configuration parameters set in
yarn-site.xml. 1) Configurations related to paths... such as local-dirs,
log-dirs.. Our NM crashes during startup if they are set to relative paths
rather than absolute paths. To avoid such failures we can enforce checks
(absolute paths) before startup . i.e. before we actually startup...( i.e.
directory handler creating directories).
2) Also for all the parameters using hostname:port unless we are ok with
default port.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[
https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736098#comment-13736098
]
Joseph Kniest commented on YARN-1019:
-
So if I find this:
public AppReportFetcher(Configuration conf) {
this.conf = conf;
YarnRPC rpc = YarnRPC.create(this.conf);
InetSocketAddress rmAddress = conf.getSocketAddr(
YarnConfiguration.RM_ADDRESS,
YarnConfiguration.DEFAULT_RM_ADDRESS,
YarnConfiguration.DEFAULT_RM_PORT);
.
.
.
before that call at the bottom of the snipped should I have some logic that
throws an exception if YarnConfiguration.RM_ADDRESS isn't in a valid format? Is
that what we want to be doing in all the various places in the codebase where
this object's fields are accessed?
YarnConfiguration validation for local disk path and http addresses.
Key: YARN-1019
URL: https://issues.apache.org/jira/browse/YARN-1019
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Omkar Vinit Joshi
Priority: Minor
Labels: newbie
Today we are not validating certain configuration parameters set in
yarn-site.xml. 1) Configurations related to paths... such as local-dirs,
log-dirs.. Our NM crashes during startup if they are set to relative paths
rather than absolute paths. To avoid such failures we can enforce checks
(absolute paths) before startup . i.e. before we actually startup...( i.e.
directory handler creating directories).
2) Also for all the parameters using hostname:port unless we are ok with
default port.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731053#comment-13731053 ] Joseph Kniest commented on YARN-1019: - Hi, new to yarn, where do I look in the code base for this? YarnConfiguration validation for local disk path and http addresses. Key: YARN-1019 URL: https://issues.apache.org/jira/browse/YARN-1019 Project: Hadoop YARN Issue Type: Improvement Reporter: Omkar Vinit Joshi Priority: Minor Labels: newbie Today we are not validating certain configuration parameters set in yarn-site.xml. 1) Configurations related to paths... such as local-dirs, log-dirs.. Our NM crashes during startup if they are set to relative paths rather than absolute paths. To avoid such failures we can enforce checks (absolute paths) before startup . i.e. before we actually startup...( i.e. directory handler creating directories). 2) Also for all the parameters using hostname:port unless we are ok with default port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731060#comment-13731060 ] Omkar Vinit Joshi commented on YARN-1019: - Hi, Welcome to yarn group. Probably you can get started from here [Checkout Code|http://wiki.apache.org/hadoop/HowToContribute]. Subscribe to user / dev mailing list and ask questions there (General questions such as how to checkout code/ issues running code). Here we usually discuss the current issue related problems. To get started. Run YARN..simple map reduce program. Once you are familiar with this you can take up one of the tickets marked as newbie and start working on that. YarnConfiguration validation for local disk path and http addresses. Key: YARN-1019 URL: https://issues.apache.org/jira/browse/YARN-1019 Project: Hadoop YARN Issue Type: Improvement Reporter: Omkar Vinit Joshi Priority: Minor Labels: newbie Today we are not validating certain configuration parameters set in yarn-site.xml. 1) Configurations related to paths... such as local-dirs, log-dirs.. Our NM crashes during startup if they are set to relative paths rather than absolute paths. To avoid such failures we can enforce checks (absolute paths) before startup . i.e. before we actually startup...( i.e. directory handler creating directories). 2) Also for all the parameters using hostname:port unless we are ok with default port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731106#comment-13731106 ] Joseph Kniest commented on YARN-1019: - Thanks, I've done all that, built the latest from source, kicked off sample mapreduce job, looking for where this is handled in the code YarnConfiguration validation for local disk path and http addresses. Key: YARN-1019 URL: https://issues.apache.org/jira/browse/YARN-1019 Project: Hadoop YARN Issue Type: Improvement Reporter: Omkar Vinit Joshi Priority: Minor Labels: newbie Today we are not validating certain configuration parameters set in yarn-site.xml. 1) Configurations related to paths... such as local-dirs, log-dirs.. Our NM crashes during startup if they are set to relative paths rather than absolute paths. To avoid such failures we can enforce checks (absolute paths) before startup . i.e. before we actually startup...( i.e. directory handler creating directories). 2) Also for all the parameters using hostname:port unless we are ok with default port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731120#comment-13731120 ] Omkar Vinit Joshi commented on YARN-1019: - Start with YarnConfiguration.java. track all the places from where it is getting used and fix all path related and host:port checks. Also once done upload a patch. Someone will take a look at it. Make sure your patch file is something like jira-number-date-in--mm-dd.number.patch format. It will help reviewers. Also make sure your code is formatted well. Make sure your changes are as minimal as possible. You are set then. Start contributing!! YarnConfiguration validation for local disk path and http addresses. Key: YARN-1019 URL: https://issues.apache.org/jira/browse/YARN-1019 Project: Hadoop YARN Issue Type: Improvement Reporter: Omkar Vinit Joshi Priority: Minor Labels: newbie Today we are not validating certain configuration parameters set in yarn-site.xml. 1) Configurations related to paths... such as local-dirs, log-dirs.. Our NM crashes during startup if they are set to relative paths rather than absolute paths. To avoid such failures we can enforce checks (absolute paths) before startup . i.e. before we actually startup...( i.e. directory handler creating directories). 2) Also for all the parameters using hostname:port unless we are ok with default port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731462#comment-13731462 ] Joseph Kniest commented on YARN-1019: - Ok so this module YarnConfiguration, does other portions of the codebase access this for the config info like directories and what not and I need to find all those places? How does that information get passed to this object? Ultimately, we want to find where this object gets instantiated and ensure that it doesn't get relative paths correct? What exactly do we want with number 2 of this issue? I'm confused about that one YarnConfiguration validation for local disk path and http addresses. Key: YARN-1019 URL: https://issues.apache.org/jira/browse/YARN-1019 Project: Hadoop YARN Issue Type: Improvement Reporter: Omkar Vinit Joshi Priority: Minor Labels: newbie Today we are not validating certain configuration parameters set in yarn-site.xml. 1) Configurations related to paths... such as local-dirs, log-dirs.. Our NM crashes during startup if they are set to relative paths rather than absolute paths. To avoid such failures we can enforce checks (absolute paths) before startup . i.e. before we actually startup...( i.e. directory handler creating directories). 2) Also for all the parameters using hostname:port unless we are ok with default port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[
https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731551#comment-13731551
]
Omkar Vinit Joshi commented on YARN-1019:
-
[~josephkniest] to give you more insight into how it is used (General
configuration read in hadoop).
bq. Ok so this module YarnConfiguration, does other portions of the codebase
access this for the config info like directories and what not and I need to
find all those places?
Probably not. If you are using eclipse for hadoop development then just do call
hierarchy for the variable under consideration. say, for
YarnConfiguration#RM_ADDRESS. You will see where it is getting used. That way
your search is reduced. You should here ignore places where it is getting used
inside test code. You don't need to validate test code. But you will have to
add unit test case later to verify your changes.
bq. How does that information get passed to this object?
Probably you don't need to worry about this. You can trace
{code}
new YarnConfiguration()
{code}
call. It will read from configuration files. for YARN : yarn-site.xml, HDFS
:hdfs-site.xml, CORE : core-site.xml
bq. Ultimately, we want to find where this object gets instantiated and ensure
that it doesn't get relative paths correct?
Yes for all the places where we are getting file paths we need to ensure this.
Make sure it is not OS specific. i.e. it works for WINDOWS/LINUX/MAC.
bq. What exactly do we want with number 2 of this issue? I'm confused about
that one
when we expect for example RM_ADDRESS then we expect it to be host:port.
just validate it.
Finally once you have done the changes you need to create patch and upload it
via More Actions -- Attach Files and then Submit Patch.
YarnConfiguration validation for local disk path and http addresses.
Key: YARN-1019
URL: https://issues.apache.org/jira/browse/YARN-1019
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Omkar Vinit Joshi
Priority: Minor
Labels: newbie
Today we are not validating certain configuration parameters set in
yarn-site.xml. 1) Configurations related to paths... such as local-dirs,
log-dirs.. Our NM crashes during startup if they are set to relative paths
rather than absolute paths. To avoid such failures we can enforce checks
(absolute paths) before startup . i.e. before we actually startup...( i.e.
directory handler creating directories).
2) Also for all the parameters using hostname:port unless we are ok with
default port.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
