[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14534371#comment-14534371 ] Sunil G commented on YARN-1019: --- Looks like Configuration.java in windows machine has some UTF chars and causing file saving problem,. i will rebase from a unix machine. > YarnConfiguration validation for local disk path and http addresses. > > > Key: YARN-1019 > URL: https://issues.apache.org/jira/browse/YARN-1019 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.0.5-alpha >Reporter: Omkar Vinit Joshi >Assignee: Sunil G >Priority: Minor > Labels: BB2015-05-TBR, newbie > Attachments: YARN-1019.0.patch > > > Today we are not validating certain configuration parameters set in > yarn-site.xml. 1) Configurations related to paths... such as local-dirs, > log-dirs.. Our NM crashes during startup if they are set to relative paths > rather than absolute paths. To avoid such failures we can enforce checks > (absolute paths) before startup . i.e. before we actually startup...( i.e. > directory handler creating directories). > 2) Also for all the parameters using "hostname:port" unless we are ok with > default port. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14534149#comment-14534149 ] Sunil G commented on YARN-1019: --- I would like to take over this as part of bugBash. Please reassign the same if you are working in it. I am writing test case for this fix and will upload a patch soon. > YarnConfiguration validation for local disk path and http addresses. > > > Key: YARN-1019 > URL: https://issues.apache.org/jira/browse/YARN-1019 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.0.5-alpha >Reporter: Omkar Vinit Joshi >Priority: Minor > Labels: BB2015-05-TBR, newbie > Attachments: YARN-1019.0.patch > > > Today we are not validating certain configuration parameters set in > yarn-site.xml. 1) Configurations related to paths... such as local-dirs, > log-dirs.. Our NM crashes during startup if they are set to relative paths > rather than absolute paths. To avoid such failures we can enforce checks > (absolute paths) before startup . i.e. before we actually startup...( i.e. > directory handler creating directories). > 2) Also for all the parameters using "hostname:port" unless we are ok with > default port. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524736#comment-14524736 ] Hadoop QA commented on YARN-1019: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | patch | 0m 0s | The patch command could not apply the patch during dryrun. | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12616524/YARN-1019.0.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / f1a152c | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7596/console | This message was automatically generated. > YarnConfiguration validation for local disk path and http addresses. > > > Key: YARN-1019 > URL: https://issues.apache.org/jira/browse/YARN-1019 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.0.5-alpha >Reporter: Omkar Vinit Joshi >Priority: Minor > Labels: newbie > Attachments: YARN-1019.0.patch > > > Today we are not validating certain configuration parameters set in > yarn-site.xml. 1) Configurations related to paths... such as local-dirs, > log-dirs.. Our NM crashes during startup if they are set to relative paths > rather than absolute paths. To avoid such failures we can enforce checks > (absolute paths) before startup . i.e. before we actually startup...( i.e. > directory handler creating directories). > 2) Also for all the parameters using "hostname:port" unless we are ok with > default port. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836397#comment-13836397 ] Hadoop QA commented on YARN-1019: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12616524/YARN-1019.0.patch against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2567//console This message is automatically generated. > YarnConfiguration validation for local disk path and http addresses. > > > Key: YARN-1019 > URL: https://issues.apache.org/jira/browse/YARN-1019 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.0.5-alpha >Reporter: Omkar Vinit Joshi >Priority: Minor > Labels: newbie > Attachments: YARN-1019.0.patch > > > Today we are not validating certain configuration parameters set in > yarn-site.xml. 1) Configurations related to paths... such as local-dirs, > log-dirs.. Our NM crashes during startup if they are set to relative paths > rather than absolute paths. To avoid such failures we can enforce checks > (absolute paths) before startup . i.e. before we actually startup...( i.e. > directory handler creating directories). > 2) Also for all the parameters using "hostname:port" unless we are ok with > default port. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13736098#comment-13736098 ] Joseph Kniest commented on YARN-1019: - So if I find this: public AppReportFetcher(Configuration conf) { this.conf = conf; YarnRPC rpc = YarnRPC.create(this.conf); InetSocketAddress rmAddress = conf.getSocketAddr( YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS, YarnConfiguration.DEFAULT_RM_PORT); . . . before that call at the bottom of the snipped should I have some logic that throws an exception if YarnConfiguration.RM_ADDRESS isn't in a valid format? Is that what we want to be doing in all the various places in the codebase where this object's fields are accessed? > YarnConfiguration validation for local disk path and http addresses. > > > Key: YARN-1019 > URL: https://issues.apache.org/jira/browse/YARN-1019 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Omkar Vinit Joshi >Priority: Minor > Labels: newbie > > Today we are not validating certain configuration parameters set in > yarn-site.xml. 1) Configurations related to paths... such as local-dirs, > log-dirs.. Our NM crashes during startup if they are set to relative paths > rather than absolute paths. To avoid such failures we can enforce checks > (absolute paths) before startup . i.e. before we actually startup...( i.e. > directory handler creating directories). > 2) Also for all the parameters using "hostname:port" unless we are ok with > default port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13731551#comment-13731551 ] Omkar Vinit Joshi commented on YARN-1019: - [~josephkniest] to give you more insight into how it is used (General configuration read in hadoop). bq. Ok so this module YarnConfiguration, does other portions of the codebase access this for the config info like directories and what not and I need to find all those places? Probably not. If you are using eclipse for hadoop development then just do call hierarchy for the variable under consideration. say, for YarnConfiguration#RM_ADDRESS. You will see where it is getting used. That way your search is reduced. You should here ignore places where it is getting used inside test code. You don't need to validate test code. But you will have to add unit test case later to verify your changes. bq. How does that information get passed to this object? Probably you don't need to worry about this. You can trace {code} new YarnConfiguration() {code} call. It will read from configuration files. for YARN : yarn-site.xml, HDFS :hdfs-site.xml, CORE : core-site.xml bq. Ultimately, we want to find where this object gets instantiated and ensure that it doesn't get relative paths correct? Yes for all the places where we are getting file paths we need to ensure this. Make sure it is not OS specific. i.e. it works for WINDOWS/LINUX/MAC. bq. What exactly do we want with number 2 of this issue? I'm confused about that one when we expect for example RM_ADDRESS then we expect it to be :. just validate it. Finally once you have done the changes you need to create patch and upload it via "More Actions --> Attach Files" and then Submit Patch. > YarnConfiguration validation for local disk path and http addresses. > > > Key: YARN-1019 > URL: https://issues.apache.org/jira/browse/YARN-1019 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Omkar Vinit Joshi >Priority: Minor > Labels: newbie > > Today we are not validating certain configuration parameters set in > yarn-site.xml. 1) Configurations related to paths... such as local-dirs, > log-dirs.. Our NM crashes during startup if they are set to relative paths > rather than absolute paths. To avoid such failures we can enforce checks > (absolute paths) before startup . i.e. before we actually startup...( i.e. > directory handler creating directories). > 2) Also for all the parameters using "hostname:port" unless we are ok with > default port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13731462#comment-13731462 ] Joseph Kniest commented on YARN-1019: - Ok so this module YarnConfiguration, does other portions of the codebase access this for the config info like directories and what not and I need to find all those places? How does that information get passed to this object? Ultimately, we want to find where this object gets instantiated and ensure that it doesn't get relative paths correct? What exactly do we want with number 2 of this issue? I'm confused about that one > YarnConfiguration validation for local disk path and http addresses. > > > Key: YARN-1019 > URL: https://issues.apache.org/jira/browse/YARN-1019 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Omkar Vinit Joshi >Priority: Minor > Labels: newbie > > Today we are not validating certain configuration parameters set in > yarn-site.xml. 1) Configurations related to paths... such as local-dirs, > log-dirs.. Our NM crashes during startup if they are set to relative paths > rather than absolute paths. To avoid such failures we can enforce checks > (absolute paths) before startup . i.e. before we actually startup...( i.e. > directory handler creating directories). > 2) Also for all the parameters using "hostname:port" unless we are ok with > default port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13731120#comment-13731120 ] Omkar Vinit Joshi commented on YARN-1019: - Start with YarnConfiguration.java. track all the places from where it is getting used and fix all path related and host:port checks. Also once done upload a patch. Someone will take a look at it. Make sure your patch file is something like "-..patch" format. It will help reviewers. Also make sure your code is formatted well. Make sure your changes are as minimal as possible. You are set then. Start contributing!! > YarnConfiguration validation for local disk path and http addresses. > > > Key: YARN-1019 > URL: https://issues.apache.org/jira/browse/YARN-1019 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Omkar Vinit Joshi >Priority: Minor > Labels: newbie > > Today we are not validating certain configuration parameters set in > yarn-site.xml. 1) Configurations related to paths... such as local-dirs, > log-dirs.. Our NM crashes during startup if they are set to relative paths > rather than absolute paths. To avoid such failures we can enforce checks > (absolute paths) before startup . i.e. before we actually startup...( i.e. > directory handler creating directories). > 2) Also for all the parameters using "hostname:port" unless we are ok with > default port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13731106#comment-13731106 ] Joseph Kniest commented on YARN-1019: - Thanks, I've done all that, built the latest from source, kicked off sample mapreduce job, looking for where this is handled in the code > YarnConfiguration validation for local disk path and http addresses. > > > Key: YARN-1019 > URL: https://issues.apache.org/jira/browse/YARN-1019 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Omkar Vinit Joshi >Priority: Minor > Labels: newbie > > Today we are not validating certain configuration parameters set in > yarn-site.xml. 1) Configurations related to paths... such as local-dirs, > log-dirs.. Our NM crashes during startup if they are set to relative paths > rather than absolute paths. To avoid such failures we can enforce checks > (absolute paths) before startup . i.e. before we actually startup...( i.e. > directory handler creating directories). > 2) Also for all the parameters using "hostname:port" unless we are ok with > default port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13731060#comment-13731060 ] Omkar Vinit Joshi commented on YARN-1019: - Hi, Welcome to yarn group. Probably you can get started from here [Checkout Code|http://wiki.apache.org/hadoop/HowToContribute]. Subscribe to user / dev mailing list and ask questions there (General questions such as how to checkout code/ issues running code). Here we usually discuss the current issue related problems. To get started. Run YARN..simple map reduce program. Once you are familiar with this you can take up one of the tickets marked as "newbie" and start working on that. > YarnConfiguration validation for local disk path and http addresses. > > > Key: YARN-1019 > URL: https://issues.apache.org/jira/browse/YARN-1019 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Omkar Vinit Joshi >Priority: Minor > Labels: newbie > > Today we are not validating certain configuration parameters set in > yarn-site.xml. 1) Configurations related to paths... such as local-dirs, > log-dirs.. Our NM crashes during startup if they are set to relative paths > rather than absolute paths. To avoid such failures we can enforce checks > (absolute paths) before startup . i.e. before we actually startup...( i.e. > directory handler creating directories). > 2) Also for all the parameters using "hostname:port" unless we are ok with > default port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
[ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13731053#comment-13731053 ] Joseph Kniest commented on YARN-1019: - Hi, new to yarn, where do I look in the code base for this? > YarnConfiguration validation for local disk path and http addresses. > > > Key: YARN-1019 > URL: https://issues.apache.org/jira/browse/YARN-1019 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Omkar Vinit Joshi >Priority: Minor > Labels: newbie > > Today we are not validating certain configuration parameters set in > yarn-site.xml. 1) Configurations related to paths... such as local-dirs, > log-dirs.. Our NM crashes during startup if they are set to relative paths > rather than absolute paths. To avoid such failures we can enforce checks > (absolute paths) before startup . i.e. before we actually startup...( i.e. > directory handler creating directories). > 2) Also for all the parameters using "hostname:port" unless we are ok with > default port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira