[
https://issues.apache.org/jira/browse/YARN-9899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-9899:
-------------------------------
Description:
YARN-9699 laid down the groundworks of a converter from FS to CS config.
During the development of the converter, we came up with the following things
to fix.
1. If we don't specify a mandatory option, we have this stacktrace for example:
{code:java}
org.apache.commons.cli.MissingOptionException: Missing required option: o
at org.apache.commons.cli.Parser.checkRequiredOptions(Parser.java:299)
at org.apache.commons.cli.Parser.parse(Parser.java:231)
at org.apache.commons.cli.Parser.parse(Parser.java:85)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:100)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1572){code}
We should provide a more concise and meaningful error message (without
stacktrace on the CLI, but we should log the exception with stacktrace to the
RM log).
An explanation of the missing option is also required.
2. We may think about how to handle exceptions from commons CLI:
MissingArgumentException vs. MissingOptionException
3. We need to provide a -h / --help option for the CLI that prints all the
possible options / arguments.
4. Last but not least: We should move the CLI command to a more reasonable
place:
As YARN-9699 implemented it, the command can be invoked like:
{code:java}
/opt/hadoop/bin/yarn resourcemanager -convert-fs-configuration -y
/opt/hadoop/etc/hadoop/yarn-site.xml -f
/opt/hadoop/etc/hadoop/fair-scheduler.xml -r
~systest/sample-rules-config.properties -o /tmp/fs-cs-output
{code}
This is problematic, as if YARN RM is already running, we need to stop it in
order to start the RM again with the conversion switch.
5. Add unit test coverage for {{QueuePlacementConverter}}
6. Close some feature gaps.
was:
YARN-9699 laid down the groundworks of a converter from FS to CS config.
During the development of the converter, we came up with the following things
to fix.
1. If we don't specify a mandatory option, we have this stacktrace for example:
{code:java}
org.apache.commons.cli.MissingOptionException: Missing required option: o
at org.apache.commons.cli.Parser.checkRequiredOptions(Parser.java:299)
at org.apache.commons.cli.Parser.parse(Parser.java:231)
at org.apache.commons.cli.Parser.parse(Parser.java:85)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:100)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1572){code}
We should provide a more concise and meaningful error message (without
stacktrace on the CLI, but we should log the exception with stacktrace to the
RM log).
An explanation of the missing option is also required.
2. We may think about how to handle exceptions from commons CLI:
MissingArgumentException vs. MissingOptionException
3. We need to provide a -h / --help option for the CLI that prints all the
possible options / arguments.
4. Last but not least: We should move the CLI command to a more reasonable
place:
As YARN-9699 implemented it, the command can be invoked like:
{code:java}
/opt/hadoop/bin/yarn resourcemanager -convert-fs-configuration -y
/opt/hadoop/etc/hadoop/yarn-site.xml -f
/opt/hadoop/etc/hadoop/fair-scheduler.xml -r
~systest/sample-rules-config.properties -o /tmp/fs-cs-output
{code}
This is problematic, as if YARN RM is already running, we need to stop it in
order to start the RM again with the conversion switch.
5. Close some feature gaps.
> Migration tool that help to generate CS config based on FS config [Phase 2]
> ----------------------------------------------------------------------------
>
> Key: YARN-9899
> URL: https://issues.apache.org/jira/browse/YARN-9899
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Szilard Nemeth
> Assignee: Peter Bacsko
> Priority: Major
>
> YARN-9699 laid down the groundworks of a converter from FS to CS config.
> During the development of the converter, we came up with the following things
> to fix.
> 1. If we don't specify a mandatory option, we have this stacktrace for
> example:
>
> {code:java}
> org.apache.commons.cli.MissingOptionException: Missing required option: o
> at org.apache.commons.cli.Parser.checkRequiredOptions(Parser.java:299)
> at org.apache.commons.cli.Parser.parse(Parser.java:231)
> at org.apache.commons.cli.Parser.parse(Parser.java:85)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:100)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1572){code}
>
> We should provide a more concise and meaningful error message (without
> stacktrace on the CLI, but we should log the exception with stacktrace to the
> RM log).
> An explanation of the missing option is also required.
> 2. We may think about how to handle exceptions from commons CLI:
> MissingArgumentException vs. MissingOptionException
> 3. We need to provide a -h / --help option for the CLI that prints all the
> possible options / arguments.
> 4. Last but not least: We should move the CLI command to a more reasonable
> place:
> As YARN-9699 implemented it, the command can be invoked like:
> {code:java}
> /opt/hadoop/bin/yarn resourcemanager -convert-fs-configuration -y
> /opt/hadoop/etc/hadoop/yarn-site.xml -f
> /opt/hadoop/etc/hadoop/fair-scheduler.xml -r
> ~systest/sample-rules-config.properties -o /tmp/fs-cs-output
> {code}
> This is problematic, as if YARN RM is already running, we need to stop it in
> order to start the RM again with the conversion switch.
> 5. Add unit test coverage for {{QueuePlacementConverter}}
> 6. Close some feature gaps.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
