[
https://issues.apache.org/jira/browse/YARN-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972851#comment-15972851
]
Billie Rinaldi commented on YARN-6335:
--------------------------------------
bq. Looks like the "yarn.resource.normalization.enabled" will only control
whether slider AM will normalize the request at client side or not. But,
regardless of this setting, the normalization will always be done in YARN
scheduler to round up the resource size.
This normalization is specifically for capping the resource request at the
maximum value allowed by YARN. Slider used to automatically lower the resource
request when it was too high, but some users ran into issues because their app
could not run with lower resources. They preferred the app to fail due to
requesting resources that were too high, which is why this parameter was
introduced.
bq. What's the difference between these two paths: the path defined in
deleteZookeeperNode and the path defined by registryPathForInstance ?
The first one is the ZK node created for the app's use, separate from the
registry nodes that are used by Slider. In Slider, the DEFAULT_ZK_PATH variable
was set for the app here:
https://github.com/apache/incubator-slider/blob/develop/slider-core/src/main/java/org/apache/slider/client/SliderClient.java#L1751-L1775
and here:
https://github.com/apache/incubator-slider/blob/develop/slider-core/src/main/java/org/apache/slider/core/build/InstanceBuilder.java#L491
but it looks like this code has been removed in YARN native services. We should
reintroduce this functionality.
bq. Even if we check the existence of app Dir at last line of the method, if
the create happens to be done right after this check and before the method
returns, it is still the same problem. To user this just looks as if the create
immediately happens after destroy. It is a micro optimization, but the
semantics is still not deterministic.
Okay, I will remove the last check. I am more concerned about the case where
something went wrong on the Slider side, rather than a create/destroy race
condition on user side. It is very frustrating as a user for destroy to succeed
and the HDFS directory still to exist, preventing creation of the app again.
But since we are checking the return value of fs.delete, this shouldn't happen.
> Port slider's groovy unit tests to yarn native services
> -------------------------------------------------------
>
> Key: YARN-6335
> URL: https://issues.apache.org/jira/browse/YARN-6335
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Billie Rinaldi
> Assignee: Billie Rinaldi
> Fix For: yarn-native-services
>
> Attachments: YARN-6335-yarn-native-services.001.patch,
> YARN-6335-yarn-native-services.002.patch,
> YARN-6335-yarn-native-services.003.patch,
> YARN-6335-yarn-native-services.004.patch,
> YARN-6335-yarn-native-services.005.patch
>
>
> Slider has a lot of useful unit tests implemented in groovy. We could convert
> these to Java for YARN native services. This scope of this ticket will
> include unit / minicluster tests only and will not include Slider's funtests
> which require a running cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
