[
https://issues.apache.org/jira/browse/YARN-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795685#comment-13795685
]
Junping Du commented on YARN-1289:
----------------------------------
Thanks [~hitesh] and [~kkambatl] for comments. I just have some thoughts here,
see if it make sense.
bq. YARN does not and should not have any implicit dependencies on mapreduce.
Agree. However, according to my understanding, it doesn't stop YARN to support
mapreduce service by default which is still common sense today. Isn't it? There
already are some mapreduce related configurations and even with default values
there in yarn-default.xml. I just think we should keep consistent there (given
we have no plan to remove them out of yarn).
bq. If someone wanted to change the implementation of the shuffle service to a
potentially better/faster implementation, defining a default would create a
problem.
I don't see any problem there as default value is easily get overridden. Am I
missing something here?
bq. what is the expectation if MR in later versions switches to use a different
service? Is that expectation that default services will keep on changing over
time based on MR implementation changes?
I think keep default value changing with MR implementations is not a problem
today as project MR and YARN has the same release cycle and neither YARN or MR
get released separately (correct me if there is plan to separate them out). On
the contrary, without default settings, like this case, user has to spend time
to track the configuration changes from "mapreduce.shuffle" to
"mapreduce_shuffle". IMO, the value of default value is to save everyone's time
of setting common values which quite fit cases here.
Thoughts?
> Configuration "yarn.nodemanager.aux-services" should have default value for
> mapreduce_shuffle.
> ----------------------------------------------------------------------------------------------
>
> Key: YARN-1289
> URL: https://issues.apache.org/jira/browse/YARN-1289
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Reporter: wenwupeng
> Assignee: Junping Du
> Attachments: YARN-1289.patch
>
>
> Failed to run benchmark when not configure yarn.nodemanager.aux-services
> value in yarn-site.xml', it is better to configure default value.
> 13/10/09 22:19:23 INFO mapreduce.Job: Task Id :
> attempt_1381371516570_0001_m_000000_1, Status : FAILED
> Container launch failed for container_1381371516570_0001_01_000005 :
> org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The
> auxService:mapreduce_shuffle does not exist
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
