Hsin-Liang Huang created YARN-8315:
--------------------------------------
Summary: HDP 3.0.0 perfromance is slower than HDP 2.6.4
Key: YARN-8315
URL: https://issues.apache.org/jira/browse/YARN-8315
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 3.0.0
Environment: I have a HDP 2.6.4 cluster and HDP 3.0.0 cluster, I set
up to have the same settings for these two cluster such as java heap size,
container size etc. They are both 4 node cluster with 3 data nodes. I took
almost all the default setting on HDP 3.0.0 except that I modify the minimum
container size to 64MB instead of 1024MB in both cluster.
Reporter: Hsin-Liang Huang
Hi, I am comparing the performance between HDP 3.0.0 and HDP 2.6.4 and I
discovered HDP 3.0.0 is much slower than HDP 2.6.4 if the job acquire more yarn
containers and we also pin point the problem is after the job is done, when it
tried to clean up all the containers to exit the application, that's the place
where it consumed more time than HDP 2.6.4. I used the simple yarn app that
Hortonworks put out on github [https://github.com/hortonworks/simple-yarn-app]
to do the testing. Below is my testing result from acquiring 8 containers in
both HDP 3.0.0 and HDP 2.6.4 cluster environment.
=============================================================
HDP 3.0.0:
command: time hadoop jar
/usr/hdp/3.0.0.0-829/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.0.0.3.0.0.0-829.jar
Client -classpath simple-yarn-app-1.1.0.jar -cmd "java
com.hortonworks.simpleyarnapp.ApplicationMaster /bin/date 8"
18/05/17 11:06:42 INFO unmanagedamlauncher.UnmanagedAMLauncher: Initializing
Client
18/05/17 11:06:42 INFO unmanagedamlauncher.UnmanagedAMLauncher: Starting Client
18/05/17 11:06:43 INFO client.RMProxy: Connecting to ResourceManager at
whiny1.fyre.ibm.com/172.16.165.211:8050
18/05/17 11:06:43 INFO client.AHSProxy: Connecting to Application History
server at whiny2.fyre.ibm.com/172.16.200.160:10200
18/05/17 11:06:43 INFO unmanagedamlauncher.UnmanagedAMLauncher: Setting up
application submission context for ASM
18/05/17 11:06:43 INFO unmanagedamlauncher.UnmanagedAMLauncher: Setting
unmanaged AM
18/05/17 11:06:43 INFO unmanagedamlauncher.UnmanagedAMLauncher: Submitting
application to ASM
18/05/17 11:06:43 INFO impl.YarnClientImpl: Submitted application
application_1526572577866_0011
18/05/17 11:06:44 INFO unmanagedamlauncher.UnmanagedAMLauncher: Got application
report from ASM for, appId=11,
appAttemptId=appattempt_1526572577866_0011_000001, clientToAMToken=null,
appDiagnostics=AM container is launched, waiting for AM container to Register
with RM, appMasterHost=N/A, appQueue=default, appMasterRpcPort=-1,
appStartTime=1526584003704, yarnAppState=ACCEPTED,
distributedFinalState=UNDEFINED, appTrackingUrl=N/A, appUser=hlhuang
18/05/17 11:06:44 INFO unmanagedamlauncher.UnmanagedAMLauncher: Launching AM
with application attempt id appattempt_1526572577866_0011_000001
18/05/17 11:06:46 INFO client.RMProxy: Connecting to ResourceManager at
whiny1.fyre.ibm.com/172.16.165.211:8030
registerApplicationMaster 0
registerApplicationMaster 1
18/05/17 11:06:47 INFO conf.Configuration: found resource resource-types.xml at
file:/etc/hadoop/3.0.0.0-829/0/resource-types.xml
Making res-req 0
Making res-req 1
Making res-req 2
Making res-req 3
Making res-req 4
Making res-req 5
Making res-req 6
Making res-req 7
Launching container container_e08_1526572577866_0011_01_000001
Launching container container_e08_1526572577866_0011_01_000002
Launching container container_e08_1526572577866_0011_01_000003
Launching container container_e08_1526572577866_0011_01_000004
Launching container container_e08_1526572577866_0011_01_000005
Launching container container_e08_1526572577866_0011_01_000006
Launching container container_e08_1526572577866_0011_01_000007
Launching container container_e08_1526572577866_0011_01_000008
Completed container container_e08_1526572577866_0011_01_000001
Completed container container_e08_1526572577866_0011_01_000002
Completed container container_e08_1526572577866_0011_01_000003
Completed container container_e08_1526572577866_0011_01_000004
Completed container container_e08_1526572577866_0011_01_000008
Completed container container_e08_1526572577866_0011_01_000005
Completed container container_e08_1526572577866_0011_01_000006
Completed container container_e08_1526572577866_0011_01_000007
18/05/17 11:06:54 INFO unmanagedamlauncher.UnmanagedAMLauncher: AM process
exited with value: 0
18/05/17 11:06:55 INFO unmanagedamlauncher.UnmanagedAMLauncher: Got application
report from ASM for, appId=11,
appAttemptId=appattempt_1526572577866_0011_000001, clientToAMToken=null,
appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0,
appStartTime=1526584003704, yarnAppState=FINISHED,
distributedFinalState=SUCCEEDED, appTrackingUrl=N/A, appUser=hlhuang
18/05/17 11:06:55 INFO unmanagedamlauncher.UnmanagedAMLauncher: App ended with
state: FINISHED and status: SUCCEEDED
18/05/17 11:06:55 INFO unmanagedamlauncher.UnmanagedAMLauncher: Application has
completed successfully.
{color:#FF0000}real 0m14.716s{color}
{color:#FF0000}user 0m11.642s{color}
{color:#FF0000}sys 0m0.616s{color}
HDP 2.6.4
command:
time hadoop jar
/usr/hdp/2.6.4.0-91/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar
Client -classpath simple-yarn-app-1.1.0.jar -cmd "java
com.hortonworks.simpleyarnapp.ApplicationMaster /bin/date 8"
18/05/17 11:15:13 INFO unmanagedamlauncher.UnmanagedAMLauncher: Initializing
Client
18/05/17 11:15:14 INFO unmanagedamlauncher.UnmanagedAMLauncher: Starting Client
18/05/17 11:15:14 INFO client.RMProxy: Connecting to ResourceManager at
swop2.fyre.ibm.com/172.16.166.190:8050
18/05/17 11:15:14 INFO client.AHSProxy: Connecting to Application History
server at swop2.fyre.ibm.com/172.16.166.190:10200
18/05/17 11:15:14 INFO unmanagedamlauncher.UnmanagedAMLauncher: Setting up
application submission context for ASM
18/05/17 11:15:14 INFO unmanagedamlauncher.UnmanagedAMLauncher: Setting
unmanaged AM
18/05/17 11:15:14 INFO unmanagedamlauncher.UnmanagedAMLauncher: Submitting
application to ASM
18/05/17 11:15:14 INFO impl.YarnClientImpl: Submitted application
application_1526573197180_0007
18/05/17 11:15:15 INFO unmanagedamlauncher.UnmanagedAMLauncher: Got application
report from ASM for, appId=7,
appAttemptId=appattempt_1526573197180_0007_000001, clientToAMToken=null,
appDiagnostics=AM container is launched, waiting for AM container to Register
with RM, appMasterHost=N/A, appQueue=default, appMasterRpcPort=-1,
appStartTime=1526584513523, yarnAppState=ACCEPTED,
distributedFinalState=UNDEFINED, appTrackingUrl=N/A, appUser=hlhuang
18/05/17 11:15:15 INFO unmanagedamlauncher.UnmanagedAMLauncher: Launching AM
with application attempt id appattempt_1526573197180_0007_000001
18/05/17 11:15:17 INFO client.RMProxy: Connecting to ResourceManager at
swop2.fyre.ibm.com/172.16.166.190:8030
18/05/17 11:15:17 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
registerApplicationMaster 0
registerApplicationMaster 1
Making res-req 0
Making res-req 1
Making res-req 2
Making res-req 3
Making res-req 4
Making res-req 5
Making res-req 6
Making res-req 7
18/05/17 11:15:18 INFO impl.AMRMClientImpl: Received new token for :
swop4.fyre.ibm.com:45454
18/05/17 11:15:18 INFO impl.AMRMClientImpl: Received new token for :
swop2.fyre.ibm.com:45454
Launching container container_e07_1526573197180_0007_01_000001
18/05/17 11:15:18 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
swop4.fyre.ibm.com:45454
Launching container container_e07_1526573197180_0007_01_000002
18/05/17 11:15:18 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
swop2.fyre.ibm.com:45454
18/05/17 11:15:19 INFO impl.AMRMClientImpl: Received new token for :
swop3.fyre.ibm.com:45454
Launching container container_e07_1526573197180_0007_01_000003
18/05/17 11:15:19 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
swop3.fyre.ibm.com:45454
18/05/17 11:15:19 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
swop4.fyre.ibm.com:45454
Launching container container_e07_1526573197180_0007_01_000004
Launching container container_e07_1526573197180_0007_01_000005
18/05/17 11:15:19 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
swop2.fyre.ibm.com:45454
Completed container container_e07_1526573197180_0007_01_000002
Completed container container_e07_1526573197180_0007_01_000001
Launching container container_e07_1526573197180_0007_01_000006
18/05/17 11:15:19 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
swop3.fyre.ibm.com:45454
Launching container container_e07_1526573197180_0007_01_000007
18/05/17 11:15:19 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
swop4.fyre.ibm.com:45454
Completed container container_e07_1526573197180_0007_01_000005
Completed container container_e07_1526573197180_0007_01_000004
Completed container container_e07_1526573197180_0007_01_000003
Launching container container_e07_1526573197180_0007_01_000008
18/05/17 11:15:19 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
swop2.fyre.ibm.com:45454
Completed container container_e07_1526573197180_0007_01_000006
Completed container container_e07_1526573197180_0007_01_000008
Completed container container_e07_1526573197180_0007_01_000007
18/05/17 11:15:19 INFO unmanagedamlauncher.UnmanagedAMLauncher: AM process
exited with value: 0
18/05/17 11:15:20 INFO unmanagedamlauncher.UnmanagedAMLauncher: Got application
report from ASM for, appId=7,
appAttemptId=appattempt_1526573197180_0007_000001, clientToAMToken=null,
appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0,
appStartTime=1526584513523, yarnAppState=FINISHED,
distributedFinalState=SUCCEEDED, appTrackingUrl=N/A, appUser=hlhuang
18/05/17 11:15:20 INFO unmanagedamlauncher.UnmanagedAMLauncher: App ended with
state: FINISHED and status: SUCCEEDED
18/05/17 11:15:20 INFO unmanagedamlauncher.UnmanagedAMLauncher: Application has
completed successfully.
{color:#14892c}real 0m7.897s{color}
{color:#14892c}user 0m8.335s{color}
{color:#14892c}sys 0m0.593s{color}
The difference is {color:#d04437}14 seconds{color} in HDP 3.0.0 vs
{color:#14892c}7 seconds{color} in HDP 2.6.4 environment. Twice amount of the
time for HDP 3.0.0.
If I just acquire one container for this application, then the difference is
{color:#d04437}0m9.696s{color} (HDP 3.0.0) vs {color:#14892c}0m6.832s{color}
(HDP 2.6.4)
Has anyone noticed this performance problem and working on it?
Thanks for any help!!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
