[ 
https://issues.apache.org/jira/browse/YARN-8315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hsin-Liang Huang updated YARN-8315:
-----------------------------------
    Description: I can't find the button to delete this, so I just removed the 
text to avoid sensitive information in the previous posting.   (was: Hi,   I am 
comparing the performance between HDP 3.0.0 and HDP 2.6.4 and I discovered HDP 
3.0.0 is much slower than HDP 2.6.4 if the job acquire more yarn containers and 
we also pin point the problem is after the job is done,  when it tried to clean 
up all the containers to exit the application, that's the place where it 
consumed more time than HDP 2.6.4.   I used the simple yarn app that 
Hortonworks put out on github [https://github.com/hortonworks/simple-yarn-app] 
to do the testing.  Below is my testing result from acquiring 8 containers in 
both HDP 3.0.0 and HDP 2.6.4 cluster environment. 

=============================================================

HDP 3.0.0: 

command:  time hadoop jar 
/usr/hdp/3.0.0.0-829/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.0.0.3.0.0.0-829.jar
 Client -classpath simple-yarn-app-1.1.0.jar -cmd "java 
com.hortonworks.simpleyarnapp.ApplicationMaster /bin/date 8"

18/05/17 11:06:42 INFO unmanagedamlauncher.UnmanagedAMLauncher: Initializing 
Client
18/05/17 11:06:42 INFO unmanagedamlauncher.UnmanagedAMLauncher: Starting Client
18/05/17 11:06:43 INFO client.RMProxy: Connecting to ResourceManager at 
whiny1.fyre.ibm.com/172.16.165.211:8050
18/05/17 11:06:43 INFO client.AHSProxy: Connecting to Application History 
server at whiny2.fyre.ibm.com/172.16.200.160:10200
18/05/17 11:06:43 INFO unmanagedamlauncher.UnmanagedAMLauncher: Setting up 
application submission context for ASM
18/05/17 11:06:43 INFO unmanagedamlauncher.UnmanagedAMLauncher: Setting 
unmanaged AM
18/05/17 11:06:43 INFO unmanagedamlauncher.UnmanagedAMLauncher: Submitting 
application to ASM
18/05/17 11:06:43 INFO impl.YarnClientImpl: Submitted application 
application_1526572577866_0011
18/05/17 11:06:44 INFO unmanagedamlauncher.UnmanagedAMLauncher: Got application 
report from ASM for, appId=11, 
appAttemptId=appattempt_1526572577866_0011_000001, clientToAMToken=null, 
appDiagnostics=AM container is launched, waiting for AM container to Register 
with RM, appMasterHost=N/A, appQueue=default, appMasterRpcPort=-1, 
appStartTime=1526584003704, yarnAppState=ACCEPTED, 
distributedFinalState=UNDEFINED, appTrackingUrl=N/A, appUser=hlhuang
18/05/17 11:06:44 INFO unmanagedamlauncher.UnmanagedAMLauncher: Launching AM 
with application attempt id appattempt_1526572577866_0011_000001
18/05/17 11:06:46 INFO client.RMProxy: Connecting to ResourceManager at 
whiny1.fyre.ibm.com/172.16.165.211:8030
registerApplicationMaster 0
registerApplicationMaster 1
18/05/17 11:06:47 INFO conf.Configuration: found resource resource-types.xml at 
file:/etc/hadoop/3.0.0.0-829/0/resource-types.xml
Making res-req 0
Making res-req 1
Making res-req 2
Making res-req 3
Making res-req 4
Making res-req 5
Making res-req 6
Making res-req 7
Launching container container_e08_1526572577866_0011_01_000001
Launching container container_e08_1526572577866_0011_01_000002
Launching container container_e08_1526572577866_0011_01_000003
Launching container container_e08_1526572577866_0011_01_000004
Launching container container_e08_1526572577866_0011_01_000005
Launching container container_e08_1526572577866_0011_01_000006
Launching container container_e08_1526572577866_0011_01_000007
Launching container container_e08_1526572577866_0011_01_000008
Completed container container_e08_1526572577866_0011_01_000001
Completed container container_e08_1526572577866_0011_01_000002
Completed container container_e08_1526572577866_0011_01_000003
Completed container container_e08_1526572577866_0011_01_000004
Completed container container_e08_1526572577866_0011_01_000008
Completed container container_e08_1526572577866_0011_01_000005
Completed container container_e08_1526572577866_0011_01_000006
Completed container container_e08_1526572577866_0011_01_000007
18/05/17 11:06:54 INFO unmanagedamlauncher.UnmanagedAMLauncher: AM process 
exited with value: 0
18/05/17 11:06:55 INFO unmanagedamlauncher.UnmanagedAMLauncher: Got application 
report from ASM for, appId=11, 
appAttemptId=appattempt_1526572577866_0011_000001, clientToAMToken=null, 
appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0, 
appStartTime=1526584003704, yarnAppState=FINISHED, 
distributedFinalState=SUCCEEDED, appTrackingUrl=N/A, appUser=hlhuang
18/05/17 11:06:55 INFO unmanagedamlauncher.UnmanagedAMLauncher: App ended with 
state: FINISHED and status: SUCCEEDED
18/05/17 11:06:55 INFO unmanagedamlauncher.UnmanagedAMLauncher: Application has 
completed successfully.

{color:#FF0000}real 0m14.716s{color}
{color:#FF0000}user 0m11.642s{color}
{color:#FF0000}sys 0m0.616s{color}

 

 

HDP 2.6.4 

command: 

time hadoop jar 
/usr/hdp/2.6.4.0-91/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar
 Client -classpath simple-yarn-app-1.1.0.jar -cmd "java 
com.hortonworks.simpleyarnapp.ApplicationMaster /bin/date 8"

18/05/17 11:15:13 INFO unmanagedamlauncher.UnmanagedAMLauncher: Initializing 
Client
18/05/17 11:15:14 INFO unmanagedamlauncher.UnmanagedAMLauncher: Starting Client
18/05/17 11:15:14 INFO client.RMProxy: Connecting to ResourceManager at 
swop2.fyre.ibm.com/172.16.166.190:8050
18/05/17 11:15:14 INFO client.AHSProxy: Connecting to Application History 
server at swop2.fyre.ibm.com/172.16.166.190:10200
18/05/17 11:15:14 INFO unmanagedamlauncher.UnmanagedAMLauncher: Setting up 
application submission context for ASM
18/05/17 11:15:14 INFO unmanagedamlauncher.UnmanagedAMLauncher: Setting 
unmanaged AM
18/05/17 11:15:14 INFO unmanagedamlauncher.UnmanagedAMLauncher: Submitting 
application to ASM
18/05/17 11:15:14 INFO impl.YarnClientImpl: Submitted application 
application_1526573197180_0007
18/05/17 11:15:15 INFO unmanagedamlauncher.UnmanagedAMLauncher: Got application 
report from ASM for, appId=7, 
appAttemptId=appattempt_1526573197180_0007_000001, clientToAMToken=null, 
appDiagnostics=AM container is launched, waiting for AM container to Register 
with RM, appMasterHost=N/A, appQueue=default, appMasterRpcPort=-1, 
appStartTime=1526584513523, yarnAppState=ACCEPTED, 
distributedFinalState=UNDEFINED, appTrackingUrl=N/A, appUser=hlhuang
18/05/17 11:15:15 INFO unmanagedamlauncher.UnmanagedAMLauncher: Launching AM 
with application attempt id appattempt_1526573197180_0007_000001
18/05/17 11:15:17 INFO client.RMProxy: Connecting to ResourceManager at 
swop2.fyre.ibm.com/172.16.166.190:8030
18/05/17 11:15:17 INFO impl.ContainerManagementProtocolProxy: 
yarn.client.max-cached-nodemanagers-proxies : 0
registerApplicationMaster 0
registerApplicationMaster 1
Making res-req 0
Making res-req 1
Making res-req 2
Making res-req 3
Making res-req 4
Making res-req 5
Making res-req 6
Making res-req 7
18/05/17 11:15:18 INFO impl.AMRMClientImpl: Received new token for : 
swop4.fyre.ibm.com:45454
18/05/17 11:15:18 INFO impl.AMRMClientImpl: Received new token for : 
swop2.fyre.ibm.com:45454
Launching container container_e07_1526573197180_0007_01_000001
18/05/17 11:15:18 INFO impl.ContainerManagementProtocolProxy: Opening proxy : 
swop4.fyre.ibm.com:45454
Launching container container_e07_1526573197180_0007_01_000002
18/05/17 11:15:18 INFO impl.ContainerManagementProtocolProxy: Opening proxy : 
swop2.fyre.ibm.com:45454
18/05/17 11:15:19 INFO impl.AMRMClientImpl: Received new token for : 
swop3.fyre.ibm.com:45454
Launching container container_e07_1526573197180_0007_01_000003
18/05/17 11:15:19 INFO impl.ContainerManagementProtocolProxy: Opening proxy : 
swop3.fyre.ibm.com:45454
18/05/17 11:15:19 INFO impl.ContainerManagementProtocolProxy: Opening proxy : 
swop4.fyre.ibm.com:45454
Launching container container_e07_1526573197180_0007_01_000004
Launching container container_e07_1526573197180_0007_01_000005
18/05/17 11:15:19 INFO impl.ContainerManagementProtocolProxy: Opening proxy : 
swop2.fyre.ibm.com:45454
Completed container container_e07_1526573197180_0007_01_000002
Completed container container_e07_1526573197180_0007_01_000001
Launching container container_e07_1526573197180_0007_01_000006
18/05/17 11:15:19 INFO impl.ContainerManagementProtocolProxy: Opening proxy : 
swop3.fyre.ibm.com:45454
Launching container container_e07_1526573197180_0007_01_000007
18/05/17 11:15:19 INFO impl.ContainerManagementProtocolProxy: Opening proxy : 
swop4.fyre.ibm.com:45454
Completed container container_e07_1526573197180_0007_01_000005
Completed container container_e07_1526573197180_0007_01_000004
Completed container container_e07_1526573197180_0007_01_000003
Launching container container_e07_1526573197180_0007_01_000008
18/05/17 11:15:19 INFO impl.ContainerManagementProtocolProxy: Opening proxy : 
swop2.fyre.ibm.com:45454
Completed container container_e07_1526573197180_0007_01_000006
Completed container container_e07_1526573197180_0007_01_000008
Completed container container_e07_1526573197180_0007_01_000007
18/05/17 11:15:19 INFO unmanagedamlauncher.UnmanagedAMLauncher: AM process 
exited with value: 0
18/05/17 11:15:20 INFO unmanagedamlauncher.UnmanagedAMLauncher: Got application 
report from ASM for, appId=7, 
appAttemptId=appattempt_1526573197180_0007_000001, clientToAMToken=null, 
appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0, 
appStartTime=1526584513523, yarnAppState=FINISHED, 
distributedFinalState=SUCCEEDED, appTrackingUrl=N/A, appUser=hlhuang
18/05/17 11:15:20 INFO unmanagedamlauncher.UnmanagedAMLauncher: App ended with 
state: FINISHED and status: SUCCEEDED
18/05/17 11:15:20 INFO unmanagedamlauncher.UnmanagedAMLauncher: Application has 
completed successfully.

{color:#14892c}real 0m7.897s{color}
{color:#14892c}user 0m8.335s{color}
{color:#14892c}sys 0m0.593s{color}

 

The difference is {color:#d04437}14 seconds{color} in HDP 3.0.0 vs 
{color:#14892c}7 seconds{color} in HDP 2.6.4 environment.  Twice amount of the 
time for HDP 3.0.0. 

If I just acquire one container for this application, then the difference is  
{color:#d04437}0m9.696s{color}  (HDP 3.0.0) vs {color:#14892c}0m6.832s{color} 
(HDP 2.6.4)

  Has anyone noticed this performance problem and working on it? 

Thanks for any help!!

 

  

 

 

 

   )

> HDP 3.0.0 perfromance is slower than HDP 2.6.4
> ----------------------------------------------
>
>                 Key: YARN-8315
>                 URL: https://issues.apache.org/jira/browse/YARN-8315
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: yarn
>    Affects Versions: 3.0.0
>         Environment: I have a HDP 2.6.4 cluster and HDP 3.0.0 cluster,  I set 
> up to have the same settings for these two cluster such as java heap size, 
> container size etc.  They are both 4 node cluster with 3 data nodes.   I took 
> almost all the default setting on HDP 3.0.0 except that I modify the minimum 
> container size to 64MB instead of 1024MB in both cluster.  
>  
>            Reporter: Hsin-Liang Huang
>            Priority: Major
>
> I can't find the button to delete this, so I just removed the text to avoid 
> sensitive information in the previous posting. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to