[GitHub] [kylin] hit-lacus commented on pull request #1485: KYLIN-4818 Support Cube Planner Phase One in Kylin 4

[GitBox]

hit-lacus commented on pull request #1485:
URL: https://github.com/apache/kylin/pull/1485#issuecomment-740020477


   ## CuboidStatisticsJob Profile
   
   
   #### Tools
   Refer to 
https://www.linkedin.com/pulse/profiling-spark-applications-one-click-michael-spector
 .
   
   
   #### Prepare env
   
   - Hadoop env : HDP 2.4
   - Cube : KylinSales
   
   #### Kylin Conf
   
   ```properties
   
kylin.metadata.url=KYLIN_4818_1@jdbc,url=jdbc:mysql://10.1.3.90:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
   kylin.env.zookeeper-connect-string=cdh-master:2181
   kylin.env.zookeeper-base-path=/kylin/regression_testing/KYLIN-4818-1
   kylin.env.hdfs-working-dir=/kylin/regression_testing/KYLIN-4818-1
   kylin.source.hive.database-for-flat-table=regression_testing
   kylin.query.cache-enabled=false
   kylin.job.scheduler.default=100
   kylin.server.self-discovery-enabled=true
   kylin.spark-conf.auto.prior=false
   
   #kylin.cube.cubeplanner.enabled=false
   
   kylin.engine.spark-conf.spark.executor.memory=6g
   kylin.engine.spark-conf.spark.executor.memoryOverhead=1g
   kylin.engine.spark-conf.spark.executor.instances=1
   kylin.engine.spark-conf.spark.executor.cores=1
   kylin.engine.spark-cmd=/usr/local/bin/spark-submit-flamegraph
   kylin.cube.cubeplanner.enabled=true
   ```
   
   
   #### Task Statistics Tab of Spark UI
   
   <img width="1417" alt="image" 
src="https://user-images.githubusercontent.com/14030549/101375345-3ce3d900-38ea-11eb-9a47-1f6bd16963ce.png";>
   
   #### Executor Log
   ```sh
   LogType:stdout
   Log Upload Time:Mon Dec 07 16:00:58 +0000 2020
   LogLength:2920
   Log Contents:
   log4j: Trying to find [spark-executor-log4j.properties] using context 
classloader sun.misc.Launcher$AppClassLoader@18b4aac2.
   log4j: Using URL 
[file:/hadoop/yarn/local/usercache/root/appcache/application_1606276600681_1970/container_e09_1606276600681_1970_01_000002/spark-executor-log4j.properties]
 for automatic log4j configuration.
   log4j: Reading configuration from URL 
file:/hadoop/yarn/local/usercache/root/appcache/application_1606276600681_1970/container_e09_1606276600681_1970_01_000002/spark-executor-log4j.properties
   log4j: Parsing for [root] with value=[INFO,stderr].
   log4j: Level token is [INFO].
   log4j: Category root set to INFO
   log4j: Parsing appender named "stderr".
   log4j: Parsing layout options for "stderr".
   log4j: Setting property [conversionPattern] to [%d{ISO8601} %-5p [%t] %c{2} 
: %m%n].
   log4j: End of parsing for "stderr".
   log4j: Setting property [target] to [System.err].
   log4j: Parsed "stderr" options.
   log4j: Finished configuring.
   CuboidStatisticsJob-Init1-1607355948764
   CuboidStatisticsJob-Init2-1607355948998
   CuboidStatisticsJob-statisticsWithinPartition1-1607355949009
   [10002313,10000349,0,2012-12-14,88750,Consumer Electronics,Vehicle 
Electronics & GPS,Radar & Laser Detectors,1,2,FR,US,France,United 
States,Others,0,ANALYST,Beijing]
   [10004376,10000927,1,2012-08-28,175750,Home & Garden,Bedding,Blankets & 
Throws,0,5,IT,FR,Italy,France,Others,0,ANALYST,Beijing]
   [10006710,10000005,2,2012-02-16,148324,Phones,Mobile 
Accessories,CaseCoverSkins,0,1,JP,CN,Japan,China,ABIN,15,ADMIN,Shanghai]
   [10003717,10000209,3,2013-10-19,37831,Collectibles,Advertising,Merchandise & 
Memorabilia,4,3,GB,FR,United Kingdom,France,FP-non GTC,0,ANALYST,Beijing]
   [10006076,10000154,4,2012-10-22,140746,eBay Motors,Parts & 
Accessories,Vintage Car & Truck 
Parts,0,4,JP,FR,Japan,France,Others,100,ADMIN,Shanghai]
       Stats
      i   :5001
   meter1 :159
   meter2 :279412
   CuboidStatisticsJob-statisticsWithinPartition2-1607356229905
   CuboidStatisticsJob-Init1-1607356230853
   CuboidStatisticsJob-Init2-1607356231101
   CuboidStatisticsJob-statisticsWithinPartition1-1607356231101
   [10009393,10000949,5009,2012-09-06,51582,ClothinShoes & Accessories,Kids' 
ClothinShoes & Accs,Girls' Clothing (Sizes 4 & Up),2,4,US,DE,United 
States,Germany,FP-GTC,0,ADMIN,Shanghai]
   [10002759,10000199,5010,2012-01-18,20865,ClothinShoes & Accessories,Men's 
Clothing,Athletic Apparel,3,3,CN,FR,China,France,FP-GTC,0,ADMIN,Shanghai]
   [10004825,10000098,5011,2013-04-25,20485,Home & 
Garden,Furniture,Other,2,3,JP,JP,Japan,Japan,ABIN,0,ADMIN,Shanghai]
   [10005962,10000244,5012,2013-12-01,145970,Toys & Hobbies,Models & 
Kits,Automotive,5,4,JP,DE,Japan,Germany,FP-non GTC,0,ANALYST,Beijing]
   [10004074,10000541,5013,2013-09-04,24541,Sports MeCards & Fan Shop,Fan 
Apparel & Souvenirs,College-NCAA,2,2,FR,US,France,United 
States,Auction,0,ADMIN,Shanghai]
       Stats
      i   :4987
   meter1 :93
   meter2 :292977
   CuboidStatisticsJob-statisticsWithinPartition2-1607356524809
   End of LogType:stdout
   ```
   
   
   ### Flame chart 
   
   <img width="1196" alt="image" 
src="https://user-images.githubusercontent.com/14030549/101375552-7f0d1a80-38ea-11eb-9c4b-29c04531899a.png";>
   
   <img width="1188" alt="image" 
src="https://user-images.githubusercontent.com/14030549/101375669-a368f700-38ea-11eb-9f4d-ae6b5f57fece.png";>
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org