hit-lacus commented on pull request #1485: URL: https://github.com/apache/kylin/pull/1485#issuecomment-738516066
### Performance result 2   ``` 2020-12-03 23:12:38,132 INFO [pool-1-thread-1] job.ParentSourceChooser : Sampling start ... 2020-12-03 23:12:39,765 INFO [pool-1-thread-1] mapred.FileInputFormat : Total input paths to process : 24 2020-12-03 23:12:39,781 INFO [pool-1-thread-1] mapred.FileInputFormat : Total input paths to process : 24 2020-12-03 23:57:13,523 INFO [pool-1-thread-1] job.ParentSourceChooser : Sampling finished and cost 2675 s . 2020-12-03 23:57:13,633 INFO [pool-1-thread-1] job.CubeBuildJob : Cuboid statistics return 5325 records and cost 2680027 ms. ... 2020-12-03 23:57:25,203 INFO [pool-1-thread-1] algorithm.CuboidRecommender : Cube Planner Algorithm started at 1607011045203 2020-12-03 23:57:25,203 INFO [pool-1-thread-1] greedy.GreedyAlgorithm : Greedy Algorithm started. 2020-12-03 23:57:35,163 INFO [pool-1-thread-1] greedy.GreedyAlgorithm : Greedy Algorithm finished. 2020-12-03 23:57:35,163 INFO [pool-1-thread-1] algorithm.CuboidRecommender : Cube Planner Algorithm ended at 9960 2020-12-03 23:57:35,163 INFO [pool-1-thread-1] algorithm.CuboidRecommender : Cube Planner Algorithm chooses 103 most effective cuboids to build among of all 5325 cuboids. 2020-12-03 23:57:35,186 INFO [pool-1-thread-1] cube.CubeManager : Updating cube instance 'UserActionStreamCube_NO_RULE' ... 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3293249, "rows": 93121, "size": 1650320 "deviation": 0.065184} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3186753, "rows": 57565, "size": 1077938 "deviation": 0.078294} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1224705, "rows": 66223, "size": 1202705 "deviation": 0.068149} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3506177, "rows": 136516, "size": 2519256 "deviation": 0.370440} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3174401, "rows": 68147, "size": 1280403 "deviation": 0.082058} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3343361, "rows": 68190, "size": 1386303 "deviation": -0.021015} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1155137, "rows": 55710, "size": 1031150 "deviation": 0.069000} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3694593, "rows": 319909, "size": 6111691 "deviation": 0.008349} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1204289, "rows": 30062, "size": 608350 "deviation": 0.066496} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3248129, "rows": 30062, "size": 610204 "deviation": 0.050595} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3313665, "rows": 30062, "size": 609296 "deviation": 0.063535} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1159169, "rows": 66223, "size": 1200973 "deviation": 0.072845} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3604481, "rows": 136516, "size": 2554878 "deviation": 0.370865} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3194945, "rows": 57565, "size": 1077873 "deviation": 0.078294} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3375105, "rows": 1918, "size": 51344 "deviation": -0.008863} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3317761, "rows": 8702, "size": 193147 "deviation": -0.005401} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1278977, "rows": 46023, "size": 1002753 "deviation": -0.014993} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1245185, "rows": 646, "size": 19856 "deviation": -0.012384} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1052673, "rows": 3430, "size": 82046 "deviation": 0.080466} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1466369, "rows": 148673, "size": 2716391 "deviation": 0.339228} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3297281, "rows": 105286, "size": 1851573 "deviation": 0.061993} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3203073, "rows": 4622, "size": 108074 "deviation": -0.011900} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1523713, "rows": 148673, "size": 2717809 "deviation": 0.342779} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1253441, "rows": 30062, "size": 608092 "deviation": 0.066496} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3309633, "rows": 24630, "size": 511784 "deviation": 0.055217} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1261633, "rows": 30062, "size": 608148 "deviation": 0.066496} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1232897, "rows": 66223, "size": 1200475 "deviation": 0.068149} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1294337, "rows": 4462, "size": 102077 "deviation": -0.010533} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1208321, "rows": 36628, "size": 727700 "deviation": 0.059900} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1282049, "rows": 16760, "size": 375787 "deviation": 0.061158} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3289089, "rows": 105286, "size": 1852023 "deviation": 0.061993} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1163329, "rows": 55710, "size": 1031749 "deviation": 0.069000} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3260417, "rows": 8702, "size": 193349 "deviation": -0.018616} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1205249, "rows": 114795, "size": 2112674 "deviation": -0.002727} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1802241, "rows": 290711, "size": 5481202 "deviation": 0.021086} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1139713, "rows": 114795, "size": 2114195 "deviation": -0.005314} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1540097, "rows": 95485, "size": 1870148 "deviation": 0.456030} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3325953, "rows": 8702, "size": 193217 "deviation": -0.005401} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1246273, "rows": 275526, "size": 5367294 "deviation": 0.034599} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1228865, "rows": 55710, "size": 1039541 "deviation": 0.064136} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1220673, "rows": 55710, "size": 1034202 "deviation": 0.064136} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1262593, "rows": 114795, "size": 2123248 "deviation": -0.005314} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1221633, "rows": 114795, "size": 2099417 "deviation": -0.013163} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1257473, "rows": 36628, "size": 727539 "deviation": 0.059900} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1310721, "rows": 20600, "size": 433474 "deviation": 0.488835} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1511425, "rows": 298150, "size": 5688387 "deviation": 0.219453} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1106945, "rows": 114795, "size": 2101989 "deviation": -0.015497} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3235841, "rows": 8702, "size": 196390 "deviation": -0.033096} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3170369, "rows": 57565, "size": 1098337 "deviation": 0.081369} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1400833, "rows": 148673, "size": 2717367 "deviation": 0.342779} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3311617, "rows": 170608, "size": 3296217 "deviation": -0.003411} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1156097, "rows": 114795, "size": 2098034 "deviation": -0.013171} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1142785, "rows": 36628, "size": 727640 "deviation": 0.056760} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3252225, "rows": 8702, "size": 193145 "deviation": -0.018616} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1151041, "rows": 166007, "size": 3248496 "deviation": 0.364310} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1278017, "rows": 13730, "size": 299557 "deviation": 0.053969} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1105985, "rows": 31029, "size": 626727 "deviation": 0.053208} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1048641, "rows": 2824, "size": 69367 "deviation": 0.071884} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3219521, "rows": 93121, "size": 1655453 "deviation": 0.062940} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1515521, "rows": 148673, "size": 2717723 "deviation": 0.342779} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":8388607, "rows": 536004, "size": 25528217 "deviation": -0.028395} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1216577, "rows": 166007, "size": 3238469 "deviation": 0.366545} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3221505, "rows": 249662, "size": 4578151 "deviation": 0.017472} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1286145, "rows": 4462, "size": 102076 "deviation": -0.010533} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3358721, "rows": 8702, "size": 196544 "deviation": -0.018616} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3432449, "rows": 181908, "size": 3310633 "deviation": 0.242221} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1280001, "rows": 131683, "size": 2688150 "deviation": -0.008414} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1107969, "rows": 189338, "size": 3573465 "deviation": -0.012929} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3190785, "rows": 68147, "size": 1255838 "deviation": 0.080972} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3310593, "rows": 68190, "size": 1369992 "deviation": -0.021015} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3182657, "rows": 170060, "size": 3319254 "deviation": 0.373615} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3198977, "rows": 68147, "size": 1256268 "deviation": 0.080972} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3171329, "rows": 150525, "size": 2749858 "deviation": 0.007341} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1265665, "rows": 36628, "size": 727614 "deviation": 0.059900} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1152001, "rows": 290391, "size": 5601111 "deviation": 0.031981} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1171457, "rows": 4462, "size": 102106 "deviation": -0.016809} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3245057, "rows": 68190, "size": 1369785 "deviation": -0.021015} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3231745, "rows": 105286, "size": 1851088 "deviation": 0.051887} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3342401, "rows": 24630, "size": 519769 "deviation": 0.043524} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1269761, "rows": 2342, "size": 59462 "deviation": -0.016652} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1249345, "rows": 119489, "size": 2509462 "deviation": 0.405837} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1110017, "rows": 37629, "size": 745649 "deviation": 0.057562} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1839169, "rows": 475663, "size": 9233783 "deviation": 0.237715} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3223553, "rows": 105286, "size": 1853052 "deviation": 0.051887} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1164289, "rows": 114795, "size": 2099943 "deviation": -0.013171} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1368065, "rows": 149954, "size": 2726092 "deviation": 0.335430} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1167361, "rows": 66223, "size": 1200670 "deviation": 0.072845} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3346433, "rows": 30062, "size": 618222 "deviation": 0.050595} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1236993, "rows": 4462, "size": 102118 "deviation": -0.021515} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3571713, "rows": 136516, "size": 2521782 "deviation": 0.376198} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3146145, "rows": 410062, "size": 9445381 "deviation": -0.007448} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1138753, "rows": 30062, "size": 608401 "deviation": 0.045639} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3350529, "rows": 8702, "size": 196570 "deviation": -0.018616} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3301377, "rows": 8702, "size": 196525 "deviation": -0.018272} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1229825, "rows": 114795, "size": 2098774 "deviation": -0.013163} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3285057, "rows": 93121, "size": 1650566 "deviation": 0.065184} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1507393, "rows": 285490, "size": 5472526 "deviation": 0.240698} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":4194303, "rows": 536004, "size": 23463241 "deviation": -0.024839} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1206273, "rows": 218665, "size": 4050955 "deviation": -0.000480} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1077313, "rows": 185124, "size": 3529116 "deviation": 0.351375} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3227713, "rows": 93121, "size": 1654870 "deviation": 0.062940} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":3244097, "rows": 24630, "size": 511292 "deviation": 0.043524} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics : {"cuboid":1254401, "rows": 114795, "size": 2113127 "deviation": -0.005314} 2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Saving hdfs://cdh-master:8020/regression_testing/KYLIN-4818/KYLIN_4818//cube_statistics/UserActionStreamCube_NO_RULE/04712822-6067-7e3a-eb97-cf1bd2bd2317.json 04712822-6067-7e3a-eb97-cf1bd2bd2317_20201011000000_20201013000000_2UC ``` ### Duartion Summary Cuboid need to statistics : 5325 . Cuboid remain : 103 . Sampling cost : 2675 s . Total duration (Cubodi statistics, build dictionary, build snapshots, build cuboid.) cost : 51mins, 31sec. Sampling duration proportion : **86.89** % ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
