[
https://issues.apache.org/jira/browse/KYLIN-4818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243649#comment-17243649
]
ASF GitHub Bot commented on KYLIN-4818:
---------------------------------------
hit-lacus commented on pull request #1485:
URL: https://github.com/apache/kylin/pull/1485#issuecomment-738516066
### Performance result 2
```
2020-12-03 23:12:38,132 INFO [pool-1-thread-1] job.ParentSourceChooser :
Sampling start ...
2020-12-03 23:12:39,765 INFO [pool-1-thread-1] mapred.FileInputFormat :
Total input paths to process : 24
2020-12-03 23:12:39,781 INFO [pool-1-thread-1] mapred.FileInputFormat :
Total input paths to process : 24
2020-12-03 23:57:13,523 INFO [pool-1-thread-1] job.ParentSourceChooser :
Sampling finished and cost 2675 s .
2020-12-03 23:57:13,633 INFO [pool-1-thread-1] job.CubeBuildJob : Cuboid
statistics return 5325 records and cost 2680027 ms.
...
2020-12-03 23:57:25,203 INFO [pool-1-thread-1] algorithm.CuboidRecommender
: Cube Planner Algorithm started at 1607011045203
2020-12-03 23:57:25,203 INFO [pool-1-thread-1] greedy.GreedyAlgorithm :
Greedy Algorithm started.
2020-12-03 23:57:35,163 INFO [pool-1-thread-1] greedy.GreedyAlgorithm :
Greedy Algorithm finished.
2020-12-03 23:57:35,163 INFO [pool-1-thread-1] algorithm.CuboidRecommender
: Cube Planner Algorithm ended at 9960
2020-12-03 23:57:35,163 INFO [pool-1-thread-1] algorithm.CuboidRecommender
: Cube Planner Algorithm chooses 103 most effective cuboids to build among of
all 5325 cuboids.
2020-12-03 23:57:35,186 INFO [pool-1-thread-1] cube.CubeManager : Updating
cube instance 'UserActionStreamCube_NO_RULE'
...
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3293249, "rows": 93121, "size": 1650320 "deviation": 0.065184}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3186753, "rows": 57565, "size": 1077938 "deviation": 0.078294}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1224705, "rows": 66223, "size": 1202705 "deviation": 0.068149}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3506177, "rows": 136516, "size": 2519256 "deviation": 0.370440}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3174401, "rows": 68147, "size": 1280403 "deviation": 0.082058}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3343361, "rows": 68190, "size": 1386303 "deviation": -0.021015}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1155137, "rows": 55710, "size": 1031150 "deviation": 0.069000}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3694593, "rows": 319909, "size": 6111691 "deviation": 0.008349}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1204289, "rows": 30062, "size": 608350 "deviation": 0.066496}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3248129, "rows": 30062, "size": 610204 "deviation": 0.050595}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3313665, "rows": 30062, "size": 609296 "deviation": 0.063535}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1159169, "rows": 66223, "size": 1200973 "deviation": 0.072845}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3604481, "rows": 136516, "size": 2554878 "deviation": 0.370865}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3194945, "rows": 57565, "size": 1077873 "deviation": 0.078294}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3375105, "rows": 1918, "size": 51344 "deviation": -0.008863}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3317761, "rows": 8702, "size": 193147 "deviation": -0.005401}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1278977, "rows": 46023, "size": 1002753 "deviation": -0.014993}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1245185, "rows": 646, "size": 19856 "deviation": -0.012384}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1052673, "rows": 3430, "size": 82046 "deviation": 0.080466}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1466369, "rows": 148673, "size": 2716391 "deviation": 0.339228}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3297281, "rows": 105286, "size": 1851573 "deviation": 0.061993}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3203073, "rows": 4622, "size": 108074 "deviation": -0.011900}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1523713, "rows": 148673, "size": 2717809 "deviation": 0.342779}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1253441, "rows": 30062, "size": 608092 "deviation": 0.066496}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3309633, "rows": 24630, "size": 511784 "deviation": 0.055217}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1261633, "rows": 30062, "size": 608148 "deviation": 0.066496}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1232897, "rows": 66223, "size": 1200475 "deviation": 0.068149}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1294337, "rows": 4462, "size": 102077 "deviation": -0.010533}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1208321, "rows": 36628, "size": 727700 "deviation": 0.059900}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1282049, "rows": 16760, "size": 375787 "deviation": 0.061158}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3289089, "rows": 105286, "size": 1852023 "deviation": 0.061993}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1163329, "rows": 55710, "size": 1031749 "deviation": 0.069000}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3260417, "rows": 8702, "size": 193349 "deviation": -0.018616}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1205249, "rows": 114795, "size": 2112674 "deviation": -0.002727}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1802241, "rows": 290711, "size": 5481202 "deviation": 0.021086}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1139713, "rows": 114795, "size": 2114195 "deviation": -0.005314}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1540097, "rows": 95485, "size": 1870148 "deviation": 0.456030}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3325953, "rows": 8702, "size": 193217 "deviation": -0.005401}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1246273, "rows": 275526, "size": 5367294 "deviation": 0.034599}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1228865, "rows": 55710, "size": 1039541 "deviation": 0.064136}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1220673, "rows": 55710, "size": 1034202 "deviation": 0.064136}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1262593, "rows": 114795, "size": 2123248 "deviation": -0.005314}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1221633, "rows": 114795, "size": 2099417 "deviation": -0.013163}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1257473, "rows": 36628, "size": 727539 "deviation": 0.059900}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1310721, "rows": 20600, "size": 433474 "deviation": 0.488835}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1511425, "rows": 298150, "size": 5688387 "deviation": 0.219453}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1106945, "rows": 114795, "size": 2101989 "deviation": -0.015497}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3235841, "rows": 8702, "size": 196390 "deviation": -0.033096}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3170369, "rows": 57565, "size": 1098337 "deviation": 0.081369}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1400833, "rows": 148673, "size": 2717367 "deviation": 0.342779}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3311617, "rows": 170608, "size": 3296217 "deviation": -0.003411}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1156097, "rows": 114795, "size": 2098034 "deviation": -0.013171}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1142785, "rows": 36628, "size": 727640 "deviation": 0.056760}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3252225, "rows": 8702, "size": 193145 "deviation": -0.018616}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1151041, "rows": 166007, "size": 3248496 "deviation": 0.364310}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1278017, "rows": 13730, "size": 299557 "deviation": 0.053969}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1105985, "rows": 31029, "size": 626727 "deviation": 0.053208}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1048641, "rows": 2824, "size": 69367 "deviation": 0.071884}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3219521, "rows": 93121, "size": 1655453 "deviation": 0.062940}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1515521, "rows": 148673, "size": 2717723 "deviation": 0.342779}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":8388607, "rows": 536004, "size": 25528217 "deviation": -0.028395}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1216577, "rows": 166007, "size": 3238469 "deviation": 0.366545}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3221505, "rows": 249662, "size": 4578151 "deviation": 0.017472}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1286145, "rows": 4462, "size": 102076 "deviation": -0.010533}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3358721, "rows": 8702, "size": 196544 "deviation": -0.018616}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3432449, "rows": 181908, "size": 3310633 "deviation": 0.242221}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1280001, "rows": 131683, "size": 2688150 "deviation": -0.008414}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1107969, "rows": 189338, "size": 3573465 "deviation": -0.012929}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3190785, "rows": 68147, "size": 1255838 "deviation": 0.080972}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3310593, "rows": 68190, "size": 1369992 "deviation": -0.021015}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3182657, "rows": 170060, "size": 3319254 "deviation": 0.373615}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3198977, "rows": 68147, "size": 1256268 "deviation": 0.080972}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3171329, "rows": 150525, "size": 2749858 "deviation": 0.007341}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1265665, "rows": 36628, "size": 727614 "deviation": 0.059900}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1152001, "rows": 290391, "size": 5601111 "deviation": 0.031981}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1171457, "rows": 4462, "size": 102106 "deviation": -0.016809}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3245057, "rows": 68190, "size": 1369785 "deviation": -0.021015}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3231745, "rows": 105286, "size": 1851088 "deviation": 0.051887}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3342401, "rows": 24630, "size": 519769 "deviation": 0.043524}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1269761, "rows": 2342, "size": 59462 "deviation": -0.016652}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1249345, "rows": 119489, "size": 2509462 "deviation": 0.405837}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1110017, "rows": 37629, "size": 745649 "deviation": 0.057562}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1839169, "rows": 475663, "size": 9233783 "deviation": 0.237715}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3223553, "rows": 105286, "size": 1853052 "deviation": 0.051887}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1164289, "rows": 114795, "size": 2099943 "deviation": -0.013171}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1368065, "rows": 149954, "size": 2726092 "deviation": 0.335430}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1167361, "rows": 66223, "size": 1200670 "deviation": 0.072845}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3346433, "rows": 30062, "size": 618222 "deviation": 0.050595}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1236993, "rows": 4462, "size": 102118 "deviation": -0.021515}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3571713, "rows": 136516, "size": 2521782 "deviation": 0.376198}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3146145, "rows": 410062, "size": 9445381 "deviation": -0.007448}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1138753, "rows": 30062, "size": 608401 "deviation": 0.045639}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3350529, "rows": 8702, "size": 196570 "deviation": -0.018616}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3301377, "rows": 8702, "size": 196525 "deviation": -0.018272}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1229825, "rows": 114795, "size": 2098774 "deviation": -0.013163}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3285057, "rows": 93121, "size": 1650566 "deviation": 0.065184}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1507393, "rows": 285490, "size": 5472526 "deviation": 0.240698}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":4194303, "rows": 536004, "size": 23463241 "deviation": -0.024839}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1206273, "rows": 218665, "size": 4050955 "deviation": -0.000480}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1077313, "rows": 185124, "size": 3529116 "deviation": 0.351375}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3227713, "rows": 93121, "size": 1654870 "deviation": 0.062940}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":3244097, "rows": 24630, "size": 511292 "deviation": 0.043524}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Statistics
: {"cuboid":1254401, "rows": 114795, "size": 2113127 "deviation": -0.005314}
2020-12-04 00:07:03,703 INFO [Thread-1] job.CubeBuildJob : Saving
hdfs://cdh-master:8020/regression_testing/KYLIN-4818/KYLIN_4818//cube_statistics/UserActionStreamCube_NO_RULE/04712822-6067-7e3a-eb97-cf1bd2bd2317.json
04712822-6067-7e3a-eb97-cf1bd2bd2317_20201011000000_20201013000000_2UC
```
### Duartion Summary
Cuboid need to statistics : 5325 .
Cuboid remain : 103 .
Sampling cost : 2675 s .
Total duration (Cubodi statistics, build dictionary, build snapshots, build
cuboid.) cost : 51mins, 31sec.
Sampling duration proportion : **86.89** %
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Calculate cuboid statistics in Kylin 4
> --------------------------------------
>
> Key: KYLIN-4818
> URL: https://issues.apache.org/jira/browse/KYLIN-4818
> Project: Kylin
> Issue Type: Sub-task
> Components: Spark Engine
> Reporter: Xiaoxiang Yu
> Assignee: Xiaoxiang Yu
> Priority: Major
> Fix For: v4.0.0-beta
>
>
> Refer to SparkFactDistinct.java in Kylin 3, I will try to use spark to
> calculate(estimate) rowcount/size for cuboid candidate. Rowcount/size of
> cuboid si the input for cubeplanner phase one and phase two.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
