zzcclp edited a comment on pull request #1495: URL: https://github.com/apache/kylin/pull/1495#issuecomment-740415145
## Performance Testing ### Test Env - Hadoop 2.7.0 on docker. - Commit : [ed0649b1](https://github.com/apache/kylin/pull/1495/commits/ed0649b140529bdfafea8cce846962b6ca9c3f73) - Sparder Env : spark.executor.cores=1 spark.executor.instances=6 spark.executor.memory=2G spark.executor.memoryOverhead=1G spark.sql.shuffle.partitions=6 ### Test Sends 5 SQLs at the same time ### Before this patch The shuffle partition number of all querys is 6, which equals to the total cores number, and totally spent 3.09s to finish these 5 SQLs. 5 SQLs submitted at the same time.   SQL1, there are 6 partitions at the second stage, even though there are only 216KB data to read.  SQL2, there are still 6 partitions at the second stage, even though there are only 262KB data to read.  ### After this patch The shuffle partition number of each query is calculated according to the scanned bytes of each query, and totally spent 2.55s to finish these 5 SQLs. 5 SQLs submitted at the same time.   SQL1, there is only 1 partition at the second stage, and spent the half of the time it took before this patch.  SQL2, there is only 1 partition at the second stage, and spent the half of the time it took before this patch.  ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
