[jira] [Commented] (KYLIN-5111) Record the time spent for each stage of query in kylin4's log

Tue, 19 Apr 2022 18:06:05 -0700 


    [ 
https://issues.apache.org/jira/browse/KYLIN-5111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17524648#comment-17524648
 ] 

ASF GitHub Bot commented on KYLIN-5111:
---------------------------------------

hit-lacus commented on PR #1859:
URL: https://github.com/apache/kylin/pull/1859#issuecomment-1103319832

   ## Background
   #### Stages of Kylin query
   
![image](https://user-images.githubusercontent.com/14030549/164125810-909b0d74-b6c4-40d5-b238-fd5e7e286b46.png)
   
   #### Detail of Spark related Job
   
![image](https://user-images.githubusercontent.com/14030549/164125989-8c41188d-ebd5-4f3d-bcb1-49b78af7ea94.png)
   
   
   ## What I do
   - Add log to show more precise metrics and statistics of stages. 
   - Refine the calculation of `WAIT_FOR_EXECUTION` and `EXECUTION`
   
   
   #### Metrics and statistics of stages
   ```java
   2022-04-18 23:42:38,687 DEBUG [Query 
ddd81385-3a65-b3df-fa95-c92e6d7ad60a-45] query.util.SparkJobTraceV2:58 
       Query job finished.
   2022-04-18 23:42:38,778 INFO  [Query 
ddd81385-3a65-b3df-fa95-c92e6d7ad60a-45] query.util.SparkJobTraceV2:54 
       Job 1 is submitted at 23:42:38.610 and completed at 23:42:38.652.It has 
105 tasks, succeed 1 tasks, 0 failed tasks,0 killed tasks. Killed tasks info: 
EMPTY. Stages Stage:9,0, launch time of first and last task are 23:42:38.628 
and 23:42:38.628,  bytesRead is [0,0,0,0,0], runningTime is [6, 6, 6, 6, 
6];Stage:5,0;Stage:6,0;Stage:7,0;Stage:8,0.
   2022-04-18 23:42:38,869 INFO  [Query 
ddd81385-3a65-b3df-fa95-c92e6d7ad60a-45] query.util.SparkJobTraceV2:54 
       Job 0 is submitted at 23:42:33.056 and completed at 23:42:38.571.It has 
105 tasks, succeed 105 tasks, 0 failed tasks,0 killed tasks. Killed tasks info: 
EMPTY. Stages Stage:0,0, launch time of first and last task are 23:42:33.145 
and 23:42:37.141,  bytesRead is [25245,27917,29155,30857,33575], runningTime is 
[12, 19, 25, 43, 2125];Stage:1,0, launch time of first and last task are 
23:42:37.285 and 23:42:37.285,  bytesRead is [0,0,0,0,0], runningTime is [254, 
254, 254, 254, 254];Stage:2,0, launch time of first and last task are 
23:42:37.166 and 23:42:38.386,  bytesRead is [25245,27917,29155,30857,33575], 
runningTime is [8, 9, 10, 14, 112];Stage:3,0, launch time of first and last 
task are 23:42:38.442 and 23:42:38.442,  bytesRead is [0,0,0,0,0], runningTime 
is [85, 85, 85, 85, 85];Stage:4,0, launch time of first and last task are 
23:42:38.554 and 23:42:38.554,  bytesRead is [0,0,0,0,0], runningTime is [3, 3, 
3, 3, 3].
   2022-04-18 23:42:38,873 INFO  [Query 
ddd81385-3a65-b3df-fa95-c92e6d7ad60a-45] query.util.SparkJobTraceV2:54 
       jobExecutionTime will change from 10954 to 5596 .
   2022-04-18 23:42:38,877 INFO  [Query 
ddd81385-3a65-b3df-fa95-c92e6d7ad60a-45] query.util.SparkJobTraceV2:54 
       Kylin submit query at 23:42:27.733, first spark job submitted at 
23:42:33.056, last spark job finished at 23:42:38.652, query finished at 
23:42:38.687 .
   ```
   
   
   




> Record the time spent for each stage of query in kylin4's log
> -------------------------------------------------------------
>
>                 Key: KYLIN-5111
>                 URL: https://issues.apache.org/jira/browse/KYLIN-5111
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Metrics
>    Affects Versions: v4.0.0
>            Reporter: Yaqian Zhang
>            Assignee: Yaqian Zhang
>            Priority: Minor
>             Fix For: v4.0.1
>
>




--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Reply via email to