[
https://issues.apache.org/jira/browse/HIVE-27878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-27878:
--------------------------------
Description:
While running a cluster query (hs2, tez am, tez+llap), we have a handy summary
attached as: [^tpcds78_10GB_query_summary_example.txt]
However, we get no info about simply fetch tasks, we can see only this below:
{code}
0: jdbc:hive2://hs2-lbodor-aws-bug.dw-dw-team> select * from customer limit 10;
INFO : Compiling
command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61):
select * from customer limit 10
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema:
Schema(fieldSchemas:[FieldSchema(name:customer.c_customer_sk, type:bigint,
comment:null), FieldSchema(name:customer.c_customer_id, type:char(16),
comment:null), FieldSchema(name:customer.c_current_cdemo_sk, type:bigint,
comment:null), FieldSchema(name:customer.c_current_hdemo_sk, type:bigint,
comment:null), FieldSchema(name:customer.c_current_addr_sk, type:bigint,
comment:null), FieldSchema(name:customer.c_first_shipto_date_sk, type:bigint,
comment:null), FieldSchema(name:customer.c_first_sales_date_sk, type:bigint,
comment:null), FieldSchema(name:customer.c_salutation, type:char(10),
comment:null), FieldSchema(name:customer.c_first_name, type:char(20),
comment:null), FieldSchema(name:customer.c_last_name, type:char(30),
comment:null), FieldSchema(name:customer.c_preferred_cust_flag, type:char(1),
comment:null), FieldSchema(name:customer.c_birth_day, type:int, comment:null),
FieldSchema(name:customer.c_birth_month, type:int, comment:null),
FieldSchema(name:customer.c_birth_year, type:int, comment:null),
FieldSchema(name:customer.c_birth_country, type:varchar(20), comment:null),
FieldSchema(name:customer.c_login, type:char(13), comment:null),
FieldSchema(name:customer.c_email_address, type:char(50), comment:null),
FieldSchema(name:customer.c_last_review_date_sk, type:bigint, comment:null)],
properties:null)
INFO : Completed compiling
command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61); Time
taken: 0.257 seconds
INFO : Executing
command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61):
select * from customer limit 10
INFO : Completed executing
command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61); Time
taken: 0.007 seconds
INFO : OK
+-------------------------+-------------------------+------------------------------+------------------------------+-----------------------------+----------------------------------+---------------------------------+------------------------+------------------------+---------------------------------+---------------------------------+-----------------------+-------------------------+------------------------+---------------------------+-------------------+----------------------------------------------------+---------------------------------+
| customer.c_customer_sk | customer.c_customer_id |
customer.c_current_cdemo_sk | customer.c_current_hdemo_sk |
customer.c_current_addr_sk | customer.c_first_shipto_date_sk |
customer.c_first_sales_date_sk | customer.c_salutation |
customer.c_first_name | customer.c_last_name |
customer.c_preferred_cust_flag | customer.c_birth_day |
customer.c_birth_month | customer.c_birth_year | customer.c_birth_country |
customer.c_login | customer.c_email_address |
customer.c_last_review_date_sk |
+-------------------------+-------------------------+------------------------------+------------------------------+-----------------------------+----------------------------------+---------------------------------+------------------------+------------------------+---------------------------------+---------------------------------+-----------------------+-------------------------+------------------------+---------------------------+-------------------+----------------------------------------------------+---------------------------------+
| 1 | AAAAAAAABAAAAAAA | 980124
| 7135 | 32946 | 2452238
| 2452208 | Mr. |
Javier | Lewis | Y
| 9 | 12 | 1936
| CHILE | NULL |
[email protected] | 2452508
|
| 2 | AAAAAAAACAAAAAAA | 819667
| 1461 | 181655 | 2452318
| 2452288 | Dr. |
Amy | Moses | Y
| 9 | 4 | 1966
| TOGO | NULL | [email protected]
| 2452318 |
| 3 | AAAAAAAADAAAAAAA | 1473522
| 6247 | 248572 | 2449130
| 2449100 | Miss |
Latisha | Hamilton | N
| 18 | 9 | 1979
| NIUE | NULL | [email protected]
| 2452313 |
| 4 | AAAAAAAAEAAAAAAA | 1703214
| 3986 | 189558 | 2450030
| 2450000 | Dr. |
Michael | White | N
| 7 | 6 | 1983
| MEXICO | NULL | [email protected]
| 2452361 |
| 5 | AAAAAAAAFAAAAAAA | 953372
| 4470 | 86368 | 2449438
| 2449408 | Sir |
Robert | Moran | N
| 8 | 5 | 1956
| FIJI | NULL | [email protected]
| 2452469 |
| 6 | AAAAAAAAGAAAAAAA | 213219
| 6374 | 77082 | 2451883
| 2451853 | Ms. |
Brunilda | Sharp | N
| 4 | 12 | 1925
| SURINAME | NULL |
[email protected] | 2452430
|
| 7 | AAAAAAAAHAAAAAAA | 68377
| 3219 | 194814 | 2451438
| 2451408 | Ms. |
Fonda | Wiles | Y
| 24 | 4 | 1985
| GAMBIA | NULL |
[email protected] | 2452360
|
| 8 | AAAAAAAAIAAAAAAA | 1215897
| 2471 | 66598 | 2449406
| 2449376 | Sir |
Ollie | Shipman | N
| 26 | 12 | 1938
| KOREA, REPUBLIC OF | NULL | [email protected]
| 2452334 |
| 9 | AAAAAAAAJAAAAAAA | 1168667
| 1404 | 99388 | 2452275
| 2452245 | Sir |
Karl | Gilbert | N
| 26 | 10 | 1966
| MONTSERRAT | NULL |
[email protected] | 2452454
|
| 10 | AAAAAAAAKAAAAAAA | 1207553
| 5143 | 119580 | 2451353
| 2451323 | Ms. |
Albert | Brunson | N
| 15 | 10 | 1973
| JORDAN | NULL | [email protected]
| 2452641 |
+-------------------------+-------------------------+------------------------------+------------------------------+-----------------------------+----------------------------------+---------------------------------+------------------------+------------------------+---------------------------------+---------------------------------+-----------------------+-------------------------+------------------------+---------------------------+-------------------+----------------------------------------------------+---------------------------------+
{code}
Looking at attached summary: [^tpcds78_10GB_query_summary_example.txt] , I
believe we should adapt some parts of that to HS2-only queries, basically
anything which is not related to vertices and runtime outside of HS2, e.g.
something like this:
{code}
INFO : Status: Query finished successfully in 101.08 seconds
INFO :
INFO : Query Execution Summary
INFO :
----------------------------------------------------------------------------------------------
INFO : OPERATION DURATION
INFO :
----------------------------------------------------------------------------------------------
INFO : Compile Query 1.81s
INFO : Prepare Plan 0.46s
INFO : FileSystem Counters Summary
INFO :
INFO : Scheme: FILE
INFO :
----------------------------------------------------------------------------------------------
INFO: BYTES_READ READ_OPS LARGE_READ_OPS BYTES_WRITTEN
WRITE_OPS
INFO :
----------------------------------------------------------------------------------------------
INFO : 0B 0 0 4.86KB
0
INFO :
----------------------------------------------------------------------------------------------
INFO :
INFO : Scheme: S3A
INFO :
----------------------------------------------------------------------------------------------
INFO : BYTES_READ READ_OPS LARGE_READ_OPS BYTES_WRITTEN
WRITE_OPS
INFO :
----------------------------------------------------------------------------------------------
INFO : 68.53KB 4 0 0B
0
INFO :
----------------------------------------------------------------------------------------------
INFO : File System Counters:
INFO : FILE_BYTES_READ: 123426528
INFO : FILE_BYTES_WRITTEN: 264749828
INFO : FILE_READ_OPS: 0
INFO : FILE_LARGE_READ_OPS: 0
INFO : FILE_WRITE_OPS: 0
INFO : S3A_BYTES_READ: 438200740
INFO : S3A_BYTES_WRITTEN: 0
INFO : S3A_READ_OPS: 1496
INFO : S3A_LARGE_READ_OPS: 0
INFO : S3A_WRITE_OPS: 0
{code}
was:
While running a cluster query (hs2, tez am, tez+llap), we have a handy summary
attached as: [^tpcds78_10GB_query_summary_example.txt]
However, we get no info about simply fetch tasks, we can see only this below:
{code}
0: jdbc:hive2://hs2-lbodor-aws-bug.dw-dw-team> select * from customer limit 10;
INFO : Compiling
command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61):
select * from customer limit 10
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema:
Schema(fieldSchemas:[FieldSchema(name:customer.c_customer_sk, type:bigint,
comment:null), FieldSchema(name:customer.c_customer_id, type:char(16),
comment:null), FieldSchema(name:customer.c_current_cdemo_sk, type:bigint,
comment:null), FieldSchema(name:customer.c_current_hdemo_sk, type:bigint,
comment:null), FieldSchema(name:customer.c_current_addr_sk, type:bigint,
comment:null), FieldSchema(name:customer.c_first_shipto_date_sk, type:bigint,
comment:null), FieldSchema(name:customer.c_first_sales_date_sk, type:bigint,
comment:null), FieldSchema(name:customer.c_salutation, type:char(10),
comment:null), FieldSchema(name:customer.c_first_name, type:char(20),
comment:null), FieldSchema(name:customer.c_last_name, type:char(30),
comment:null), FieldSchema(name:customer.c_preferred_cust_flag, type:char(1),
comment:null), FieldSchema(name:customer.c_birth_day, type:int, comment:null),
FieldSchema(name:customer.c_birth_month, type:int, comment:null),
FieldSchema(name:customer.c_birth_year, type:int, comment:null),
FieldSchema(name:customer.c_birth_country, type:varchar(20), comment:null),
FieldSchema(name:customer.c_login, type:char(13), comment:null),
FieldSchema(name:customer.c_email_address, type:char(50), comment:null),
FieldSchema(name:customer.c_last_review_date_sk, type:bigint, comment:null)],
properties:null)
INFO : Completed compiling
command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61); Time
taken: 0.257 seconds
INFO : Executing
command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61):
select * from customer limit 10
INFO : Completed executing
command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61); Time
taken: 0.007 seconds
INFO : OK
+-------------------------+-------------------------+------------------------------+------------------------------+-----------------------------+----------------------------------+---------------------------------+------------------------+------------------------+---------------------------------+---------------------------------+-----------------------+-------------------------+------------------------+---------------------------+-------------------+----------------------------------------------------+---------------------------------+
| customer.c_customer_sk | customer.c_customer_id |
customer.c_current_cdemo_sk | customer.c_current_hdemo_sk |
customer.c_current_addr_sk | customer.c_first_shipto_date_sk |
customer.c_first_sales_date_sk | customer.c_salutation |
customer.c_first_name | customer.c_last_name |
customer.c_preferred_cust_flag | customer.c_birth_day |
customer.c_birth_month | customer.c_birth_year | customer.c_birth_country |
customer.c_login | customer.c_email_address |
customer.c_last_review_date_sk |
+-------------------------+-------------------------+------------------------------+------------------------------+-----------------------------+----------------------------------+---------------------------------+------------------------+------------------------+---------------------------------+---------------------------------+-----------------------+-------------------------+------------------------+---------------------------+-------------------+----------------------------------------------------+---------------------------------+
| 1 | AAAAAAAABAAAAAAA | 980124
| 7135 | 32946 | 2452238
| 2452208 | Mr. |
Javier | Lewis | Y
| 9 | 12 | 1936
| CHILE | NULL |
[email protected] | 2452508
|
| 2 | AAAAAAAACAAAAAAA | 819667
| 1461 | 181655 | 2452318
| 2452288 | Dr. |
Amy | Moses | Y
| 9 | 4 | 1966
| TOGO | NULL | [email protected]
| 2452318 |
| 3 | AAAAAAAADAAAAAAA | 1473522
| 6247 | 248572 | 2449130
| 2449100 | Miss |
Latisha | Hamilton | N
| 18 | 9 | 1979
| NIUE | NULL | [email protected]
| 2452313 |
| 4 | AAAAAAAAEAAAAAAA | 1703214
| 3986 | 189558 | 2450030
| 2450000 | Dr. |
Michael | White | N
| 7 | 6 | 1983
| MEXICO | NULL | [email protected]
| 2452361 |
| 5 | AAAAAAAAFAAAAAAA | 953372
| 4470 | 86368 | 2449438
| 2449408 | Sir |
Robert | Moran | N
| 8 | 5 | 1956
| FIJI | NULL | [email protected]
| 2452469 |
| 6 | AAAAAAAAGAAAAAAA | 213219
| 6374 | 77082 | 2451883
| 2451853 | Ms. |
Brunilda | Sharp | N
| 4 | 12 | 1925
| SURINAME | NULL |
[email protected] | 2452430
|
| 7 | AAAAAAAAHAAAAAAA | 68377
| 3219 | 194814 | 2451438
| 2451408 | Ms. |
Fonda | Wiles | Y
| 24 | 4 | 1985
| GAMBIA | NULL |
[email protected] | 2452360
|
| 8 | AAAAAAAAIAAAAAAA | 1215897
| 2471 | 66598 | 2449406
| 2449376 | Sir |
Ollie | Shipman | N
| 26 | 12 | 1938
| KOREA, REPUBLIC OF | NULL | [email protected]
| 2452334 |
| 9 | AAAAAAAAJAAAAAAA | 1168667
| 1404 | 99388 | 2452275
| 2452245 | Sir |
Karl | Gilbert | N
| 26 | 10 | 1966
| MONTSERRAT | NULL |
[email protected] | 2452454
|
| 10 | AAAAAAAAKAAAAAAA | 1207553
| 5143 | 119580 | 2451353
| 2451323 | Ms. |
Albert | Brunson | N
| 15 | 10 | 1973
| JORDAN | NULL | [email protected]
| 2452641 |
+-------------------------+-------------------------+------------------------------+------------------------------+-----------------------------+----------------------------------+---------------------------------+------------------------+------------------------+---------------------------------+---------------------------------+-----------------------+-------------------------+------------------------+---------------------------+-------------------+----------------------------------------------------+---------------------------------+
{code}
Looking at attached summary: [^tpcds78_10GB_query_summary_example.txt]
> Implement query summary for HS2-only queries
> --------------------------------------------
>
> Key: HIVE-27878
> URL: https://issues.apache.org/jira/browse/HIVE-27878
> Project: Hive
> Issue Type: Improvement
> Reporter: László Bodor
> Priority: Major
> Attachments: tpcds78_10GB_query_summary_example.txt
>
>
> While running a cluster query (hs2, tez am, tez+llap), we have a handy
> summary attached as: [^tpcds78_10GB_query_summary_example.txt]
> However, we get no info about simply fetch tasks, we can see only this below:
> {code}
> 0: jdbc:hive2://hs2-lbodor-aws-bug.dw-dw-team> select * from customer limit
> 10;
> INFO : Compiling
> command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61):
> select * from customer limit 10
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:customer.c_customer_sk, type:bigint,
> comment:null), FieldSchema(name:customer.c_customer_id, type:char(16),
> comment:null), FieldSchema(name:customer.c_current_cdemo_sk, type:bigint,
> comment:null), FieldSchema(name:customer.c_current_hdemo_sk, type:bigint,
> comment:null), FieldSchema(name:customer.c_current_addr_sk, type:bigint,
> comment:null), FieldSchema(name:customer.c_first_shipto_date_sk, type:bigint,
> comment:null), FieldSchema(name:customer.c_first_sales_date_sk, type:bigint,
> comment:null), FieldSchema(name:customer.c_salutation, type:char(10),
> comment:null), FieldSchema(name:customer.c_first_name, type:char(20),
> comment:null), FieldSchema(name:customer.c_last_name, type:char(30),
> comment:null), FieldSchema(name:customer.c_preferred_cust_flag, type:char(1),
> comment:null), FieldSchema(name:customer.c_birth_day, type:int,
> comment:null), FieldSchema(name:customer.c_birth_month, type:int,
> comment:null), FieldSchema(name:customer.c_birth_year, type:int,
> comment:null), FieldSchema(name:customer.c_birth_country, type:varchar(20),
> comment:null), FieldSchema(name:customer.c_login, type:char(13),
> comment:null), FieldSchema(name:customer.c_email_address, type:char(50),
> comment:null), FieldSchema(name:customer.c_last_review_date_sk, type:bigint,
> comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61);
> Time taken: 0.257 seconds
> INFO : Executing
> command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61):
> select * from customer limit 10
> INFO : Completed executing
> command(queryId=hive_20231116094515_5f3e5f7b-da6a-4026-9b36-be63ee11ff61);
> Time taken: 0.007 seconds
> INFO : OK
> +-------------------------+-------------------------+------------------------------+------------------------------+-----------------------------+----------------------------------+---------------------------------+------------------------+------------------------+---------------------------------+---------------------------------+-----------------------+-------------------------+------------------------+---------------------------+-------------------+----------------------------------------------------+---------------------------------+
> | customer.c_customer_sk | customer.c_customer_id |
> customer.c_current_cdemo_sk | customer.c_current_hdemo_sk |
> customer.c_current_addr_sk | customer.c_first_shipto_date_sk |
> customer.c_first_sales_date_sk | customer.c_salutation |
> customer.c_first_name | customer.c_last_name |
> customer.c_preferred_cust_flag | customer.c_birth_day |
> customer.c_birth_month | customer.c_birth_year | customer.c_birth_country
> | customer.c_login | customer.c_email_address |
> customer.c_last_review_date_sk |
> +-------------------------+-------------------------+------------------------------+------------------------------+-----------------------------+----------------------------------+---------------------------------+------------------------+------------------------+---------------------------------+---------------------------------+-----------------------+-------------------------+------------------------+---------------------------+-------------------+----------------------------------------------------+---------------------------------+
> | 1 | AAAAAAAABAAAAAAA | 980124
> | 7135 | 32946 | 2452238
> | 2452208 | Mr.
> | Javier | Lewis | Y
> | 9 | 12 | 1936
> | CHILE | NULL |
> [email protected] | 2452508
> |
> | 2 | AAAAAAAACAAAAAAA | 819667
> | 1461 | 181655 | 2452318
> | 2452288 | Dr.
> | Amy | Moses | Y
> | 9 | 4 | 1966
> | TOGO | NULL |
> [email protected] | 2452318
> |
> | 3 | AAAAAAAADAAAAAAA | 1473522
> | 6247 | 248572 | 2449130
> | 2449100 | Miss
> | Latisha | Hamilton | N
> | 18 | 9 | 1979
> | NIUE | NULL |
> [email protected] | 2452313
> |
> | 4 | AAAAAAAAEAAAAAAA | 1703214
> | 3986 | 189558 | 2450030
> | 2450000 | Dr.
> | Michael | White | N
> | 7 | 6 | 1983
> | MEXICO | NULL |
> [email protected] | 2452361
> |
> | 5 | AAAAAAAAFAAAAAAA | 953372
> | 4470 | 86368 | 2449438
> | 2449408 | Sir
> | Robert | Moran | N
> | 8 | 5 | 1956
> | FIJI | NULL |
> [email protected] | 2452469
> |
> | 6 | AAAAAAAAGAAAAAAA | 213219
> | 6374 | 77082 | 2451883
> | 2451853 | Ms.
> | Brunilda | Sharp | N
> | 4 | 12 | 1925
> | SURINAME | NULL |
> [email protected] | 2452430
> |
> | 7 | AAAAAAAAHAAAAAAA | 68377
> | 3219 | 194814 | 2451438
> | 2451408 | Ms.
> | Fonda | Wiles | Y
> | 24 | 4 | 1985
> | GAMBIA | NULL |
> [email protected] | 2452360
> |
> | 8 | AAAAAAAAIAAAAAAA | 1215897
> | 2471 | 66598 | 2449406
> | 2449376 | Sir
> | Ollie | Shipman | N
> | 26 | 12 | 1938
> | KOREA, REPUBLIC OF | NULL |
> [email protected] | 2452334
> |
> | 9 | AAAAAAAAJAAAAAAA | 1168667
> | 1404 | 99388 | 2452275
> | 2452245 | Sir
> | Karl | Gilbert | N
> | 26 | 10 | 1966
> | MONTSERRAT | NULL |
> [email protected] | 2452454
> |
> | 10 | AAAAAAAAKAAAAAAA | 1207553
> | 5143 | 119580 | 2451353
> | 2451323 | Ms.
> | Albert | Brunson | N
> | 15 | 10 | 1973
> | JORDAN | NULL |
> [email protected] | 2452641
> |
> +-------------------------+-------------------------+------------------------------+------------------------------+-----------------------------+----------------------------------+---------------------------------+------------------------+------------------------+---------------------------------+---------------------------------+-----------------------+-------------------------+------------------------+---------------------------+-------------------+----------------------------------------------------+---------------------------------+
> {code}
> Looking at attached summary: [^tpcds78_10GB_query_summary_example.txt] , I
> believe we should adapt some parts of that to HS2-only queries, basically
> anything which is not related to vertices and runtime outside of HS2, e.g.
> something like this:
> {code}
> INFO : Status: Query finished successfully in 101.08 seconds
> INFO :
> INFO : Query Execution Summary
> INFO :
> ----------------------------------------------------------------------------------------------
> INFO : OPERATION DURATION
> INFO :
> ----------------------------------------------------------------------------------------------
> INFO : Compile Query 1.81s
> INFO : Prepare Plan 0.46s
> INFO : FileSystem Counters Summary
> INFO :
> INFO : Scheme: FILE
> INFO :
> ----------------------------------------------------------------------------------------------
> INFO: BYTES_READ READ_OPS LARGE_READ_OPS BYTES_WRITTEN
> WRITE_OPS
> INFO :
> ----------------------------------------------------------------------------------------------
> INFO : 0B 0 0 4.86KB
> 0
> INFO :
> ----------------------------------------------------------------------------------------------
> INFO :
> INFO : Scheme: S3A
> INFO :
> ----------------------------------------------------------------------------------------------
> INFO : BYTES_READ READ_OPS LARGE_READ_OPS BYTES_WRITTEN
> WRITE_OPS
> INFO :
> ----------------------------------------------------------------------------------------------
> INFO : 68.53KB 4 0 0B
> 0
> INFO :
> ----------------------------------------------------------------------------------------------
> INFO : File System Counters:
> INFO : FILE_BYTES_READ: 123426528
> INFO : FILE_BYTES_WRITTEN: 264749828
> INFO : FILE_READ_OPS: 0
> INFO : FILE_LARGE_READ_OPS: 0
> INFO : FILE_WRITE_OPS: 0
> INFO : S3A_BYTES_READ: 438200740
> INFO : S3A_BYTES_WRITTEN: 0
> INFO : S3A_READ_OPS: 1496
> INFO : S3A_LARGE_READ_OPS: 0
> INFO : S3A_WRITE_OPS: 0
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
