[jira] [Assigned] (HIVE-21455) Too verbose logging in AvroGenericRecordReader

2019-03-27 Thread Peter Vary (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary reassigned HIVE-21455:
-

Assignee: Miklos Szurap  (was: S. Poortman19-88)

> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 1.2.0, 1.1.0, 2.0.0, 2.1.0, 3.0.0, 3.1.0
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Attachments: HIVE-21455.2.patch, HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too 
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the 
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO  org.apache.hadoop.mapred.FileInputFormat: 
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema 
> and thousands of datafiles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-21455) Too verbose logging in AvroGenericRecordReader

2019-03-24 Thread Simon poortman (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon poortman reassigned HIVE-21455:
-

Assignee: Simon poortman  (was: Miklos Szurap)

> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 1.2.0, 1.1.0, 2.0.0, 2.1.0, 3.0.0, 3.1.0
>Reporter: Miklos Szurap
>Assignee: Simon poortman
>Priority: Minor
> Attachments: HIVE-21455.2.patch, HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too 
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the 
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO  org.apache.hadoop.mapred.FileInputFormat: 
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema 
> and thousands of datafiles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-21455) Too verbose logging in AvroGenericRecordReader

2019-03-19 Thread David Mollitor (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor reassigned HIVE-21455:
-

Assignee: Miklos Szurap  (was: David Mollitor)

> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 1.2.0, 1.1.0, 2.0.0, 2.1.0, 3.0.0, 3.1.0
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Attachments: HIVE-21455.2.patch, HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too 
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the 
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO  org.apache.hadoop.mapred.FileInputFormat: 
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema 
> and thousands of datafiles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-21455) Too verbose logging in AvroGenericRecordReader

2019-03-19 Thread David Mollitor (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor reassigned HIVE-21455:
-

Assignee: David Mollitor  (was: Miklos Szurap)

> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 1.2.0, 1.1.0, 2.0.0, 2.1.0, 3.0.0, 3.1.0
>Reporter: Miklos Szurap
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too 
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the 
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO  org.apache.hadoop.mapred.FileInputFormat: 
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema 
> and thousands of datafiles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-21455) Too verbose logging in AvroGenericRecordReader

2019-03-18 Thread Miklos Szurap (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Miklos Szurap reassigned HIVE-21455:


Assignee: Miklos Szurap

> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Attachments: HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too 
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the 
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO  org.apache.hadoop.mapred.FileInputFormat: 
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema 
> and thousands of datafiles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)