[ https://issues.apache.org/jira/browse/HIVE-28144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HIVE-28144: ---------------------------------- Labels: pull-request-available (was: ) > Remove overly verbose debug messages from MetastoreDirectSqlUtils > ----------------------------------------------------------------- > > Key: HIVE-28144 > URL: https://issues.apache.org/jira/browse/HIVE-28144 > Project: Hive > Issue Type: Task > Components: Standalone Metastore > Reporter: Stamatis Zampetakis > Assignee: Stamatis Zampetakis > Priority: Major > Labels: pull-request-available > > When BITVECTOR or KLL stats are disabled/not present in the metastore [the > following > message|https://github.com/apache/hive/blob/8eee4aa9d1bd6f4193471f5d014324bbaf552041/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L594] > may appear way too often in the HMS logs. > {noformat} > 2024-03-22T01:50:57,849 DEBUG [CachedStore-CacheUpdateService: Thread-240] > metastore.MetastoreDirectSqlUtils: Expected blob type but got java.lang.String > {noformat} > In fact in some cases, the message appears more than once for every single > partition that is present in the table(s) being queried. When the number of > partitions is important it can easily clog the logs with redundant and > useless information. > To put things in perspective while running the cbo_query10.q on the > statistics of TPC-DS30TB dataset the message occupies more than 50% (26MB) of > the total log file (46MB). > {noformat} > $mvn test -Dtest=TestTezTPCDS30TBPerfCliDriver -Dqfile=cbo_query10.q > $grep -a "Expected blob type but got java.lang.String" > target/tmp/log/hive.log | wc -c > 26129538 > $ wc -c target/tmp/log/hive.log > 46959003 target/tmp/log/hive.log > {noformat} > The presence of the message does not tells us much on its own. In conjunction > with the code we can infer that we are not fetching BITVECTOR/KLL stats from > the metastore but this could be done in a different place without having to > print the same message 170K times. > > Removing this message saves disk space, avoids frequent log rotation, and > improves the overall readability of the log file. > There is another redundant > message which appears when transforming a [database value to > Boolean|https://github.com/apache/hive/blob/8eee4aa9d1bd6f4193471f5d014324bbaf552041/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L557]. > The message is redundant since it is followed directly by an exception so > there is no reason to have both. This message may not appear as often as the > previous one but given that it doesn't add much value it can also be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)