Alex Behm has posted comments on this change. Change subject: Improve logging of table loading. ......................................................................
Patch Set 1: (10 comments) Thanks, Bharath. Good suggestions. http://gerrit.cloudera.org:8080/#/c/5709/1//COMMIT_MSG Commit Message: Line 7: Improve logging of table loading. > JIRA id? I think it helps in backporting to other branches. Don't think we're going to backport this anywhere, but done anyway. http://gerrit.cloudera.org:8080/#/c/5709/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: Line 528: LOG.trace("Loading native functions for database: " + db.getName()); > Should we enable logging related to fns (here and below) ? I think it helps I agree. The basic loading steps should be included in the logs. To make the intention clearer, I changed the new log messages to LOG.info() because LOG.debug() could mislead a reader. Line 898: LOG.debug(String.format("Refreshing metadata: %s", tbl.getFullName())); > Given we support both table & partition granularity, Is it better to explic Agree. Done. Line 1036: tableName.getDb_name(), tableName.getTable_name())); > this and invalidateDb()? Can potentially help the frequency with which inva Agree. invalidateDb() is a private function only called from reset(), so I added the logging there (indicates an "invalidate metadata"). Line 1262: public Table reloadPartition(Table tbl, List<TPartitionKeyValue> partitionSpec) > Should we add this too? Yea I think so. http://gerrit.cloudera.org:8080/#/c/5709/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: Line 681: LOG.debug("Creating empty partition objects: " + getFullName()); > Log statment would probably be confusing for non-partitioned tables? Shoul I'm not even sure this message is useful. Removed. PS1, Line 1057: db_.getName() + "." + name_ > getFullName()? Done Line 1081: LOG.debug("Incrementally loaded metadata for: " + tableName); > I have a feeling these could be noisy and can do a lot of logging. We only get into this code path for INSERT, ALTER TABLE, TRUNCATE TABLE and DROP STATS. Do you still think it will be too noisy? http://gerrit.cloudera.org:8080/#/c/5709/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: Line 570: LOG.debug("Loading table metadata: " + tbl.getFullName()); > This will be logged twice for non-hdfs tables, once here and once inside Ta Assuming you mean HDFS tables. The HDFS loading message is a little more detailed, it says whether an incremental or complete load is being done. I think it also makes sense to track the caller of load in this case. What do you think? http://gerrit.cloudera.org:8080/#/c/5709/1/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: Line 915: LOG.trace(String.format("Missing tables were not received in %dms. Load " + > Isn't this a warn/error? Done -- To view, visit http://gerrit.cloudera.org:8080/5709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8de96d0cb6d09b2272b1925d42cb059367fe7196 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Alex Behm <[email protected]> Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-HasComments: Yes
