Hello,I am writing a application which needs the metastore in hive. I am preparing to use webhcat to get the information about tables and process them. But a simple request takes over eight seconds to respond on localhost. Why is this so slow, and how can I fix it? $ time curl -s 'http://localhost:50111/templeton/v1/ddl/database/default/table/haha?user.name=ctdean' {"columns": [{"name":"id","type":"int"}], "database":"default", "table":"haha"}
real 0m8.400s user 0m0.053s sys 0m0.019s The webhcat.log is very short, it seems to run a hcat.py, I have looked up the log, and I can't figure out what's going on here. Here is the info when I simply run hcat. It seems take all the time. Thank you very much for your kind reply. $hcat.py -e "use default; desc haha; " SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 18/04/21 16:38:13 INFO conf.HiveConf: Found configuration file file:/usr/local/hive/conf/hive-site.xml 18/04/21 16:38:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 18/04/21 16:38:16 INFO session.SessionState: Created HDFS directory: /tmp/hive/kousouda/05096382-f9b6-4dae-aee2-dfa6750c0668 18/04/21 16:38:16 INFO session.SessionState: Created local directory: /tmp/hive/java/kousouda/05096382-f9b6-4dae-aee2-dfa6750c0668 18/04/21 16:38:16 INFO session.SessionState: Created HDFS directory: /tmp/hive/kousouda/05096382-f9b6-4dae-aee2-dfa6750c0668/_tmp_space.db 18/04/21 16:38:16 INFO ql.Driver: Compiling command(queryId=kousouda_20180421163816_58c38a44-25e3-4665-8bb5-a9b17fdf2d62): use default 18/04/21 16:38:17 INFO metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 18/04/21 16:38:17 INFO metastore.ObjectStore: ObjectStore, initialize called 18/04/21 16:38:18 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 18/04/21 16:38:18 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 18/04/21 16:38:18 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 18/04/21 16:38:20 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL 18/04/21 16:38:20 INFO metastore.ObjectStore: Initialized ObjectStore 18/04/21 16:38:20 INFO metastore.HiveMetaStore: Added admin role in metastore 18/04/21 16:38:20 INFO metastore.HiveMetaStore: Added public role in metastore 18/04/21 16:38:20 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: get_all_functions 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=get_all_functions 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: get_database: default 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=get_database: default 18/04/21 16:38:20 INFO ql.Driver: Semantic Analysis Completed 18/04/21 16:38:20 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null) 18/04/21 16:38:20 INFO ql.Driver: Completed compiling command(queryId=kousouda_20180421163816_58c38a44-25e3-4665-8bb5-a9b17fdf2d62); Time taken: 3.936 seconds 18/04/21 16:38:20 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 18/04/21 16:38:20 INFO ql.Driver: Executing command(queryId=kousouda_20180421163816_58c38a44-25e3-4665-8bb5-a9b17fdf2d62): use default 18/04/21 16:38:20 INFO sqlstd.SQLStdHiveAccessController: Created SQLStdHiveAccessController for session context : HiveAuthzSessionContext [sessionString=05096382-f9b6-4dae-aee2-dfa6750c0668, clientType=HIVECLI] 18/04/21 16:38:20 WARN session.SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory. 18/04/21 16:38:20 INFO hive.metastore: Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: Cleaning up thread local RawStore... 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=Cleaning up thread local RawStore... 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: Done cleaning up thread local RawStore 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore 18/04/21 16:38:20 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: get_database: default 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=get_database: default 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 18/04/21 16:38:20 INFO metastore.ObjectStore: ObjectStore, initialize called 18/04/21 16:38:20 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL 18/04/21 16:38:20 INFO metastore.ObjectStore: Initialized ObjectStore 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: get_database: default 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=get_database: default 18/04/21 16:38:20 INFO ql.Driver: Completed executing command(queryId=kousouda_20180421163816_58c38a44-25e3-4665-8bb5-a9b17fdf2d62); Time taken: 0.202 seconds OK 18/04/21 16:38:20 INFO ql.Driver: OK Time taken: 4.16 seconds 18/04/21 16:38:20 INFO ql.Driver: Compiling command(queryId=kousouda_20180421163820_c67f5029-7496-422d-b812-fe72661cf999): desc haha 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=haha 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=get_table : db=default tbl=haha 18/04/21 16:38:20 INFO parse.DDLSemanticAnalyzer: analyzeDescribeTable done 18/04/21 16:38:20 INFO ql.Driver: Semantic Analysis Completed 18/04/21 16:38:20 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:col_name, type:string, comment:from deserializer), FieldSchema(name:data_type, type:string, comment:from deserializer), FieldSchema(name:comment, type:string, comment:from deserializer)], properties:null) 18/04/21 16:38:20 INFO exec.ListSinkOperator: Initializing operator LIST_SINK[0] 18/04/21 16:38:20 INFO ql.Driver: Completed compiling command(queryId=kousouda_20180421163820_c67f5029-7496-422d-b812-fe72661cf999); Time taken: 0.237 seconds 18/04/21 16:38:20 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 18/04/21 16:38:20 INFO ql.Driver: Executing command(queryId=kousouda_20180421163820_c67f5029-7496-422d-b812-fe72661cf999): desc haha 18/04/21 16:38:20 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=haha 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=get_table : db=default tbl=haha 18/04/21 16:38:20 INFO ql.Driver: Completed executing command(queryId=kousouda_20180421163820_c67f5029-7496-422d-b812-fe72661cf999); Time taken: 0.081 seconds OK 18/04/21 16:38:20 INFO ql.Driver: OK 18/04/21 16:38:20 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 18/04/21 16:38:20 INFO mapred.FileInputFormat: Total input paths to process : 1 18/04/21 16:38:20 INFO exec.ListSinkOperator: Closing operator LIST_SINK[0] id int Time taken: 0.377 seconds 18/04/21 16:38:20 INFO session.SessionState: Deleted directory: /tmp/hive/kousouda/05096382-f9b6-4dae-aee2-dfa6750c0668 on fs with scheme hdfs 18/04/21 16:38:20 INFO session.SessionState: Deleted directory: /tmp/hive/java/kousouda/05096382-f9b6-4dae-aee2-dfa6750c0668 on fs with scheme file 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: Cleaning up thread local RawStore... 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=Cleaning up thread local RawStore... 18/04/21 16:38:20 INFO metastore.HiveMetaStore: 0: Done cleaning up thread local RawStore 18/04/21 16:38:20 INFO HiveMetaStore.audit: ugi=kousouda ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
