HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Dear all, After force quiting a query because it took too long I am unable to run any queries or commands on hive because it keeps throwing a hive exception: unable to instantiate session hive meta store client. What would be the best way to solve this issue? I am using the derby database as a metastore. Mark
Serialization of HiveRelNodes
Dear all, First of all, I am sorry if I sent this message twice, I received no confirmation of my previous email actually reaching the mailing list. For my project I want to be able to access the HiveRelNodes for some extended semantic analysis. By extending the hookcontext and adding a new hook I have been able to access them serverside. To gain access to them from a client perspective I was thinking about serializing them and then deserialize them from the client (perhaps by storing the serialized form in a table). Since the RelNode and HiveRelNode classes do not implement the Serializable interface I thought maybe I could use the RelJsonWriter and RelJsonReader instead. However the RelJsonWriter is not able to convert the HiveRelNodes into a json format. Would anyone perhaps know of a different solution to serializing and deserializing these HiveRelNodes? With kind regards, Mark
Accessing Calcite RelNodes
Dear all, For a project I want to retrieve the Calcite RelNodes during the execution of the queries (perhaps even before query optimisation if possible) and perform some analysis on those RelNodes. However, I am a bit stuck with the best way to approach this. Has something similar already been done which I can leverage? Looking at the Hive Hooks, there exists a HookContext which contains the query state as a property. From there you can initialise a CalCitePlanner and call the genLogicalPlan function using an ASTNode. Would the original ASTNode of the query be stored somewhere? Is this also the preferred approach to get access to these calcite logical plans? With kind regards, Mark
Creating materialized views error acquiring locks
Dear all, I am trying to execute queries to create materialized views in Hive to see how they affect the relational Calcite plan because of the query rewriting system in Hive. I copied the TestColumnAccess class https://github.com/apache/hive/blob/master/ql/src/test/org/apache/hadoop/hive/ql/parse/TestColumnAccess.java I have defined materialized view mv1 as "select * from t1 join t2 on (t1.id1 = t2.id1)". I have modified table t1 and t2 to be stored as ORC and also set the table property "transactional" to true. In the HiveConf I set the HIVE_TXN_MANAGER to "org.apache.hadoop.hive.ql.lockmgr.DbTxnManager" and set the HIVE_SUPPORT_CONCURRENCY to true. However, when I then run the test I get the following error "Error in acquiring locks: Need to record valid WriteID list but there is no valid TxnID list". Can someone perhaps explain to me what is going wrong and what steps I can undertake to resolve the issue? With kind regards, Mark
Candidate materialized views from query history
Dear all, For research purposes I have been looking into finding potential materialized views based on a given set of queries. For instance, if you have 2 queries joining table A and B, perhaps it would be advantageous to materialize said join and utilise Calcite's ability to rewrite queries to use this newly formed materialized view. I have been looking these past few weeks into the Hive source code but I must admit, I am a little overwhelmed with the code base. I was wondering if someone could perhaps point me to a few classes of interest to get me started. I have posted a similar question by accident on the user list. I'm sorry if it seems like I am spamming the mailing lists because of this. With kind regards, Mark
Importing into eclipse
Dear all, Sorry if this is not the right communication channel for this type of question but I am a bit stuck. I am trying to import Apache Hive into eclipse using the Apache Hive developers FAQ: https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ In my eclipse workspace, I cloned the github repository. I removed the .m2/repository/org/apache/hive folder. I executed the described maven command: "mvn clean package eclipse:clean eclipse:eclipse -Pitests -DskipTests -DdownloadSources -DdownloadJavadocs". Then I tried to import the project into the workspace first as an existing maven project from the root folder and when that did not work I tried it as an existing project which also did not work. Running maven install to try and resolve the issues in eclipse fails because of failing tests in the hive query language project. Upon further inspection it seems to fail because hadoop_home and hadoop.home.dir are unset at org.apache.hadoop.util.Shell.checkHadoopHome. However, hadoop_home is set in my path (version 3.1.1). Again, I'm sorry if this is not the right place for these types of questions. I am a bit new to hive (and hadoop in general) but I would greatly appreciate it if someone can recognize what I did wrong and what I can do to resolve it. With kind regards, Mark
