Hi, As per my use case I need to submit multiple queries to Spark SQL in parallel but due to HiveContext being thread safe the jobs are getting submitted sequentially. I could see many threads are waiting for HiveContext.
"on-spray-can-akka.actor.default-dispatcher-26" - Thread t@149 java.lang.Thread.State: BLOCKED at org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:201) *- waiting to lock <3b714d16> (a org.apache.spark.sql.hive.HiveContext$$anon$2) owned by "on-spray-can-akka.actor.default-dispatcher-14" t@117* ------------------------------------------------------------ "on-spray-can-akka.actor.default-dispatcher-16" - Thread t@122 java.lang.Thread.State: BLOCKED at org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:201) *- waiting to lock <3b714d16> (a org.apache.spark.sql.hive.HiveContext$$anon$2) owned by "on-spray-can-akka.actor.default-dispatcher-14" t@117* I could use multiple HiveContext in different JVMs to submit the queries in parallel but is there any other way I could achieve this using a *single HiveContext*? Thanks, Dinesh. Software Engineer, Philips India
