Re: Unit testing with HiveContext
Thanks Ted, using HiveTest as my context worked. It still left a metastore directory and Derby log in my current working directory though; I manually added a shutdown hook to delete them and all was well. On Wed, Apr 8, 2015 at 4:33 PM, Ted Yu yuzhih...@gmail.com wrote: Please take a look at sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala : protected def configure(): Unit = { warehousePath.delete() metastorePath.delete() setConf(javax.jdo.option.ConnectionURL, sjdbc:derby:;databaseName=$metastorePath;create=true) setConf(hive.metastore.warehouse.dir, warehousePath.toString) } Cheers On Wed, Apr 8, 2015 at 1:07 PM, Daniel Siegmann daniel.siegm...@teamaol.com wrote: I am trying to unit test some code which takes an existing HiveContext and uses it to execute a CREATE TABLE query (among other things). Unfortunately I've run into some hurdles trying to unit test this, and I'm wondering if anyone has a good approach. The metastore DB is automatically created in the local directory, but it doesn't seem to be cleaned up afterward. Is there any way to get Spark to clean this up when the context is stopped? Or can I point this to some other location, such as a temp directory? Trying to create a table fails because it is using the default warehouse directory (/user/hive/warehouse). Is there some way to change this without hard-coding a directory in a hive-site.xml; again, I'd prefer to point it to a temp directory so it will be automatically removed. I tried a couple of things that didn't work: - hiveContext.sql(SET hive.metastore.warehouse.dir=/tmp/dir/xyz) - hiveContext.setConf(hive.metastore.warehouse.dir, /tmp/dir/xyz) Any advice from those who have been here before would be appreciated.
Unit testing with HiveContext
I am trying to unit test some code which takes an existing HiveContext and uses it to execute a CREATE TABLE query (among other things). Unfortunately I've run into some hurdles trying to unit test this, and I'm wondering if anyone has a good approach. The metastore DB is automatically created in the local directory, but it doesn't seem to be cleaned up afterward. Is there any way to get Spark to clean this up when the context is stopped? Or can I point this to some other location, such as a temp directory? Trying to create a table fails because it is using the default warehouse directory (/user/hive/warehouse). Is there some way to change this without hard-coding a directory in a hive-site.xml; again, I'd prefer to point it to a temp directory so it will be automatically removed. I tried a couple of things that didn't work: - hiveContext.sql(SET hive.metastore.warehouse.dir=/tmp/dir/xyz) - hiveContext.setConf(hive.metastore.warehouse.dir, /tmp/dir/xyz) Any advice from those who have been here before would be appreciated.
Re: Unit testing with HiveContext
Please take a look at sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala : protected def configure(): Unit = { warehousePath.delete() metastorePath.delete() setConf(javax.jdo.option.ConnectionURL, sjdbc:derby:;databaseName=$metastorePath;create=true) setConf(hive.metastore.warehouse.dir, warehousePath.toString) } Cheers On Wed, Apr 8, 2015 at 1:07 PM, Daniel Siegmann daniel.siegm...@teamaol.com wrote: I am trying to unit test some code which takes an existing HiveContext and uses it to execute a CREATE TABLE query (among other things). Unfortunately I've run into some hurdles trying to unit test this, and I'm wondering if anyone has a good approach. The metastore DB is automatically created in the local directory, but it doesn't seem to be cleaned up afterward. Is there any way to get Spark to clean this up when the context is stopped? Or can I point this to some other location, such as a temp directory? Trying to create a table fails because it is using the default warehouse directory (/user/hive/warehouse). Is there some way to change this without hard-coding a directory in a hive-site.xml; again, I'd prefer to point it to a temp directory so it will be automatically removed. I tried a couple of things that didn't work: - hiveContext.sql(SET hive.metastore.warehouse.dir=/tmp/dir/xyz) - hiveContext.setConf(hive.metastore.warehouse.dir, /tmp/dir/xyz) Any advice from those who have been here before would be appreciated.