Re: shutdown local hivecontext?
Well. I managed to solve that issue after running my tests on a linux
system instead of windows (which I was originally using). However, now I
have an error when I try to reset the hive context using hc.reset(). It
tries to create a file inside directory /user/my_user_name instead of the
usual linux path /home/my_user_name, which fails.
On Thu, Aug 6, 2015 at 3:12 PM, Cesar Flores wrote:
> Well, I try this approach, and still have issues. Apparently TestHive can
> not delete the hive metastore directory. The complete error that I have is:
>
> 15/08/06 15:01:29 ERROR Driver: FAILED: Execution Error, return code 1
> from org.apache.hadoop.hive.ql.exec.DDLTask.
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NullPointerException
> 15/08/06 15:01:29 ERROR TestHive:
> ==
> HIVE FAILURE OUTPUT
> ==
> SET spark.sql.test=
> SET
> javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=C:\cygwin64\tmp\sparkHiveMetastore1376676777991703945;create=true
> SET
> hive.metastore.warehouse.dir=C:\cygwin64\tmp\sparkHiveWarehouse5264564710014125096
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NullPointerException
>
> ==
> END HIVE FAILURE OUTPUT
> ==
>
> [error] Uncaught exception when running
> com.dotomi.pipeline.utilitytransformers.SorterTransformerSuite:
> java.lang.ExceptionInInitializerError
> [trace] Stack trace suppressed: run last pipeline/test:testOnly for the
> full output.
> 15/08/06 15:01:29 ERROR Utils: Exception while deleting Spark temp dir:
> C:\cygwin64\tmp\sparkHiveMetastore1376676777991703945
> java.io.IOException: Failed to delete:
> C:\cygwin64\tmp\sparkHiveMetastore1376676777991703945
> at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:932)
> at
> org.apache.spark.util.Utils$$anon$4$$anonfun$run$1$$anonfun$apply$mcV$sp$2.apply(Utils.scala:181)
> at
> org.apache.spark.util.Utils$$anon$4$$anonfun$run$1$$anonfun$apply$mcV$sp$2.apply(Utils.scala:179)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> at
> org.apache.spark.util.Utils$$anon$4$$anonfun$run$1.apply$mcV$sp(Utils.scala:179)
> at
> org.apache.spark.util.Utils$$anon$4$$anonfun$run$1.apply(Utils.scala:177)
> at
> org.apache.spark.util.Utils$$anon$4$$anonfun$run$1.apply(Utils.scala:177)
> at
> org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> at org.apache.spark.util.Utils$$anon$4.run(Utils.scala:177)
>
> Any new idea about how to avoid this error? I think the problem may be
> running the tests on sbt, as the created directories are locked until I
> exit the sbt command shell from where I run the tests. Please let me know
> if you have any other suggestion.
>
>
> Thanks
>
> On Mon, Aug 3, 2015 at 5:56 PM, Michael Armbrust
> wrote:
>
>> TestHive takes care of creating a temporary directory for each invocation
>> so that multiple test runs won't conflict.
>>
>> On Mon, Aug 3, 2015 at 3:09 PM, Cesar Flores wrote:
>>
>>>
>>> We are using a local hive context in order to run unit tests. Our unit
>>> tests runs perfectly fine if we run why by one using sbt as the next
>>> example:
>>>
>>> >sbt test-only com.company.pipeline.scalers.ScalerSuite.scala
>>> >sbt test-only com.company.pipeline.labels.ActiveUsersLabelsSuite.scala
>>>
>>> However, if we try to run them as:
>>>
>>> >sbt test-only com.company.pipeline.*
>>>
>>> we start to run into issues. It appears that the issue is that the hive
>>> context is not properly shutdown after finishing the first test. Does any
>>> one know how to attack this problem? The test part in my build.sbt file
>>> looks like:
>>>
>>> libraryDependencies += "org.scalatest" % "scalatest_2.10" % "2.0" %
>>> "test",
>>> parallelExecution in Test := false,
>>> fork := true,
>>> javaOptions ++= Seq("-Xms512M", "-Xmx2048M", "-XX:MaxPermSize=2048M",
>>> "-XX:+CMSClassUnloadingEnabled")
>>>
>>> We are working under Spark 1.3.0
>>>
>>>
>>> Thanks
>>> --
>>> Cesar Flores
>>>
>>
>>
>
>
> --
> Cesar Flores
>
--
Cesar Flores
Re: shutdown local hivecontext?
Well, I try this approach, and still have issues. Apparently TestHive can
not delete the hive metastore directory. The complete error that I have is:
15/08/06 15:01:29 ERROR Driver: FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
15/08/06 15:01:29 ERROR TestHive:
==
HIVE FAILURE OUTPUT
==
SET spark.sql.test=
SET
javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=C:\cygwin64\tmp\sparkHiveMetastore1376676777991703945;create=true
SET
hive.metastore.warehouse.dir=C:\cygwin64\tmp\sparkHiveWarehouse5264564710014125096
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
==
END HIVE FAILURE OUTPUT
==
[error] Uncaught exception when running
com.dotomi.pipeline.utilitytransformers.SorterTransformerSuite:
java.lang.ExceptionInInitializerError
[trace] Stack trace suppressed: run last pipeline/test:testOnly for the
full output.
15/08/06 15:01:29 ERROR Utils: Exception while deleting Spark temp dir:
C:\cygwin64\tmp\sparkHiveMetastore1376676777991703945
java.io.IOException: Failed to delete:
C:\cygwin64\tmp\sparkHiveMetastore1376676777991703945
at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:932)
at
org.apache.spark.util.Utils$$anon$4$$anonfun$run$1$$anonfun$apply$mcV$sp$2.apply(Utils.scala:181)
at
org.apache.spark.util.Utils$$anon$4$$anonfun$run$1$$anonfun$apply$mcV$sp$2.apply(Utils.scala:179)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at
org.apache.spark.util.Utils$$anon$4$$anonfun$run$1.apply$mcV$sp(Utils.scala:179)
at
org.apache.spark.util.Utils$$anon$4$$anonfun$run$1.apply(Utils.scala:177)
at
org.apache.spark.util.Utils$$anon$4$$anonfun$run$1.apply(Utils.scala:177)
at
org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
at org.apache.spark.util.Utils$$anon$4.run(Utils.scala:177)
Any new idea about how to avoid this error? I think the problem may be
running the tests on sbt, as the created directories are locked until I
exit the sbt command shell from where I run the tests. Please let me know
if you have any other suggestion.
Thanks
On Mon, Aug 3, 2015 at 5:56 PM, Michael Armbrust
wrote:
> TestHive takes care of creating a temporary directory for each invocation
> so that multiple test runs won't conflict.
>
> On Mon, Aug 3, 2015 at 3:09 PM, Cesar Flores wrote:
>
>>
>> We are using a local hive context in order to run unit tests. Our unit
>> tests runs perfectly fine if we run why by one using sbt as the next
>> example:
>>
>> >sbt test-only com.company.pipeline.scalers.ScalerSuite.scala
>> >sbt test-only com.company.pipeline.labels.ActiveUsersLabelsSuite.scala
>>
>> However, if we try to run them as:
>>
>> >sbt test-only com.company.pipeline.*
>>
>> we start to run into issues. It appears that the issue is that the hive
>> context is not properly shutdown after finishing the first test. Does any
>> one know how to attack this problem? The test part in my build.sbt file
>> looks like:
>>
>> libraryDependencies += "org.scalatest" % "scalatest_2.10" % "2.0" %
>> "test",
>> parallelExecution in Test := false,
>> fork := true,
>> javaOptions ++= Seq("-Xms512M", "-Xmx2048M", "-XX:MaxPermSize=2048M",
>> "-XX:+CMSClassUnloadingEnabled")
>>
>> We are working under Spark 1.3.0
>>
>>
>> Thanks
>> --
>> Cesar Flores
>>
>
>
--
Cesar Flores
Re: shutdown local hivecontext?
TestHive takes care of creating a temporary directory for each invocation
so that multiple test runs won't conflict.
On Mon, Aug 3, 2015 at 3:09 PM, Cesar Flores wrote:
>
> We are using a local hive context in order to run unit tests. Our unit
> tests runs perfectly fine if we run why by one using sbt as the next
> example:
>
> >sbt test-only com.company.pipeline.scalers.ScalerSuite.scala
> >sbt test-only com.company.pipeline.labels.ActiveUsersLabelsSuite.scala
>
> However, if we try to run them as:
>
> >sbt test-only com.company.pipeline.*
>
> we start to run into issues. It appears that the issue is that the hive
> context is not properly shutdown after finishing the first test. Does any
> one know how to attack this problem? The test part in my build.sbt file
> looks like:
>
> libraryDependencies += "org.scalatest" % "scalatest_2.10" % "2.0" % "test",
> parallelExecution in Test := false,
> fork := true,
> javaOptions ++= Seq("-Xms512M", "-Xmx2048M", "-XX:MaxPermSize=2048M",
> "-XX:+CMSClassUnloadingEnabled")
>
> We are working under Spark 1.3.0
>
>
> Thanks
> --
> Cesar Flores
>
shutdown local hivecontext?
We are using a local hive context in order to run unit tests. Our unit
tests runs perfectly fine if we run why by one using sbt as the next
example:
>sbt test-only com.company.pipeline.scalers.ScalerSuite.scala
>sbt test-only com.company.pipeline.labels.ActiveUsersLabelsSuite.scala
However, if we try to run them as:
>sbt test-only com.company.pipeline.*
we start to run into issues. It appears that the issue is that the hive
context is not properly shutdown after finishing the first test. Does any
one know how to attack this problem? The test part in my build.sbt file
looks like:
libraryDependencies += "org.scalatest" % "scalatest_2.10" % "2.0" % "test",
parallelExecution in Test := false,
fork := true,
javaOptions ++= Seq("-Xms512M", "-Xmx2048M", "-XX:MaxPermSize=2048M",
"-XX:+CMSClassUnloadingEnabled")
We are working under Spark 1.3.0
Thanks
--
Cesar Flores
