Hi Marcelo, yes this is a known Spark SQL bug and we've got PRs to fix it (2887 & 2967). Not merged yet because newly merged Hive 0.13.1 support causes some conflicts. Thanks for reporting this :)
On Tue, Oct 28, 2014 at 6:41 AM, Marcelo Vanzin <van...@cloudera.com> wrote: > Well, looks like a huge coincidence, but this was just sent to github: > https://github.com/apache/spark/pull/2967 > > On Mon, Oct 27, 2014 at 3:25 PM, Marcelo Vanzin <van...@cloudera.com> > wrote: > > Hey guys, > > > > I've been using the "HiveFromSpark" example to test some changes and I > > ran into an issue that manifests itself as an NPE inside Hive code > > because some configuration object is null. > > > > Tracing back, it seems that `sessionState` being a lazy val in > > HiveContext is causing it. That variably is only evaluated in [1], > > while the call in [2] causes a Driver to be initialized by [3], which > > the tries to use the thread-local session state ([4]) which hasn't > > been set yet. > > > > This could be seen as a Hive bug ([3] should probably be calling the > > constructor that takes a conf object), but is there a reason why these > > fields are lazy in HiveContext? I explicitly called > > SessionState.setCurrentSessionState() before the > > CommandProcessorFactory call and that seems to fix the issue too. > > > > [1] > https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala#L305 > > [2] > https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala#L289 > > [3] > https://github.com/apache/hive/blob/9c63b2fdc35387d735f4c9d08761203711d4974b/ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java#L104 > > [4] > https://github.com/apache/hive/blob/9c63b2fdc35387d735f4c9d08761203711d4974b/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L286 > > > > -- > > Marcelo > > > > -- > Marcelo > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
