All:
I recently ran into a scenario where spark-shell could communicate with
Hive but another application of mine (Spark Notebook) could not. When I
tried to get a reference to a table using sql.table("tab") Spark Notebook
threw an exception but spark-shell did not.
I was trying to track down the difference between the two applications and
was having a hard time figuring out what it was.
The problem was resolved by tweaking a hive-site.xml security setting, but
I'm still curious about how it works.
It seems like spark-shell knows how to look at
$SPARK_HOME/conf/hive-site.xml and communicate with the HiveServer
directly. But my other application doesn't know anything about
hive-site.xml and must communicate with another piece of Spark to get the
information. Originally this indirect communication didn't work, but after
the tweak to hive-site.xml it does.
How does the communication between the driver and Hive work? And is
spark-shell somehow special in this regard?
Thanks,
-- Eric
