Thanks Pavan for forwarding.With respect to the issue of not seeing data in
hive tables, I figured out that oozie spark action is writing to the default
/user/hive/warehouse location, but not to the location that I have set in
hive-site.xml by setting hive.metastore.warehouse.dir...I tried to set this
location programmatically in the spark job by explicitly doing a
sqlContext.setConf("hive.metastore.warehouse.dir","...")
as well as by trying to set a :val options = Map("path" ->
".....")df.write.mode(tableMode).options(options).saveAsTable(outTable) But
nothing worked...How can I have the oozie spark-action write the table to a
different warehouse location other than the default ?
thanksSandeep
On Monday, March 14, 2016 5:09 AM, pavan kumar Kolamuri
<[email protected]> wrote:
++ oozie users
If someone experienced this, please help.
---------- Forwarded message ----------
From: Sandeep Bajaj <[email protected]>
Date: Sat, Mar 12, 2016 at 1:16 AM
Subject: Oozie spark action
To: "[email protected]" <[email protected]>
Pavan
Thanks for adding support for spark action by resolving :
[OOZIE-1983] Add spark action executor - ASF JIRA
<https://issues.apache.org/jira/browse/OOZIE-1983>
[OOZIE-1983] Add spark action executor - ASF JIRA
<https://issues.apache.org/jira/browse/OOZIE-1983>
https://reviews.apache.org/r/27614/
View on issues.apache.org <https://issues.apache.org/jira/browse/OOZIE-1983>
Preview by Yahoo
However I have been having some issues using it and was hoping if you could
help. One of the errors that I see occasionally is the "TFS filesytem not
supported" , as discussed in this thread as well:
https://community.hortonworks.com/questions/17791/oozie-submit-spark-action-but-get-a-error-not-impl.html
Secondly as I try to write to hive tables from spark, I dont see any
content in the hive table...there is no associated error in the spark/oozie
logs
Both the above issues dont happen when I run spark-submit. My understanding
is that the spark-action is implemented as a spark-submit job...Do u know
what might be potentially different ?
thanks in advance for your help
Sandeep
--
Regards
Pavan Kumar Kolamuri
