Hi all,
I’m attempting to create a workflow which uses sqoop to import data from my
postgres db into a hive table. My sqoop configuration runs fine outside of
Oozie.
My main question: Does oozie support sqoop importing directly into hive? I
see some comments claiming it doesn’t, but I haven’t found anything definitive
in the Oozie docs.
For the curious, here’s what I’m experiencing. I’d be happy to provide more
details if they would help.
When I run my Oozie workflow, I see the following:
* Sqoop is able to import data into HDFS under the directory specified by
—target-dir
* When Sqoop attempts to copy this to the Hive table, it gets a permission
error as it’s attempting to write to a table as user ‘yarn’ as opposed to the
executing user. The table is owned by the executing user and yarn doesn’t have
write permissions.
29743 [Thread-31] INFO org.apache.sqoop.hive.HiveImport - Authorization
failed:java.security.AccessControlException: action WRITE not permitted on path
hdfs://cloudfeeds-visual-n02.test.ord1.ci.rackspace.net:8020/apps/hive/warehouse/entries_staging_test_ord_2014_12_01
for user yarn. Use SHOW GRANT to get more details.
30089 [main] ERROR org.apache.sqoop.tool.ImportTool - Encountered IOException
running import job: java.io.IOException: Hive exited with status 147
at
org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:385)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:335)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:239)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:425)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:506)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
at
org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:206)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Thanks for any pointers,
Greg
