Hi Hive Folks,
We're writing to S3-backed tables, but hoping to use HDFS for staging and
merging. When we set hive.exec.stagingdir to an HDFS location, we get:
16/04/19 22:59:54 [main]: ERROR ql.Driver: FAILED: IllegalArgumentException
Wrong FS:
hdfs://<redacted>:8020/tmp/hivestaging_hive_2016-04-19_22-59-52_470_66381161375278005-1,
expected: s3a://<redacted>
java.lang.IllegalArgumentException: Wrong FS:
hdfs://ip-172-20-109-129.node.dc1.consul:8020/tmp/hivestaging_hive_2016-04-19_22-59-52_470_66381161375278005-1,
expected: s3a://dwh-dev
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:646)
at
org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:466)
at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:230)
at
org.apache.hadoop.hive.ql.Context.getExtTmpPathRelTo(Context.java:426)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6271)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9007)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8898)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9743)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9636)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10109)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:329)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10120)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:211)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:456)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:316)
at
org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1181)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1229)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Is there an easy way to do this, without patching Hive or Hadoop?
Thanks!
