just for the record...
this config "hive.exec.stagingdir" determines that ".hive_staging"
sub-directory. when it defaults to the table path and the table path is in
s3 that's where i get the exception:
Failed with exception java.io.IOException: rename for src path:
s3a://trulia-dwr-cluster-dev/hi
Thanks Elliot. I think you might be onto something there. :)
Making that tiny little switch sure seemed attractive but judging from the
Jira's out there the ramifications of that setting are far more involved
and nuanced than i thought.
awright. you make some convincing arguments there. looks lik
My gut feeling is that this is not something you should do (except for
fun!) I'm fairly confident that somewhere in Hive, MR, or Tez, you'll hit
some code that requires consistent, atomic move/copy/list/overwrite
semantics from the warehouse filesystem. This is not something that the
vanilla S3AFil
no. permissions are good. i believe the case to be that s3a does not have
a "move" and/or "rename" semantic but i can't be the first one to encounter
this. somebody out there has to have gone done this path way before me
surely.
searching the cyber i find this:
https://issues.apache.org/jira/
Is it a permission issue on the folder?
> On 15 Nov 2016, at 06:28, Stephen Sprague wrote:
>
> so i figured i try and set hive.metastore.warehouse.dir=s3a://bucket/hive and
> see what would happen.
>
> running this query:
>
> insert overwrite table omniture.hit_data_aws partition
> (d
so i figured i try and set hive.metastore.warehouse.dir=s3a://bucket/hive
and see what would happen.
running this query:
insert overwrite table omniture.hit_data_aws partition
(date_key=20161113) select * from staging.hit_data_aws_ext_20161113 limit 1;
yields this error:
Failed with exce
