Further investigation appears to show this going wrong in a copy phase of
the plan. The correctly functioning HDFS → HDFS import copy stage looks
like this:

STAGE PLANS:
  Stage: Stage-1
    Copy
      source: hdfs://host:8020/staging/my_table/year_month=2015-12
      destination:
hdfs://host:8020/tmp/hive/hadoop/4f155e62-cec1-4b35-95e5-647ab5a74d3d/hive_2016-01-07_17-27-48_864_1838369633925145253-1/-ext-10000


Whereas the S3 → S3 import copy stage shows an unexpected destination,
which was presumably meant to be a temporary location on the source file
system but is in fact simply the parent directory:


STAGE PLANS:
  Stage: Stage-1
    Copy
      source: s3n://exports-bucket/my_table/year_month=2015-12
      destination: s3n://exports-bucket/my_table


These stage plans were obtained using:

EXPLAIN
IMPORT FROM 'spource
LOCATION 'destination';


I'm beginning to think that this is a bug and not something I can work
around, which is unfortunate as I'm not really in a position to deploy a
fixed version in the short term. That said, if you confirm that this is not
the intended behaviour, I'll raise a JIRA and possibly work on a fix.

Thanks - Elliot.


On 7 January 2016 at 16:53, Elliot West <[email protected]> wrote:

> More information: This works if I move the export into EMR's HDFS and then
> import from there to a new location in HDFS. It does not work across
> FileSystems:
>
>    - Import from S3 → EMR HDFS (fails in a similar manner to S3 → S3)
>    - Import from EMR HDFS → S3 (complains that HDFS FileSystem was
>    expected as the destination. Presumably the same FileSystem instance
>    is used for the source and destination).
>
>
>
> On 7 January 2016 at 12:17, Elliot West <[email protected]> wrote:
>
>> Hello,
>>
>> Following on from my earlier post concerning syncing Hive data from an on
>> premise cluster to the cloud, I've been experimenting with the
>> IMPORT/EXPORT functionality to move data from an on-premise HDP cluster to
>> Amazon EMR. I started out with some simple Exports/Imports as these can be
>> the core operations on which replication is founded. This worked fine with
>> some on-premise clusters running HDP-2.2.4.
>>
>>
>> // on cluster 1
>>
>> EXPORT TABLE my_table PARTITION (year_month='2015-12')
>> TO '/exports/my_table'
>> FOR REPLICATION ('1');
>>
>> // Copy from cluster1:/exports/my_table to cluster2:/staging/my_table
>>
>> // on cluster 2
>>
>> IMPORT FROM '/staging/my_table'
>> LOCATION '/warehouse/my_table';
>>
>> // Table created, partition created, data relocated to
>> /warehouse/my_table/year_month=2015-12
>>
>>
>> I next tried similar with HDP-2.2.4 → EMR (4.2.0) like so:
>>
>> // On premise HDP2.2.4
>> SET hiveconf:hive.exim.uri.scheme.whitelist=hdfs,pfile,s3n;
>>
>> EXPORT TABLE my_table PARTITION (year_month='2015-12')
>> TO 's3n://API_KEY:SECRET_KEY@exports-bucket/my_table'
>>
>> // on EMR
>> SET hiveconf:hive.exim.uri.scheme.whitelist=hdfs,pfile,s3n;
>>
>> IMPORT FROM 's3n://exports-bucket/my_table'
>> LOCATION 's3n://hive-warehouse-bucket/my_table'
>>
>>
>> The IMPORT behaviour I see is bizarre:
>>
>>    1. Creates the folder 's3n://hive-warehouse/my_table'
>>    2. Copies the part file from
>>    's3n://exports-bucket/my_table/year_month=2015-12' to
>>    's3n://exports-bucket/my_table' (i.e. to the parent)
>>    3. Fails with: "ERROR exec.Task: Failed with exception checkPaths:
>>    s3n://exports-bucket/my_table has nested
>>    directorys3n://exports-bucket/my_table/year_month=2015-12"
>>
>> It is as if it is attempting to set the final partition location to
>> 's3n://exports-bucket/my_table' and not
>> 's3n://hive-warehouse-bucket/my_table/year_month=2015-12' as happens with
>> HDP → HDP.
>>
>> I've tried variations, specifying the partition on import, excluding the
>> location, all with the same result. Any thoughts or assistance would be
>> appreciated.
>>
>> Thanks - Elliot.
>>
>>
>>
>

Reply via email to