[
https://issues.apache.org/jira/browse/HIVE-22088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Puneet Khatod updated HIVE-22088:
---------------------------------
Description:
If external table path has a '=' sign (coincidentally partition specifier) in
it, then dynamic partition loading fails.
*Use cases:*
Quite often the same data is used in different contexts by creating different
external tables on top of the data. Many times the tables have different
partition depths depending on how data is organized. Static partition insert on
external table with '=' (partition specifier) in path works fine but dynamic
partition fails.
*Replication:*
To replicate following simple setup could be done. Below execution is on 'Tez'.
*Source table**-*
CREATE EXTERNAL TABLE temp_dummy_table
(id STRING, type STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/source/';
*Destination Table-*
CREATE EXTERNAL TABLE temp_dummy_dest_table
(id STRING)
PARTITIONED BY (type string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/destination/{color:#FF0000}customer=abc{color}/';
*Insert into destination-*
insert overwrite table temp_dummy_dest_table partition (type)
select i.id as id, i.type as type
from temp_dummy_table i
where i.type in ('type1','type2');
*Log and Error Msgs on CLI*-
Loading data to table temp_dummy_dest_table partition (type=null)
Failed with exception Partition spec \{type=type1, customer=abc} contains
non-partition columns
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
*Possible resolution:*
The dynamic partitioning should consider only those partition specs which are
under teh defined table root path. If the path itself has partition style
format (customer=abc in above example) then that should not be considered as
partition as it is outside the scope of the table.
was:
If external table path has a '=' sign (coincidentally partition specifier) in
it, then dynamic partition loading fails.
*Use cases:*
Quite often the same data is used in different contexts by creating different
external tables on top of the data. Many times the tables have different
partition depths depending on how data is organized. Static partition insert on
external table with '=' (partition specifier) in path works fine but dynamic
partition fails.
*Replication:*
To replicate following simple setup could be done. Below execution is on 'Tez'.
*Source table**-*
CREATE EXTERNAL TABLE temp_dummy_table
(id STRING, type STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/source/';
*Destination Table-*
CREATE EXTERNAL TABLE temp_dummy_dest_table
(id STRING)
PARTITIONED BY (type string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/destination/customer=abc/';
*Insert into destination-*
insert overwrite table temp_dummy_dest_table partition (type)
select i.id as id, i.type as type
from temp_dummy_table i
where i.type in ('type1','type2');
*Log and Error Msgs on CLI*-
Loading data to table temp_dummy_dest_table partition (type=null)
Failed with exception Partition spec \{type=type1, customer=abc} contains
non-partition columns
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
*Possible resolution:*
The dynamic partitioning should consider only those partition specs which are
under teh defined table root path. If the path itself has partition style
format (customer=abc in above example) then that should not be considered as
partition as it is outside the scope of the table.
> Dynamic partition insert problem on external table with "=" in path
> -------------------------------------------------------------------
>
> Key: HIVE-22088
> URL: https://issues.apache.org/jira/browse/HIVE-22088
> Project: Hive
> Issue Type: Bug
> Environment: Hive 2.6.0.10-2 Executing on Tez.
> Config settings used:
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
> Reporter: Puneet Khatod
> Priority: Major
>
> If external table path has a '=' sign (coincidentally partition specifier) in
> it, then dynamic partition loading fails.
> *Use cases:*
> Quite often the same data is used in different contexts by creating different
> external tables on top of the data. Many times the tables have different
> partition depths depending on how data is organized. Static partition insert
> on external table with '=' (partition specifier) in path works fine but
> dynamic partition fails.
> *Replication:*
> To replicate following simple setup could be done. Below execution is on
> 'Tez'.
> *Source table**-*
> CREATE EXTERNAL TABLE temp_dummy_table
> (id STRING, type STRING)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/home/source/';
>
> *Destination Table-*
> CREATE EXTERNAL TABLE temp_dummy_dest_table
> (id STRING)
> PARTITIONED BY (type string)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/home/destination/{color:#FF0000}customer=abc{color}/';
>
> *Insert into destination-*
> insert overwrite table temp_dummy_dest_table partition (type)
> select i.id as id, i.type as type
> from temp_dummy_table i
> where i.type in ('type1','type2');
>
> *Log and Error Msgs on CLI*-
> Loading data to table temp_dummy_dest_table partition (type=null)
> Failed with exception Partition spec \{type=type1, customer=abc} contains
> non-partition columns
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
>
> *Possible resolution:*
> The dynamic partitioning should consider only those partition specs which are
> under teh defined table root path. If the path itself has partition style
> format (customer=abc in above example) then that should not be considered as
> partition as it is outside the scope of the table.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
