[
https://issues.apache.org/jira/browse/HIVE-29754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Indhumathi Muthumurugesh reassigned HIVE-29754:
-----------------------------------------------
Assignee: Indhumathi Muthumurugesh
> Tez leaves empty -tmp.HIVE_UNION_SUBDIR_N.moved directory in partition after
> INSERT OVERWRITE ... UNION ALL when one branch produces 0 rows
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29754
> URL: https://issues.apache.org/jira/browse/HIVE-29754
> Project: Hive
> Issue Type: Bug
> Components: Tez
> Reporter: Indhumathi Muthumurugesh
> Assignee: Indhumathi Muthumurugesh
> Priority: Major
>
> When executing INSERT OVERWRITE ... UNION ALL using the Tez execution engine,
> an empty temporary directory named -tmp.HIVE_UNION_SUBDIR_<N>.moved is left
> behind in the final partition directory after the query completes
> successfully.
> This does not occur with MapReduce execution engine.
> Steps to Reproduce
> 1. Create two tables and a partitioned output table:
> {code:java}
> CREATE TABLE src_prev (id INT, val STRING) STORED AS TEXTFILE;
> CREATE TABLE src_today (id INT, val STRING) STORED AS TEXTFILE;
> CREATE TABLE output (id INT, val STRING)
> PARTITIONED BY (run_dt STRING) STORED AS TEXTFILE; {code}
> 2. Load data such that ALL rows in src_prev also exist in src_today
> {code:java}
> INSERT INTO src_prev VALUES (1,'aaa'), (2,'bbb');
> INSERT INTO src_today VALUES (1,'aaa'), (2,'bbb'), (3,'ccc'); {code}
> 3. Run INSERT OVERWRITE with UNION ALL using Tez:
> {code:java}
> INSERT OVERWRITE TABLE output PARTITION (run_dt='2024-01-01')
> SELECT p.id, p.val
> FROM src_prev p
> LEFT OUTER JOIN src_today t ON p.id = t.id
> WHERE t.id IS NULL -- Branch 1: anti-join, returns 0 rows
> UNION ALL
> SELECT id, val FROM src_today; -- Branch 2: returns rows {code}
> 4. Inspect the output partition on HDFS:
> {code:java}
> hdfs dfs -ls /warehouse/output/run_dt=2024-01-01/ {code}
> Actual Result:
> {code:java}
> drwxr-xr-x -tmp.HIVE_UNION_SUBDIR_1.moved <- leftover empty directory
> -rw-r--r-- 1_000001_0
> -rw-r--r-- 2_000001_0 {code}
> Expected Result:
> {code:java}
> -rw-r--r-- 1_000001_0
> -rw-r--r-- 2_000001_0 {code}
> No temporary directories in the final partition location.
>
> -- Query on this table from spark engine fails with FileNotFountException
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
