Indhumathi Muthumurugesh created HIVE-29754:
-----------------------------------------------
Summary: Tez leaves empty -tmp.HIVE_UNION_SUBDIR_N.moved
directory in partition after INSERT OVERWRITE ... UNION ALL when one branch
produces 0 rows
Key: HIVE-29754
URL: https://issues.apache.org/jira/browse/HIVE-29754
Project: Hive
Issue Type: Bug
Components: Tez
Reporter: Indhumathi Muthumurugesh
When executing INSERT OVERWRITE ... UNION ALL using the Tez execution engine,
an empty temporary directory named -tmp.HIVE_UNION_SUBDIR_<N>.moved is left
behind in the final partition directory after the query completes successfully.
This does not occur with MapReduce execution engine.
Steps to Reproduce
1. Create two tables and a partitioned output table:
{code:java}
CREATE TABLE src_prev (id INT, val STRING) STORED AS TEXTFILE;
CREATE TABLE src_today (id INT, val STRING) STORED AS TEXTFILE;
CREATE TABLE output (id INT, val STRING)
PARTITIONED BY (run_dt STRING) STORED AS TEXTFILE; {code}
2. Load data such that ALL rows in src_prev also exist in src_today
{code:java}
INSERT INTO src_prev VALUES (1,'aaa'), (2,'bbb');
INSERT INTO src_today VALUES (1,'aaa'), (2,'bbb'), (3,'ccc'); {code}
3. Run INSERT OVERWRITE with UNION ALL using Tez:
{code:java}
INSERT OVERWRITE TABLE output PARTITION (run_dt='2024-01-01')
SELECT p.id, p.val
FROM src_prev p
LEFT OUTER JOIN src_today t ON p.id = t.id
WHERE t.id IS NULL -- Branch 1: anti-join, returns 0 rows
UNION ALL
SELECT id, val FROM src_today; -- Branch 2: returns rows {code}
4. Inspect the output partition on HDFS:
{code:java}
hdfs dfs -ls /warehouse/output/run_dt=2024-01-01/ {code}
Actual Result:
{code:java}
drwxr-xr-x -tmp.HIVE_UNION_SUBDIR_1.moved <- leftover empty directory
-rw-r--r-- 1_000001_0
-rw-r--r-- 2_000001_0 {code}
Expected Result:
{code:java}
-rw-r--r-- 1_000001_0
-rw-r--r-- 2_000001_0 {code}
No temporary directories in the final partition location.
-- Query on this table from spark engine fails with FileNotFountException
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
