[jira] [Commented] (HIVE-22077) Inserting overwrite partitions clause does not clean directories while partitions' info is not stored in metadata
[
https://issues.apache.org/jira/browse/HIVE-22077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17791918#comment-17791918
]
Kiran Velumuri commented on HIVE-22077:
---
[~Bone An] I see that this issue has been inactive for a long time. In case you
are not working on this, can I pick this up? Thanks.
> Inserting overwrite partitions clause does not clean directories while
> partitions' info is not stored in metadata
> -
>
> Key: HIVE-22077
> URL: https://issues.apache.org/jira/browse/HIVE-22077
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.1, 2.3.4, 4.0.0
>Reporter: Hui An
>Assignee: Hui An
>Priority: Major
> Attachments: HIVE-22077.patch.1
>
>
> Inserting overwrite static partitions may not clean related HDFS location if
> partitions' info is not stored in metadata.
> Steps to reproduce this issue :
>
> 1. Create a managed table :
>
> {code:sql}
> CREATE TABLE `test`(
>`id` string)
> PARTITIONED BY (
>`dayno` string)
> ROW FORMAT SERDE
>'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>'hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test'
> TBLPROPERTIES (
>'transient_lastDdlTime'='1564731656')
> {code}
>
> 2. Create partition's directory and put some data in it
>
> {code:java}
> hdfs dfs -mkdir
> hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
> hdfs dfs -put test.data
> hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
> {code}
>
> 3. Insert overwrite partition dayno=20190802
>
> {code:sql}
> INSERT OVERWRITE TABLE test PARTITION(dayno='20190802')
> SELECT "some value";
> {code}
>
> 4. We could see the test.data under partition directory is not deleted.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-22077) Inserting overwrite partitions clause does not clean directories while partitions' info is not stored in metadata
[
https://issues.apache.org/jira/browse/HIVE-22077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17114419#comment-17114419
]
Hive QA commented on HIVE-22077:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12978761/HIVE-22077.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22552/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22552/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22552/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-22 21:47:20.469
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22552/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-22 21:47:20.472
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at e649562 HIVE-22066: Upgrade Apache parent POM to version 21
(David Mollitor, reviewed by Ashutosh Chauhan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at e649562 HIVE-22066: Upgrade Apache parent POM to version 21
(David Mollitor, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-22 21:47:21.390
+ rm -rf ../yetus_PreCommit-HIVE-Build-22552
+ mkdir ../yetus_PreCommit-HIVE-Build-22552
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22552
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22552/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: does not
exist in index
Trying to apply the patch with -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:2181
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:2181
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java'
with conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-22552
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12978761 - PreCommit-HIVE-Build
> Inserting overwrite partitions clause does not clean directories while
> partitions' info is not stored in metadata
> -
>
> Key: HIVE-22077
> URL: https://issues.apache.org/jira/browse/HIVE-22077
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.1, 4.0.0, 2.3.4
>Reporter: Hui An
>Assignee: Hui An
>Priority: Major
> Attachments: HIVE-22077.patch.1
>
>
> Inserting overwrite static partitions may not clean related HDFS location if
> partitions' info is not stored in metadata.
> Steps to reproduce this issue :
>
> 1. Create a managed table :
>
> {code:sql}
> CREATE TABLE `test`(
>`id` string)
> PARTITIONED BY (
>
[jira] [Commented] (HIVE-22077) Inserting overwrite partitions clause does not clean directories while partitions' info is not stored in metadata
[
https://issues.apache.org/jira/browse/HIVE-22077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113829#comment-17113829
]
Jeffrey(Xilang) Yan commented on HIVE-22077:
We meet exactly same issue on production. Insert overwrite sql failed due to
hive metastore lock, retry the sql doesn't remove old data which make many many
duplicate data left in hdfs. It is a nightmare now, we have to find all
partition which have duplicate data.
Could someone help to revew this patch?
[~kgyrtkirk] [~jcamachorodriguez]
> Inserting overwrite partitions clause does not clean directories while
> partitions' info is not stored in metadata
> -
>
> Key: HIVE-22077
> URL: https://issues.apache.org/jira/browse/HIVE-22077
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.1, 4.0.0, 2.3.4
>Reporter: Hui An
>Assignee: Hui An
>Priority: Major
> Attachments: HIVE-22077.patch.1
>
>
> Inserting overwrite static partitions may not clean related HDFS location if
> partitions' info is not stored in metadata.
> Steps to reproduce this issue :
>
> 1. Create a managed table :
>
> {code:sql}
> CREATE TABLE `test`(
>`id` string)
> PARTITIONED BY (
>`dayno` string)
> ROW FORMAT SERDE
>'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>'hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test'
> TBLPROPERTIES (
>'transient_lastDdlTime'='1564731656')
> {code}
>
> 2. Create partition's directory and put some data in it
>
> {code:java}
> hdfs dfs -mkdir
> hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
> hdfs dfs -put test.data
> hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
> {code}
>
> 3. Insert overwrite partition dayno=20190802
>
> {code:sql}
> INSERT OVERWRITE TABLE test PARTITION(dayno='20190802')
> SELECT "some value";
> {code}
>
> 4. We could see the test.data under partition directory is not deleted.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22077) Inserting overwrite partitions clause does not clean directories while partitions' info is not stored in metadata
[
https://issues.apache.org/jira/browse/HIVE-22077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921275#comment-16921275
]
Hui An commented on HIVE-22077:
---
[~kgyrtkirk]Could you please review this patch?
> Inserting overwrite partitions clause does not clean directories while
> partitions' info is not stored in metadata
> -
>
> Key: HIVE-22077
> URL: https://issues.apache.org/jira/browse/HIVE-22077
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.1, 4.0.0, 2.3.4
>Reporter: Hui An
>Assignee: Hui An
>Priority: Major
> Attachments: HIVE-22077.patch.1
>
>
> Inserting overwrite static partitions may not clean related HDFS location if
> partitions' info is not stored in metadata.
> Steps to reproduce this issue :
>
> 1. Create a managed table :
>
> {code:sql}
> CREATE TABLE `test`(
>`id` string)
> PARTITIONED BY (
>`dayno` string)
> ROW FORMAT SERDE
>'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>'hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test'
> TBLPROPERTIES (
>'transient_lastDdlTime'='1564731656')
> {code}
>
> 2. Create partition's directory and put some data in it
>
> {code:java}
> hdfs dfs -mkdir
> hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
> hdfs dfs -put test.data
> hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
> {code}
>
> 3. Insert overwrite partition dayno=20190802
>
> {code:sql}
> INSERT OVERWRITE TABLE test PARTITION(dayno='20190802')
> SELECT "some value";
> {code}
>
> 4. We could see the test.data under partition directory is not deleted.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22077) Inserting overwrite partitions clause does not clean directories while partitions' info is not stored in metadata
[
https://issues.apache.org/jira/browse/HIVE-22077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918350#comment-16918350
]
Hive QA commented on HIVE-22077:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12978761/HIVE-22077.patch.1
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16745 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18413/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18413/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18413/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12978761 - PreCommit-HIVE-Build
> Inserting overwrite partitions clause does not clean directories while
> partitions' info is not stored in metadata
> -
>
> Key: HIVE-22077
> URL: https://issues.apache.org/jira/browse/HIVE-22077
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.1, 4.0.0, 2.3.4
>Reporter: Hui An
>Assignee: Hui An
>Priority: Major
> Attachments: HIVE-22077.patch.1
>
>
> Inserting overwrite static partitions may not clean related HDFS location if
> partitions' info is not stored in metadata.
> Steps to reproduce this issue :
>
> 1. Create a managed table :
>
> {code:sql}
> CREATE TABLE `test`(
>`id` string)
> PARTITIONED BY (
>`dayno` string)
> ROW FORMAT SERDE
>'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>'hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test'
> TBLPROPERTIES (
>'transient_lastDdlTime'='1564731656')
> {code}
>
> 2. Create partition's directory and put some data in it
>
> {code:java}
> hdfs dfs -mkdir
> hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
> hdfs dfs -put test.data
> hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
> {code}
>
> 3. Insert overwrite partition dayno=20190802
>
> {code:sql}
> INSERT OVERWRITE TABLE test PARTITION(dayno='20190802')
> SELECT "some value";
> {code}
>
> 4. We could see the test.data under partition directory is not deleted.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22077) Inserting overwrite partitions clause does not clean directories while partitions' info is not stored in metadata
[
https://issues.apache.org/jira/browse/HIVE-22077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918328#comment-16918328
]
Hive QA commented on HIVE-22077:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 2248 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18413/dev-support/hive-personality.sh
|
| git revision | master / d26516e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18413/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Inserting overwrite partitions clause does not clean directories while
> partitions' info is not stored in metadata
> -
>
> Key: HIVE-22077
> URL: https://issues.apache.org/jira/browse/HIVE-22077
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.1, 4.0.0, 2.3.4
>Reporter: Hui An
>Assignee: Hui An
>Priority: Major
> Attachments: HIVE-22077.patch.1
>
>
> Inserting overwrite static partitions may not clean related HDFS location if
> partitions' info is not stored in metadata.
> Steps to reproduce this issue :
>
> 1. Create a managed table :
>
> {code:sql}
> CREATE TABLE `test`(
>`id` string)
> PARTITIONED BY (
>`dayno` string)
> ROW FORMAT SERDE
>'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>'hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test'
> TBLPROPERTIES (
>'transient_lastDdlTime'='1564731656')
> {code}
>
[jira] [Commented] (HIVE-22077) Inserting overwrite partitions clause does not clean directories while partitions' info is not stored in metadata
[
https://issues.apache.org/jira/browse/HIVE-22077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917513#comment-16917513
]
Hui An commented on HIVE-22077:
---
In a compromising way, first check if newPartPath is empty with filter, if not,
then clean it.
> Inserting overwrite partitions clause does not clean directories while
> partitions' info is not stored in metadata
> -
>
> Key: HIVE-22077
> URL: https://issues.apache.org/jira/browse/HIVE-22077
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.1, 4.0.0, 2.3.4
>Reporter: Hui An
>Assignee: Hui An
>Priority: Major
> Attachments: HIVE-22077.patch.1
>
>
> Inserting overwrite static partitions may not clean related HDFS location if
> partitions' info is not stored in metadata.
> Steps to reproduce this issue :
>
> 1. Create a managed table :
>
> {code:sql}
> CREATE TABLE `test`(
>`id` string)
> PARTITIONED BY (
>`dayno` string)
> ROW FORMAT SERDE
>'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>'hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test'
> TBLPROPERTIES (
>'transient_lastDdlTime'='1564731656')
> {code}
>
> 2. Create partition's directory and put some data in it
>
> {code:java}
> hdfs dfs -mkdir
> hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
> hdfs dfs -put test.data
> hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
> {code}
>
> 3. Insert overwrite partition dayno=20190802
>
> {code:sql}
> INSERT OVERWRITE TABLE test PARTITION(dayno='20190802')
> SELECT "some value";
> {code}
>
> 4. We could see the test.data under partition directory is not deleted.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22077) Inserting overwrite partitions clause does not clean directories while partitions' info is not stored in metadata
[
https://issues.apache.org/jira/browse/HIVE-22077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898685#comment-16898685
]
Hui An commented on HIVE-22077:
---
This issue is caused by method loadPartitionInternal of Hive.java
{code:java}
Path oldPartPath = (oldPart != null) ? oldPart.getDataLocation() : null;
Path newPartPath = null;
if (inheritLocation) {
newPartPath = genPartPathFromTable(tbl, partSpec, tblDataLocationPath);
if(oldPart != null) {
/*
* If we are moving the partition across filesystem boundaries
* inherit from the table properties. Otherwise (same filesystem) use the
* original partition location.
*
* See: HIVE-1707 and HIVE-2117 for background
*/
FileSystem oldPartPathFS = oldPartPath.getFileSystem(getConf());
FileSystem loadPathFS = loadPath.getFileSystem(getConf());
if (FileUtils.equalsFileSystem(oldPartPathFS,loadPathFS)) {
newPartPath = oldPartPath;
}
}
} else {
newPartPath = oldPartPath == null
? genPartPathFromTable(tbl, partSpec, tblDataLocationPath) : oldPartPath;
}
{code}
Actually, oldPart is null does not mean oldPartPath is not exists in HDFS, but
it just set oldPartPath is null, and give null value to following method
replaceFiles.
> Inserting overwrite partitions clause does not clean directories while
> partitions' info is not stored in metadata
> -
>
> Key: HIVE-22077
> URL: https://issues.apache.org/jira/browse/HIVE-22077
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.1, 4.0.0, 2.3.4
>Reporter: Hui An
>Assignee: Hui An
>Priority: Major
>
> Inserting overwrite static partitions may not clean related HDFS location if
> partitions' info is not stored in metadata.
> Steps to Reproduce this issue :
>
> 1. Create a managed table :
>
> {code:sql}
> CREATE TABLE `test`(
>`id` string)
> PARTITIONED BY (
>`dayno` string)
> ROW FORMAT SERDE
>'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION |
>'hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test'
> TBLPROPERTIES (
>'transient_lastDdlTime'='1564731656')
> {code}
>
> 2. Create partition's directory and put some data under it
>
> {code:java}
> hdfs dfs -mkdir
> hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
> hdfs dfs -put test.data
> hdfs://test-dev-hdfs/user/hive/warehouse/test.db/test/dayno=20190802
> {code}
>
> 3. Insert overwrite partition dayno=20190802
>
> {code:sql}
> INSERT OVERWRITE TABLE test PARTITION(dayno='20190802')
> SELECT 1;
> {code}
>
> 4. We could see the test.data under partition directory is not deleted.
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
