[
https://issues.apache.org/jira/browse/HIVE-27072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Taraka Rama Rao Lethavadla updated HIVE-27072:
----------------------------------------------
Description:
There are couple of issues when partitions were corrupted or have additional
unwanted files that will intervene query execution and fail.
If we run query like "validate table table_name [partition(partition=a,..)]",
the output should list
* any unwanted files like empty/metadata files(like _SUCCESS etc)
* any unwanted folders not confirming to the partition naming convention like
test_folder where actual partition name looks like test=23
* Too many staging directories, if we find many then cleanup is not happening
properly after query execution
* any file permission related issues like table has one owner, partition has
another owner etc(Optional)
We have something like this in Impala [Invalidate metadata and Refresh
commands|https://impala.apache.org/docs/build/html/topics/impala_invalidate_metadata.html]
So we can have something similar to that functionality in Hive
was:
There are couple of issues when partitions were corrupted or have additional
unwanted files that will intervene query execution and fail.
If we run query like "validate table table_name [partition(partition=a,..)]",
the output should list
* any unwanted files like empty/metadata files(like _SUCCESS etc)
* any unwanted folders not confirming to the partition naming convention like
test_folder where actual partition name looks like test=23
* Too many staging directories, if we find many then cleanup is not happening
properly after query execution
* any file permission related issues like table has one owner, partition has
another owner etc(Optional)
> create an sql query to validate a given table for partitions and list out any
> discrepancies in files/folders, list out empty files etc
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-27072
> URL: https://issues.apache.org/jira/browse/HIVE-27072
> Project: Hive
> Issue Type: New Feature
> Components: HiveServer2
> Reporter: Taraka Rama Rao Lethavadla
> Priority: Major
>
> There are couple of issues when partitions were corrupted or have additional
> unwanted files that will intervene query execution and fail.
> If we run query like "validate table table_name [partition(partition=a,..)]",
> the output should list
> * any unwanted files like empty/metadata files(like _SUCCESS etc)
> * any unwanted folders not confirming to the partition naming convention
> like test_folder where actual partition name looks like test=23
> * Too many staging directories, if we find many then cleanup is not
> happening properly after query execution
> * any file permission related issues like table has one owner, partition has
> another owner etc(Optional)
> We have something like this in Impala [Invalidate metadata and Refresh
> commands|https://impala.apache.org/docs/build/html/topics/impala_invalidate_metadata.html]
> So we can have something similar to that functionality in Hive
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
