[
https://issues.apache.org/jira/browse/HIVE-27072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17690994#comment-17690994
]
Srinivasu alla commented on HIVE-27072:
---------------------------------------
[~tarak271] , , can't most of these be achieved with MSCK REPAIR and ANALYZE
TABLE COMPUTE STATS ..
How is _SUCCESS an unwanted file ?
Though your usecase may seem valid for managed tables, we shouldn't be
controlling the External tables data/info from hive .
Happy to get the inputs and work on it.
> create an sql query to validate a given table for partitions and list out any
> discrepancies in files/folders, list out empty files etc
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-27072
> URL: https://issues.apache.org/jira/browse/HIVE-27072
> Project: Hive
> Issue Type: New Feature
> Components: HiveServer2
> Reporter: Taraka Rama Rao Lethavadla
> Priority: Major
>
> There are couple of issues when partitions were corrupted or have additional
> unwanted files that will intervene query execution and fail.
> If we run query like "validate table table_name [partition(partition=a,..)]",
> the output should list
> * any unwanted files like empty/metadata files(like _SUCCESS etc)
> * any unwanted folders not confirming to the partition naming convention
> like test_folder where actual partition name looks like test=23
> * Too many staging directories, if we find many then cleanup is not
> happening properly after query execution
> * any file permission related issues like table has one owner, partition has
> another owner etc(Optional)
> We have something like this in Impala [Invalidate metadata and Refresh
> commands|https://impala.apache.org/docs/build/html/topics/impala_invalidate_metadata.html]
> So we can have something similar to that functionality in Hive
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
