saihemanth-cloudera commented on code in PR #6887:
URL: https://github.com/apache/hive/pull/6887#discussion_r4449744718
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/ColumnDeduplicator.java:
##########
@@ -119,6 +132,39 @@ MetaToolObjectStore.DedupColumnsResult run(String
catalogFilter, String dbFilter
return result;
}
+ public TableInfo getTable(long tableId) {
+ final MTable table;
+ try {
+ table = pm.getObjectById(MTable.class, new MTable.PK(tableId));
+ } catch (JDOObjectNotFoundException e) {
+ // This table might have been dropped since the beginning
+ return null;
+ }
+ Map<String, String> params = table.getParameters() == null ?
+ new HashMap<>() : table.getParameters();
+ if (clearMarker) {
+ if (!params.containsKey(COLUMN_DEDUPLICATOR_MARKER)) {
+ return null;
+ }
+ params.remove(COLUMN_DEDUPLICATOR_MARKER);
+ table.setParameters(params);
+ return null;
+ }
+ if (params.containsKey(COLUMN_DEDUPLICATOR_MARKER)) {
+ return null;
+ }
+
+ if (!isDryRun) {
+ params.put(COLUMN_DEDUPLICATOR_MARKER, "" + System.currentTimeMillis() /
1000);
Review Comment:
I think it would be better to persist the marker only after validation
succeeds, otherwise failed validation makes the table permanently skipped.
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/metatool/MetaToolTaskDedupColumns.java:
##########
@@ -89,9 +90,16 @@ void execute() {
if (result != null) {
printSummary(result, isDryRun, isVerbose);
if (result.getException() != null) {
- if (result.getException().getCause() instanceof DeadlineException
deadline) {
- // Just print the timeout message
- System.err.println(deadline.getMessage());
+ DeadlineException deadline = findDeadlineCause(result.getException());
+ if (deadline != null) {
+ long timeoutSeconds = Math.max(1, timeoutMs / 1000);
+ System.err.println("HiveMetaTool -dedupColumns timed out after " +
timeoutSeconds
+ + " second(s). The configured limit is -timeout <seconds> or "
+ + MetastoreConf.ConfVars.DEDUP_COLUMNS_TIMEOUT.getVarname() + "
(default in minutes).");
+ System.err.println("Increase the limit with -timeout <seconds> or
set "
+ + MetastoreConf.ConfVars.DEDUP_COLUMNS_TIMEOUT.getVarname()
+ + " to a larger value (minutes), then re-run. Tables already
processed can be skipped if marked "
+ + "with " + ColumnDeduplicator.COLUMN_DEDUPLICATOR_MARKER + ".");
return;
Review Comment:
A timeout should return a nonzero exit status instead of completing normally
right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
