saihemanth-cloudera commented on code in PR #5851:
URL: https://github.com/apache/hive/pull/5851#discussion_r2612490269
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##########
@@ -2864,6 +2865,58 @@ public List<Void> run(List<String> input) throws
MetaException {
});
}
+ /**
+ * Drop partitions by the given table id, and return the locations that
outside of the baseLocationToNotShow
+ * @param tableId the table id
+ * @param baseLocationToNotShow the parent directory
+ * @return locations that outside of the parent directory
+ */
+ public List<String> dropAllPartitionsAndGetLocations(Long tableId, String
baseLocationToNotShow, StringBuffer message) throws MetaException {
+ String queryText =
+ "select " + PARTITIONS + ".\"PART_ID\", " + SDS + ".\"LOCATION\" from
" + PARTITIONS
+ + (baseLocationToNotShow != null ? (" join " + SDS + " on " +
PARTITIONS + ".\"SD_ID\" = " + SDS + ".\"SD_ID\" ") : "")
+ + " where \"TBL_ID\" = " + tableId;
+ List<Long> partIds = new ArrayList<>();
+ List<String> locations = new ArrayList<>();
+ final boolean doTrace = LOG.isDebugEnabled();
+ try (QueryWrapper query = new
QueryWrapper(pm.newQuery("javax.jdo.query.SQL", queryText))) {
+ long start = doTrace ? System.nanoTime() : 0;
+ List<Object> sqlResult = executeWithArray(query.getInnerQuery(), null,
queryText);
+ long queryTime = doTrace ? System.nanoTime() : 0;
+ MetastoreDirectSqlUtils.timingTrace(doTrace, queryText, start,
queryTime);
+ for (Object result : sqlResult) {
+
partIds.add(MetastoreDirectSqlUtils.extractSqlLong(((Object[])result)[0]));
+ String partitionLocation;
+ if (baseLocationToNotShow != null &&
+ (partitionLocation =
MetastoreDirectSqlUtils.extractSqlString(((Object[])result)[1])) != null &&
+ !FileUtils.isSubdirectory(baseLocationToNotShow,
partitionLocation)) {
+ locations.add(partitionLocation);
+ }
+ }
+ }
+
+ int batch = batchSize == NO_BATCHING ? 1 : (partIds.size() + batchSize) /
batchSize;
+ AtomicLong batchIdx = new AtomicLong(1);
+ AtomicLong timeSpent = new AtomicLong(0);
+ String prefix = "Dropping partitions, batch: ";
+ Batchable.runBatched(batchSize, partIds, new Batchable<Long, Void>() {
+ @Override
+ public List<Void> run(List<Long> input) throws Exception {
+ long start = System.currentTimeMillis();
+ message.setLength(0);
+
message.append(prefix).append(batchIdx.get()).append("/").append(batch);
+ if (batchIdx.get() > 1) {
+ long leftTime = (batch - batchIdx.get()) * timeSpent.get() /
batchIdx.get();
+ message.append(", time left: ").append(leftTime).append("ms");
Review Comment:
Should the 'message' be written to the log or to the output stream before
set to length 0 at L#2906?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
