lirui-apache commented on a change in pull request #10381: [FLINK-14513][hive]
Implement listPartitionsByFilter to HiveCatalog
URL: https://github.com/apache/flink/pull/10381#discussion_r355102662
##########
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/HiveCatalog.java
##########
@@ -751,6 +753,34 @@ public void dropPartition(ObjectPath tablePath,
CatalogPartitionSpec partitionSp
}
}
+ public List<CatalogPartitionSpec> listPartitionsByFilter(ObjectPath
tablePath, List<Expression> expressions)
+ throws TableNotExistException,
TableNotPartitionedException, CatalogException {
+ Table hiveTable = getHiveTable(tablePath);
+ ensurePartitionedTable(tablePath, hiveTable);
+ List<String> partColNames =
getFieldNames(hiveTable.getPartitionKeys());
+ String filter = HiveTableUtil.makePartitionFilter(partColNames,
expressions);
+ if (filter == null) {
+ throw new UnsupportedOperationException(
+ "HiveCatalog is unable to handle the
partition filter expressions: " + expressions);
+ }
+ try {
+ PartitionSpecProxy.PartitionIterator partitions =
client.listPartitionSpecsByFilter(
+ tablePath.getDatabaseName(),
tablePath.getObjectName(), filter, (short) -1).getPartitionIterator();
+ List<CatalogPartitionSpec> res = new ArrayList<>();
+ while (partitions.hasNext()) {
+ Partition partition = partitions.next();
+ Map<String, String> spec = new HashMap<>();
+ for (int i = 0; i < partColNames.size(); i++) {
+ spec.put(partColNames.get(i),
partition.getValues().get(i));
+ }
+ res.add(new CatalogPartitionSpec(spec));
+ }
+ return res;
+ } catch (TException e) {
+ throw new UnsupportedOperationException("Failed to list
partition by filter from HMS", e);
Review comment:
It's possible the underlying DB doesn't support the filter, in which case we
would want to let planner fall back to list all partitions and filter by itself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
