lirui-apache commented on a change in pull request #10381: [FLINK-14513][hive]
Implement listPartitionsByFilter to HiveCatalog
URL: https://github.com/apache/flink/pull/10381#discussion_r355106555
##########
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/HiveCatalog.java
##########
@@ -751,6 +753,34 @@ public void dropPartition(ObjectPath tablePath,
CatalogPartitionSpec partitionSp
}
}
+ public List<CatalogPartitionSpec> listPartitionsByFilter(ObjectPath
tablePath, List<Expression> expressions)
+ throws TableNotExistException,
TableNotPartitionedException, CatalogException {
+ Table hiveTable = getHiveTable(tablePath);
+ ensurePartitionedTable(tablePath, hiveTable);
+ List<String> partColNames =
getFieldNames(hiveTable.getPartitionKeys());
+ String filter = HiveTableUtil.makePartitionFilter(partColNames,
expressions);
+ if (filter == null) {
+ throw new UnsupportedOperationException(
+ "HiveCatalog is unable to handle the
partition filter expressions: " + expressions);
+ }
+ try {
+ PartitionSpecProxy.PartitionIterator partitions =
client.listPartitionSpecsByFilter(
+ tablePath.getDatabaseName(),
tablePath.getObjectName(), filter, (short) -1).getPartitionIterator();
+ List<CatalogPartitionSpec> res = new ArrayList<>();
+ while (partitions.hasNext()) {
+ Partition partition = partitions.next();
+ Map<String, String> spec = new HashMap<>();
+ for (int i = 0; i < partColNames.size(); i++) {
+ spec.put(partColNames.get(i),
partition.getValues().get(i));
+ }
+ res.add(new CatalogPartitionSpec(spec));
+ }
+ return res;
+ } catch (TException e) {
+ throw new UnsupportedOperationException("Failed to list
partition by filter from HMS", e);
Review comment:
The exception is defined in the Catalog API contract:
```
* <p>If catalog does not support this interface at present, throw an
{@link UnsupportedOperationException}
* directly. If the catalog does not have a valid filter, throw the
{@link UnsupportedOperationException}
* directly. Planner will fallback to get all partitions and filter by
itself.
```
I'll add the expressions to error msg.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
