mchades commented on code in PR #7215:
URL: https://github.com/apache/gravitino/pull/7215#discussion_r2103924175
##########
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java:
##########
@@ -237,6 +240,44 @@ public Fileset loadFileset(NameIdentifier ident) throws
NoSuchFilesetException {
});
}
+ @Override
+ public FileInfo[] listFiles(NameIdentifier ident, String locationName,
String subPath) throws NoSuchFilesetException {
+ try {
Review Comment:
if `disableFSOps==true`, the `listFiles` operation is forbidden
##########
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java:
##########
@@ -237,6 +240,44 @@ public Fileset loadFileset(NameIdentifier ident) throws
NoSuchFilesetException {
});
}
+ @Override
+ public FileInfo[] listFiles(NameIdentifier ident, String locationName,
String subPath) throws NoSuchFilesetException {
+ try {
+ if (!store.exists(ident, Entity.EntityType.FILESET)) {
+ throw new NoSuchFilesetException(FILESET_DOES_NOT_EXIST_MSG, ident);
+ }
+
+ String actualPath = getFileLocation(ident, subPath, locationName);
+
+ Path hadoopPath = new Path(actualPath);
+ Path formalizedPath = formalizePath(hadoopPath, conf);
+
+ FileSystem fs = getFileSystem(formalizedPath, conf);
+ if (!fs.exists(formalizedPath)) {
+ return new FileInfo[0];
+ }
+
+ FileStatus[] fileStatuses = fs.listStatus(formalizedPath);
+
+ FileInfo[] fileInfos = new FileInfo[fileStatuses.length];
+ for (int i = 0; i < fileStatuses.length; i++) {
+ FileStatus status = fileStatuses[i];
+ fileInfos[i] = FileInfoDTO.builder()
+ .name(status.getPath().getName())
+ .isDir(status.isDirectory())
+ .size(status.getLen())
+ .lastModified(status.getModificationTime())
+ .path(status.getPath().toString())
+ .build();
+ }
+
+ return fileInfos;
+
+ } catch (IOException e) {
Review Comment:
You should narrow the range of the try-catch block to isolate the error
message more precisely.
##########
server/src/main/java/org/apache/gravitino/server/web/rest/FilesetOperations.java:
##########
@@ -182,6 +185,34 @@ public Response loadFileset(
}
}
+ @GET
+ @Path("{fileset}/files")
+ @Produces("application/vnd.gravitino.v1+json")
+ @Timed(name = "list-fileset-files." + MetricNames.HTTP_PROCESS_DURATION,
absolute = true)
+ @ResponseMetered(name = "list-fileset-files", absolute = true)
+ public Response listFiles(
+ @PathParam("metalake") String metalake,
+ @PathParam("catalog") String catalog,
+ @PathParam("schema") String schema,
+ @PathParam("fileset") String fileset,
+ @QueryParam("subPath") @DefaultValue("") String subPath,
+ @QueryParam("locationName") String locationName) {
+ LOG.info("Received list files request: {}.{}.{}.{}, subPath: {},
locationName:{}", metalake, catalog, schema, fileset, subPath, locationName);
+ try {
+ return Utils.doAs(
+ httpRequest,
+ () -> {
+ NameIdentifier ident = NameIdentifierUtil.ofFileset(metalake,
catalog, schema, fileset);
+ FileInfo[] files = dispatcher.listFiles(ident, locationName,
subPath);
Review Comment:
The subPath may be encoded, so you should use the decoded subPath.
##########
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java:
##########
@@ -237,6 +240,44 @@ public Fileset loadFileset(NameIdentifier ident) throws
NoSuchFilesetException {
});
}
+ @Override
+ public FileInfo[] listFiles(NameIdentifier ident, String locationName,
String subPath) throws NoSuchFilesetException {
+ try {
+ if (!store.exists(ident, Entity.EntityType.FILESET)) {
+ throw new NoSuchFilesetException(FILESET_DOES_NOT_EXIST_MSG, ident);
+ }
+
+ String actualPath = getFileLocation(ident, subPath, locationName);
+
+ Path hadoopPath = new Path(actualPath);
+ Path formalizedPath = formalizePath(hadoopPath, conf);
Review Comment:
It can be compressed into one line, and hadoopPath is not a good variable
name.
##########
server/src/main/java/org/apache/gravitino/server/web/rest/FilesetOperations.java:
##########
@@ -182,6 +185,34 @@ public Response loadFileset(
}
}
+ @GET
+ @Path("{fileset}/files")
+ @Produces("application/vnd.gravitino.v1+json")
+ @Timed(name = "list-fileset-files." + MetricNames.HTTP_PROCESS_DURATION,
absolute = true)
+ @ResponseMetered(name = "list-fileset-files", absolute = true)
+ public Response listFiles(
+ @PathParam("metalake") String metalake,
+ @PathParam("catalog") String catalog,
+ @PathParam("schema") String schema,
+ @PathParam("fileset") String fileset,
+ @QueryParam("subPath") @DefaultValue("") String subPath,
Review Comment:
the default value of subPath should be "/"
##########
catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java:
##########
@@ -237,6 +240,44 @@ public Fileset loadFileset(NameIdentifier ident) throws
NoSuchFilesetException {
});
}
+ @Override
+ public FileInfo[] listFiles(NameIdentifier ident, String locationName,
String subPath) throws NoSuchFilesetException {
+ try {
+ if (!store.exists(ident, Entity.EntityType.FILESET)) {
+ throw new NoSuchFilesetException(FILESET_DOES_NOT_EXIST_MSG, ident);
+ }
Review Comment:
The check is redundant since `getFileLocation(ident, subPath, locationName)`
accomplishes the same thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
