Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17878 )
Change subject: IMPALA-10934: Enable table definition over a single file ...................................................................... Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/17878/2/be/src/runtime/io/disk-io-mgr.cc File be/src/runtime/io/disk-io-mgr.cc: http://gerrit.cloudera.org:8080/#/c/17878/2/be/src/runtime/io/disk-io-mgr.cc@142 PS2, Line 142: // The maximum number of SFS I/O threads. : DEFINE_int32(num_sfs_io_threads, 16, "Number of SFS I/O threads"); > As a second thought on this, if we are ok with turning off file handle cach Agree that turning off file handle caching for the SFS case should not hurt much in terms of performance if indeed there is any problem with the file handle caching for SFS+S3 (will await testing). And yes, ensuring that the data cache works as expected would be useful/important. Query profile metrics for the data cache can be used to manually verify. On the previous point of potentially doing the single file processing through the frontend, based on offline discussion with Zoltan, this was considered but the backend alternative was more compelling because it allows - with relatively small changes - the same file system access mechanism to work for both Impala and Hive. This is kind of implied in the design doc attached to the corresponding Hive JIRA here: https://issues.apache.org/jira/browse/HIVE-25569. Zoltan, it would be good if the drawback of the frontend-only approach is more explicitly described in the document. Also, by representing the single file paths in a standard URI form, Ranger authorization gets transparently supported with minimal changes. -- To view, visit http://gerrit.cloudera.org:8080/17878 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32be936243aa4c8320f5d06d2b7fbf98822f82e7 Gerrit-Change-Number: 17878 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Comment-Date: Sun, 31 Oct 2021 23:44:45 +0000 Gerrit-HasComments: Yes
