Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/12569 )
Change subject: IMPALA-7140 (part 5): support fetching file info for FS tables
......................................................................
Patch Set 1:
(1 comment)
Only one conflicts in fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
440 public static List<FileDescriptor> createFileDescriptors(
441 FileSystem fs,
442 RemoteIterator<LocatedFileStatus> fileStatusIter,
443 ListMap<TNetworkAddress> hostIndex,
444 FileMetadataLoadStats loadStats) throws IOException {
445 boolean supportsBlocks = FileSystemUtil.supportsStorageIds(fs);
446 Reference<Long> numUnknownDiskIds = new
Reference<Long>(Long.valueOf(0));
447 List<FileDescriptor> newFileDescs = Lists.newArrayList();
448 while (fileStatusIter.hasNext()) {
449 LocatedFileStatus fileStatus = fileStatusIter.next();
450 if (!FileSystemUtil.isValidDataFile(fileStatus)) {
451 ++loadStats.hiddenFiles;
452 continue;
453 }
454 FileDescriptor fd;
455 if (supportsBlocks) {
456 fd = FileDescriptor.create(fileStatus,
fileStatus.getBlockLocations(), fs,
457 <<<<<<< HEAD
458 hostIndex_, numUnknownDiskIds);
459 =======
460 hostIndex, HdfsShim.isErasureCoded(fileStatus),
numUnknownDiskIds);
461 >>>>>>> 64e6719... IMPALA-7140 (part 5): support fetching file info for FS
tables
462 } else {
463 fd = FileDescriptor.createWithNoBlocks(fileStatus);
464 }
465 newFileDescs.add(fd);
466 ++loadStats.loadedFiles;
467 }
468 loadStats.unknownDiskIds += numUnknownDiskIds.getRef();
469 return newFileDescs;
470 }
http://gerrit.cloudera.org:8080/#/c/12569/1/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:
http://gerrit.cloudera.org:8080/#/c/12569/1/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@134
PS1, Line 134: // TODO(todd): this is slow because it requires loading all
partitions. Remove if possible.
> line too long (95 > 90)
Ignore this to avoid conflicts in later cherrypicks
--
To view, visit http://gerrit.cloudera.org:8080/12569
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I42d67ab754872fad094c7dacdd2e1182de1bf3e8
Gerrit-Change-Number: 12569
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Fredy Wijaya <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Vuk Ercegovac <[email protected]>
Gerrit-Comment-Date: Sun, 24 Feb 2019 00:11:11 +0000
Gerrit-HasComments: Yes
