steveloughran commented on a change in pull request #768: HADOOP-16269. ABFS:
add listFileStatus with StartFrom.
URL: https://github.com/apache/hadoop/pull/768#discussion_r278999013
##########
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -583,6 +620,69 @@ public FileStatus getFileStatus(final Path path) throws
IOException {
return fileStatuses.toArray(new FileStatus[fileStatuses.size()]);
}
+ // generate continuation token for xns account
+ private String generateContinuationTokenForXns(final String firstEntryName) {
+ if (firstEntryName == null || firstEntryName.isEmpty() ||
firstEntryName.startsWith(AbfsHttpConstants.ROOT_PATH)) {
+ throw new IllegalArgumentException("startFrom must be a dir/file name
and it can not be a full path");
+ }
+
+ StringBuilder sb = new StringBuilder();
+ sb.append(firstEntryName).append("#$").append("0");
+
+ CRC64 crc64 = new CRC64();
+ StringBuilder token = new StringBuilder();
+ token.append(crc64.compute(sb.toString().getBytes(StandardCharsets.UTF_8)))
+ .append(SINGLE_WHITE_SPACE)
+ .append("0")
+ .append(SINGLE_WHITE_SPACE)
+ .append(firstEntryName);
+
+ return Base64.encode(token.toString().getBytes(StandardCharsets.UTF_8));
+ }
+
+ // generate continuation token for non-xns account
+ private String generateContinuationTokenForNonXns(final String path, final
String firstEntryName) {
+ if (firstEntryName == null || firstEntryName.isEmpty() ||
firstEntryName.startsWith(AbfsHttpConstants.ROOT_PATH)) {
+ throw new IllegalArgumentException("startFrom must be a dir/file name
and it can not be a full path");
+ }
+
+ // Notice: non-xns continuation token requires full path (first "/" is not
included) for startFrom
+ final String startFrom = (path.isEmpty() || path.equals(ROOT_PATH))
+ ? firstEntryName
+ : path + ROOT_PATH + firstEntryName;
+
+ SimpleDateFormat simpleDateFormat = new
SimpleDateFormat(TOKEN_DATE_PATTERN, Locale.US);
+ String date = simpleDateFormat.format(new Date());
+
+ StringBuilder sb = new StringBuilder(path.length() + startFrom.length() +
date.length() + 24);
+ sb.append(String.format("%06d",
path.length())).append(CHAR_EXCLAMATION_POINT)
Review comment:
any reason not to use a single large String.format? I'm just wondering which
version is the most clear as to what is being constructed. If the
CHAR_EXCLAMATION_POINT was just "!" inlined the case for a single String.format
would be clear. Here: not so sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
