DadanielZ commented on a change in pull request #768: HADOOP-16269. ABFS: add
listFileStatus with StartFrom.
URL: https://github.com/apache/hadoop/pull/768#discussion_r279066413
##########
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -583,6 +620,69 @@ public FileStatus getFileStatus(final Path path) throws
IOException {
return fileStatuses.toArray(new FileStatus[fileStatuses.size()]);
}
+ // generate continuation token for xns account
+ private String generateContinuationTokenForXns(final String firstEntryName) {
+ if (firstEntryName == null || firstEntryName.isEmpty() ||
firstEntryName.startsWith(AbfsHttpConstants.ROOT_PATH)) {
+ throw new IllegalArgumentException("startFrom must be a dir/file name
and it can not be a full path");
+ }
+
+ StringBuilder sb = new StringBuilder();
+ sb.append(firstEntryName).append("#$").append("0");
+
+ CRC64 crc64 = new CRC64();
+ StringBuilder token = new StringBuilder();
+ token.append(crc64.compute(sb.toString().getBytes(StandardCharsets.UTF_8)))
+ .append(SINGLE_WHITE_SPACE)
+ .append("0")
+ .append(SINGLE_WHITE_SPACE)
+ .append(firstEntryName);
+
+ return Base64.encode(token.toString().getBytes(StandardCharsets.UTF_8));
+ }
+
+ // generate continuation token for non-xns account
+ private String generateContinuationTokenForNonXns(final String path, final
String firstEntryName) {
+ if (firstEntryName == null || firstEntryName.isEmpty() ||
firstEntryName.startsWith(AbfsHttpConstants.ROOT_PATH)) {
+ throw new IllegalArgumentException("startFrom must be a dir/file name
and it can not be a full path");
+ }
+
+ // Notice: non-xns continuation token requires full path (first "/" is not
included) for startFrom
+ final String startFrom = (path.isEmpty() || path.equals(ROOT_PATH))
+ ? firstEntryName
+ : path + ROOT_PATH + firstEntryName;
+
+ SimpleDateFormat simpleDateFormat = new
SimpleDateFormat(TOKEN_DATE_PATTERN, Locale.US);
+ String date = simpleDateFormat.format(new Date());
+
+ StringBuilder sb = new StringBuilder(path.length() + startFrom.length() +
date.length() + 24);
+ sb.append(String.format("%06d",
path.length())).append(CHAR_EXCLAMATION_POINT)
Review comment:
you are correct. Included "!" in the string format.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
