Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

nsivabalan commented on code in PR #17788:
URL: https://github.com/apache/hudi/pull/17788#discussion_r2729418821


##
hudi-common/src/main/java/org/apache/hudi/common/util/ExternalFilePathUtil.java:
##
@@ -43,4 +92,109 @@ public static String 
appendCommitTimeAndExternalFileMarker(String filePath, Stri
   public static boolean isExternallyCreatedFile(String fileName) {
 return fileName.endsWith(EXTERNAL_FILE_SUFFIX);
   }
+
+  /**
+   * Extracts the file group prefix from an external file name.
+   * @param fileName The external file name
+   * @return Option containing the decoded file group prefix, or empty if not 
present
+   */
+  private static Option getExternalFileGroupPrefix(String fileName) {
+if (!isExternallyCreatedFile(fileName)) {
+  return Option.empty();
+}
+int prefixMarkerIndex = fileName.indexOf(FILE_GROUP_PREFIX_MARKER);
+if (prefixMarkerIndex == -1) {
+  return Option.empty();
+}
+int start = prefixMarkerIndex + FILE_GROUP_PREFIX_MARKER.length();
+int end = fileName.lastIndexOf(EXTERNAL_FILE_SUFFIX);
+return 
Option.of(PartitionPathEncodeUtils.unescapePathName(fileName.substring(start, 
end)));
+  }
+
+  /**
+   * Extracts the original file name from an external file name (without 
commit time and markers).
+   * For example, "data.parquet_123_hudiext" returns "data.parquet"
+   * And "data.parquet_123_fg%3Dbucket-0_hudiext" also returns "data.parquet"
+   *
+   * @param fileName The external file name
+   * @return The original file name
+   */
+  private static String getOriginalFileName(String fileName) {
+if (!isExternallyCreatedFile(fileName)) {
+  return fileName;
+}
+int prefixMarkerIndex = fileName.indexOf(FILE_GROUP_PREFIX_MARKER);
+int markerEnd = prefixMarkerIndex != -1
+? prefixMarkerIndex
+: fileName.lastIndexOf(EXTERNAL_FILE_SUFFIX);
+int commitTimeStart = fileName.lastIndexOf('_', markerEnd - 1);
+return fileName.substring(0, commitTimeStart);
+  }
+
+  /**
+   * Adjusts the parent path for external files with file group prefix.
+   * For files with file group prefix, the prefix represents subdirectories 
within the partition,
+   * so we need to remove the prefix portion to get the actual partition path.
+   * Supports arbitrary nesting depths (e.g., "bucket-0/subdir1/subdir2").
+   *
+   * @param parent the parent path
+   * @param fileName the file name to check
+   * @return the adjusted parent path
+   */
+  public static StoragePath getFullPathOfPartition(StoragePath parent, String 
fileName) {
+return getExternalFileGroupPrefix(fileName)
+.map(prefix -> new StoragePath(parent.toString().substring(0, 
parent.toString().length() - prefix.length() - 1)))
+.orElse(parent);
+  }
+
+  /**
+   * Parses external file names to extract fileId and commit time.
+   * Handles both formats:
+   *   - With prefix: originalName_commitTime_fg%3D_hudiext -> fileId 
= prefix/originalName
+   *   - Without prefix: originalName_commitTime_hudiext -> fileId = 
originalName
+   *
+   * @param fileName The external file name to parse
+   * @return String array of size 2: [fileId, commitTime]
+   */
+  public static String[] parseFileIdAndCommitTimeFromExternalFile(String 
fileName) {
+String[] values = new String[2];
+// Extract original file name
+String originalName = getOriginalFileName(fileName);
+// Extract file group prefix (if present)
+Option prefix = getExternalFileGroupPrefix(fileName);

Review Comment:
   sg



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

nsivabalan merged PR #17788:
URL: https://github.com/apache/hudi/pull/17788


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

vinishjail97 commented on code in PR #17788:
URL: https://github.com/apache/hudi/pull/17788#discussion_r2729026048


##
hudi-common/src/main/java/org/apache/hudi/common/util/ExternalFilePathUtil.java:
##
@@ -43,4 +92,109 @@ public static String 
appendCommitTimeAndExternalFileMarker(String filePath, Stri
   public static boolean isExternallyCreatedFile(String fileName) {
 return fileName.endsWith(EXTERNAL_FILE_SUFFIX);
   }
+
+  /**
+   * Extracts the file group prefix from an external file name.
+   * @param fileName The external file name
+   * @return Option containing the decoded file group prefix, or empty if not 
present
+   */
+  private static Option getExternalFileGroupPrefix(String fileName) {
+if (!isExternallyCreatedFile(fileName)) {
+  return Option.empty();
+}
+int prefixMarkerIndex = fileName.indexOf(FILE_GROUP_PREFIX_MARKER);
+if (prefixMarkerIndex == -1) {
+  return Option.empty();
+}
+int start = prefixMarkerIndex + FILE_GROUP_PREFIX_MARKER.length();
+int end = fileName.lastIndexOf(EXTERNAL_FILE_SUFFIX);
+return 
Option.of(PartitionPathEncodeUtils.unescapePathName(fileName.substring(start, 
end)));
+  }
+
+  /**
+   * Extracts the original file name from an external file name (without 
commit time and markers).
+   * For example, "data.parquet_123_hudiext" returns "data.parquet"
+   * And "data.parquet_123_fg%3Dbucket-0_hudiext" also returns "data.parquet"
+   *
+   * @param fileName The external file name
+   * @return The original file name
+   */
+  private static String getOriginalFileName(String fileName) {
+if (!isExternallyCreatedFile(fileName)) {
+  return fileName;
+}
+int prefixMarkerIndex = fileName.indexOf(FILE_GROUP_PREFIX_MARKER);
+int markerEnd = prefixMarkerIndex != -1
+? prefixMarkerIndex
+: fileName.lastIndexOf(EXTERNAL_FILE_SUFFIX);
+int commitTimeStart = fileName.lastIndexOf('_', markerEnd - 1);
+return fileName.substring(0, commitTimeStart);
+  }
+
+  /**
+   * Adjusts the parent path for external files with file group prefix.
+   * For files with file group prefix, the prefix represents subdirectories 
within the partition,
+   * so we need to remove the prefix portion to get the actual partition path.
+   * Supports arbitrary nesting depths (e.g., "bucket-0/subdir1/subdir2").
+   *
+   * @param parent the parent path
+   * @param fileName the file name to check
+   * @return the adjusted parent path
+   */
+  public static StoragePath getFullPathOfPartition(StoragePath parent, String 
fileName) {
+return getExternalFileGroupPrefix(fileName)
+.map(prefix -> new StoragePath(parent.toString().substring(0, 
parent.toString().length() - prefix.length() - 1)))
+.orElse(parent);
+  }
+
+  /**
+   * Parses external file names to extract fileId and commit time.
+   * Handles both formats:
+   *   - With prefix: originalName_commitTime_fg%3D_hudiext -> fileId 
= prefix/originalName
+   *   - Without prefix: originalName_commitTime_hudiext -> fileId = 
originalName
+   *
+   * @param fileName The external file name to parse
+   * @return String array of size 2: [fileId, commitTime]
+   */
+  public static String[] parseFileIdAndCommitTimeFromExternalFile(String 
fileName) {
+String[] values = new String[2];
+// Extract original file name
+String originalName = getOriginalFileName(fileName);
+// Extract file group prefix (if present)
+Option prefix = getExternalFileGroupPrefix(fileName);

Review Comment:
   The tests are passing but I feel this method is more complex given that it's 
only called for external files?  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

nsivabalan commented on code in PR #17788:
URL: https://github.com/apache/hudi/pull/17788#discussion_r2723453047


##
hudi-common/src/main/java/org/apache/hudi/common/util/ExternalFilePathUtil.java:
##
@@ -43,4 +92,109 @@ public static String 
appendCommitTimeAndExternalFileMarker(String filePath, Stri
   public static boolean isExternallyCreatedFile(String fileName) {
 return fileName.endsWith(EXTERNAL_FILE_SUFFIX);
   }
+
+  /**
+   * Extracts the file group prefix from an external file name.
+   * @param fileName The external file name
+   * @return Option containing the decoded file group prefix, or empty if not 
present
+   */
+  private static Option getExternalFileGroupPrefix(String fileName) {
+if (!isExternallyCreatedFile(fileName)) {
+  return Option.empty();
+}
+int prefixMarkerIndex = fileName.indexOf(FILE_GROUP_PREFIX_MARKER);
+if (prefixMarkerIndex == -1) {
+  return Option.empty();
+}
+int start = prefixMarkerIndex + FILE_GROUP_PREFIX_MARKER.length();
+int end = fileName.lastIndexOf(EXTERNAL_FILE_SUFFIX);
+return 
Option.of(PartitionPathEncodeUtils.unescapePathName(fileName.substring(start, 
end)));
+  }
+
+  /**
+   * Extracts the original file name from an external file name (without 
commit time and markers).
+   * For example, "data.parquet_123_hudiext" returns "data.parquet"
+   * And "data.parquet_123_fg%3Dbucket-0_hudiext" also returns "data.parquet"
+   *
+   * @param fileName The external file name
+   * @return The original file name
+   */
+  private static String getOriginalFileName(String fileName) {
+if (!isExternallyCreatedFile(fileName)) {
+  return fileName;
+}
+int prefixMarkerIndex = fileName.indexOf(FILE_GROUP_PREFIX_MARKER);
+int markerEnd = prefixMarkerIndex != -1
+? prefixMarkerIndex
+: fileName.lastIndexOf(EXTERNAL_FILE_SUFFIX);
+int commitTimeStart = fileName.lastIndexOf('_', markerEnd - 1);
+return fileName.substring(0, commitTimeStart);
+  }
+
+  /**
+   * Adjusts the parent path for external files with file group prefix.
+   * For files with file group prefix, the prefix represents subdirectories 
within the partition,
+   * so we need to remove the prefix portion to get the actual partition path.
+   * Supports arbitrary nesting depths (e.g., "bucket-0/subdir1/subdir2").
+   *
+   * @param parent the parent path
+   * @param fileName the file name to check
+   * @return the adjusted parent path
+   */
+  public static StoragePath getFullPathOfPartition(StoragePath parent, String 
fileName) {
+return getExternalFileGroupPrefix(fileName)
+.map(prefix -> new StoragePath(parent.toString().substring(0, 
parent.toString().length() - prefix.length() - 1)))
+.orElse(parent);
+  }
+
+  /**
+   * Parses external file names to extract fileId and commit time.
+   * Handles both formats:
+   *   - With prefix: originalName_commitTime_fg%3D_hudiext -> fileId 
= prefix/originalName
+   *   - Without prefix: originalName_commitTime_hudiext -> fileId = 
originalName
+   *
+   * @param fileName The external file name to parse
+   * @return String array of size 2: [fileId, commitTime]
+   */
+  public static String[] parseFileIdAndCommitTimeFromExternalFile(String 
fileName) {
+String[] values = new String[2];
+// Extract original file name
+String originalName = getOriginalFileName(fileName);
+// Extract file group prefix (if present)
+Option prefix = getExternalFileGroupPrefix(fileName);

Review Comment:
   I see we are doing 2 passes here. can we get it done in one pass?
   
   ``` 
   public static String[] 
parseFileIdAndCommitTimeFromExternalFileOnePass(String fileName) {
 // return [fileId, commitTime]
 String[] out = new String[2];
   
 if (!isExternallyCreatedFile(fileName)) {
   // Keep same behavior you want for non-external files
   out[0] = fileName;
   out[1] = ""; // or null, or throw, depending on your contract
   return out;
 }
   
 // We assume external files end with EXTERNAL_FILE_SUFFIX (e.g. "_hudiext")
 int suffixStart = fileName.lastIndexOf(EXTERNAL_FILE_SUFFIX);
 if (suffixStart < 0) {
   // Defensive: treat as non-external or throw
   out[0] = fileName;
   out[1] = "";
   return out;
 }
   
 // Check if prefix marker exists and is before suffix
 int fgIdx = fileName.indexOf(FILE_GROUP_PREFIX_MARKER); // e.g. "_fg%3D"
 final int markerEnd = (fgIdx >= 0 && fgIdx < suffixStart) ? fgIdx : 
suffixStart;
   
 // commitTime is between last '_' before markerEnd and markerEnd
 int commitUnderscore = fileName.lastIndexOf('_', markerEnd - 1);
 if (commitUnderscore < 0) {
   throw new IllegalArgumentException("Invalid external file name (no 
commitTime): " + fileName);
 }
   
 String commitTime = fileName.substring(commitUnderscore + 1, markerEnd);
 String originalName = fileName.substring(0, commitUnderscore);
   
 // 

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

hudi-bot commented on PR #17788:
URL: https://github.com/apache/hudi/pull/17788#issuecomment-3792314709

   
   ## CI report:
   
   * d63d8efad7d4f5bf9a88e4175a758dccf852a942 Azure: 
[SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11316)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

hudi-bot commented on PR #17788:
URL: https://github.com/apache/hudi/pull/17788#issuecomment-3791926450

   
   ## CI report:
   
   * 0682a507a1b997d00076bfbdc7298c487ca14318 Azure: 
[SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11247)
 
   * d63d8efad7d4f5bf9a88e4175a758dccf852a942 Azure: 
[PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11316)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

hudi-bot commented on PR #17788:
URL: https://github.com/apache/hudi/pull/17788#issuecomment-3791913294

   
   ## CI report:
   
   * 0682a507a1b997d00076bfbdc7298c487ca14318 Azure: 
[SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11247)
 
   * d63d8efad7d4f5bf9a88e4175a758dccf852a942 UNKNOWN
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

the-other-tim-brown commented on code in PR #17788:
URL: https://github.com/apache/hudi/pull/17788#discussion_r2718036255


##
hudi-common/src/main/java/org/apache/hudi/common/util/ExternalFilePathUtil.java:
##
@@ -43,4 +92,112 @@ public static String 
appendCommitTimeAndExternalFileMarker(String filePath, Stri
   public static boolean isExternallyCreatedFile(String fileName) {
 return fileName.endsWith(EXTERNAL_FILE_SUFFIX);
   }
+
+  /**
+   * Checks if the external file name contains a file group prefix.
+   * @param fileName The file name
+   * @return True if the file has a file group prefix encoded in its name
+   */
+  private static boolean hasExternalFileGroupPrefix(String fileName) {
+return isExternallyCreatedFile(fileName) && 
fileName.contains(FILE_GROUP_PREFIX_MARKER);
+  }
+
+  /**
+   * Extracts the file group prefix from an external file name.
+   * @param fileName The external file name
+   * @return Option containing the decoded file group prefix, or empty if not 
present
+   */
+  private static Option getExternalFileGroupPrefix(String fileName) {
+if (!hasExternalFileGroupPrefix(fileName)) {
+  return Option.empty();
+}
+int start = fileName.indexOf(FILE_GROUP_PREFIX_MARKER) + 
FILE_GROUP_PREFIX_MARKER.length();
+int end = fileName.lastIndexOf(EXTERNAL_FILE_SUFFIX);
+return 
Option.of(PartitionPathEncodeUtils.unescapePathName(fileName.substring(start, 
end)));
+  }
+
+  /**
+   * Extracts the original file name from an external file name (without 
commit time and markers).
+   * For example, "data.parquet_123_hudiext" returns "data.parquet"
+   * And "data.parquet_123_fg%3Dbucket-0_hudiext" also returns "data.parquet"
+   *
+   * @param fileName The external file name
+   * @return The original file name
+   */
+  private static String getOriginalFileName(String fileName) {
+if (!isExternallyCreatedFile(fileName)) {
+  return fileName;
+}
+int markerEnd = hasExternalFileGroupPrefix(fileName)
+? fileName.indexOf(FILE_GROUP_PREFIX_MARKER)
+: fileName.lastIndexOf(EXTERNAL_FILE_SUFFIX);
+int commitTimeStart = fileName.lastIndexOf('_', markerEnd - 1);
+return fileName.substring(0, commitTimeStart);
+  }
+
+  /**
+   * Adjusts the parent path for external files with file group prefix.
+   * For files with file group prefix, the prefix represents a subdirectory 
within the partition,
+   * so we need to go up one more level to get the actual partition path.
+   *
+   * @param parent the parent path
+   * @param fileName the file name to check
+   * @return the adjusted parent path
+   */
+  public static StoragePath getActualParentPath(StoragePath parent, String 
fileName) {

Review Comment:
   A better name may be `getFullPathOfPartition`. Parent path is already a 
defined concept so this may be confusing.



##
hudi-common/src/main/java/org/apache/hudi/common/util/ExternalFilePathUtil.java:
##
@@ -43,4 +92,112 @@ public static String 
appendCommitTimeAndExternalFileMarker(String filePath, Stri
   public static boolean isExternallyCreatedFile(String fileName) {
 return fileName.endsWith(EXTERNAL_FILE_SUFFIX);
   }
+
+  /**
+   * Checks if the external file name contains a file group prefix.
+   * @param fileName The file name
+   * @return True if the file has a file group prefix encoded in its name
+   */
+  private static boolean hasExternalFileGroupPrefix(String fileName) {
+return isExternallyCreatedFile(fileName) && 
fileName.contains(FILE_GROUP_PREFIX_MARKER);
+  }
+
+  /**
+   * Extracts the file group prefix from an external file name.
+   * @param fileName The external file name
+   * @return Option containing the decoded file group prefix, or empty if not 
present
+   */
+  private static Option getExternalFileGroupPrefix(String fileName) {
+if (!hasExternalFileGroupPrefix(fileName)) {
+  return Option.empty();
+}
+int start = fileName.indexOf(FILE_GROUP_PREFIX_MARKER) + 
FILE_GROUP_PREFIX_MARKER.length();
+int end = fileName.lastIndexOf(EXTERNAL_FILE_SUFFIX);
+return 
Option.of(PartitionPathEncodeUtils.unescapePathName(fileName.substring(start, 
end)));
+  }
+
+  /**
+   * Extracts the original file name from an external file name (without 
commit time and markers).
+   * For example, "data.parquet_123_hudiext" returns "data.parquet"
+   * And "data.parquet_123_fg%3Dbucket-0_hudiext" also returns "data.parquet"
+   *
+   * @param fileName The external file name
+   * @return The original file name
+   */
+  private static String getOriginalFileName(String fileName) {
+if (!isExternallyCreatedFile(fileName)) {
+  return fileName;
+}
+int markerEnd = hasExternalFileGroupPrefix(fileName)
+? fileName.indexOf(FILE_GROUP_PREFIX_MARKER)
+: fileName.lastIndexOf(EXTERNAL_FILE_SUFFIX);
+int commitTimeStart = fileName.lastIndexOf('_', markerEnd - 1);
+return fileName.substring(0, commitTimeStart);
+  }
+
+  /**
+   *

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

hudi-bot commented on PR #17788:
URL: https://github.com/apache/hudi/pull/17788#issuecomment-3775876812

   
   ## CI report:
   
   * 0682a507a1b997d00076bfbdc7298c487ca14318 Azure: 
[SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11247)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

hudi-bot commented on PR #17788:
URL: https://github.com/apache/hudi/pull/17788#issuecomment-3775692091

   
   ## CI report:
   
   * 9d12f46f3771cb11a342ebd297dabe17cbf1c1d8 Azure: 
[SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10822)
 
   * 0682a507a1b997d00076bfbdc7298c487ca14318 Azure: 
[PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11247)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

hudi-bot commented on PR #17788:
URL: https://github.com/apache/hudi/pull/17788#issuecomment-3775685903

   
   ## CI report:
   
   * 9d12f46f3771cb11a342ebd297dabe17cbf1c1d8 Azure: 
[SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10822)
 
   * 0682a507a1b997d00076bfbdc7298c487ca14318 UNKNOWN
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

vinishjail97 commented on code in PR #17788:
URL: https://github.com/apache/hudi/pull/17788#discussion_r2710618679


##
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##
@@ -174,7 +175,12 @@ public Option getCompletionTime(String 
instantTime) {
*/
   public List addFilesToView(List statuses) {
 Map> statusesByPartitionPath = 
statuses.stream()
-.collect(Collectors.groupingBy(fileStatus -> 
FSUtils.getRelativePartitionPath(metaClient.getBasePath(), 
fileStatus.getPath().getParent(;
+.collect(Collectors.groupingBy(fileStatus -> {

Review Comment:
   Addressed. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

vinishjail97 commented on code in PR #17788:
URL: https://github.com/apache/hudi/pull/17788#discussion_r2699968305


##
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##
@@ -174,7 +175,12 @@ public Option getCompletionTime(String 
instantTime) {
*/
   public List addFilesToView(List statuses) {
 Map> statusesByPartitionPath = 
statuses.stream()
-.collect(Collectors.groupingBy(fileStatus -> 
FSUtils.getRelativePartitionPath(metaClient.getBasePath(), 
fileStatus.getPath().getParent(;
+.collect(Collectors.groupingBy(fileStatus -> {

Review Comment:
   Include paimon paths in the test for fsView.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

hudi-bot commented on PR #17788:
URL: https://github.com/apache/hudi/pull/17788#issuecomment-3717096970

   
   ## CI report:
   
   * 9d12f46f3771cb11a342ebd297dabe17cbf1c1d8 Azure: 
[SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10822)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

hudi-bot commented on PR #17788:
URL: https://github.com/apache/hudi/pull/17788#issuecomment-3716929912

   
   ## CI report:
   
   * e4f8546b5f4c7ba2a7174ac40ed28b0422384790 Azure: 
[FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10810)
 
   * 9d12f46f3771cb11a342ebd297dabe17cbf1c1d8 Azure: 
[PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10822)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

hudi-bot commented on PR #17788:
URL: https://github.com/apache/hudi/pull/17788#issuecomment-3716926647

   
   ## CI report:
   
   * e4f8546b5f4c7ba2a7174ac40ed28b0422384790 Azure: 
[FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10810)
 
   * 9d12f46f3771cb11a342ebd297dabe17cbf1c1d8 UNKNOWN
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

vinishjail97 commented on PR #17788:
URL: https://github.com/apache/hudi/pull/17788#issuecomment-3716805111

   ```
   Error:  Plugin org.jacoco:jacoco-maven-plugin:0.8.12 or one of its 
dependencies could not be resolved:
   Error:   The following artifacts could not be resolved: 
org.jacoco:jacoco-maven-plugin:pom:0.8.12 (absent): Could not transfer artifact 
org.jacoco:jacoco-maven-plugin:pom:0.8.12 from/to central 
(https://repo.maven.apache.org/maven2): repo.maven.apache.org: Temporary 
failure in name resolution
   Error:  -> [Help 1]
   org.apache.maven.plugin.PluginResolutionException: Plugin 
org.jacoco:jacoco-maven-plugin:0.8.12 or one of its dependencies could not be 
resolved:
The following artifacts could not be resolved: 
org.jacoco:jacoco-maven-plugin:pom:0.8.12 (absent): Could not transfer artifact 
org.jacoco:jacoco-maven-plugin:pom:0.8.12 from/to central 
(https://repo.maven.apache.org/maven2): repo.maven.apache.org: Temporary 
failure in name resolution
   ```
   
   There are temporary dependency resolution errors from maven. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

hudi-bot commented on PR #17788:
URL: https://github.com/apache/hudi/pull/17788#issuecomment-3712348860

   
   ## CI report:
   
   * e4f8546b5f4c7ba2a7174ac40ed28b0422384790 Azure: 
[FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10810)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

hudi-bot commented on PR #17788:
URL: https://github.com/apache/hudi/pull/17788#issuecomment-3712267084

   
   ## CI report:
   
   * e4f8546b5f4c7ba2a7174ac40ed28b0422384790 Azure: 
[PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10810)
 
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: Handle external file groups in ExternalFilePathUtil [hudi]

[via GitHub]

hudi-bot commented on PR #17788:
URL: https://github.com/apache/hudi/pull/17788#issuecomment-3712265192

   
   ## CI report:
   
   * e4f8546b5f4c7ba2a7174ac40ed28b0422384790 UNKNOWN
   
   
   Bot commands
 @hudi-bot supports the following commands:
   
- `@hudi-bot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org