Akash3121 commented on code in PR #11248:
URL: https://github.com/apache/paimon/pull/11248#discussion_r4619699409


##########
paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexBuilderUtils.java:
##########
@@ -609,11 +596,21 @@ private static List<IndexFileMeta> toIndexFileMetas(
     public static GlobalIndexWriter createIndexWriter(
             FileStoreTable table, String indexType, List<DataField> 
indexFields, Options options)
             throws IOException {
+        return createIndexWriter(
+                createGlobalIndexFileReadWrite(table), indexType, indexFields, 
options);

Review Comment:
   In `SortedGlobalIndexWriter`, `fileReadWrite` is cached across rotated files 
so that `createGlobalIndexFileReadWrite(table)` is not called per file (since 
`table.store().pathFactory().globalIndexFileFactory()` creates a new 
`RoundRobinExternalPathProvider` whose starting cursor is randomized via 
`ThreadLocalRandom`).
    
    However, in `GenericIndexTopoBuilder.BuildIndexOperator#processElement`, a 
single operator subtask can process multiple `IndexedSplit` shards and still 
calls `createIndexWriter(table, ...)` on every shard. That recreates a new 
`RoundRobinExternalPathProvider` with a fresh random starting position for each 
shard instead of round-robining across shards processed by the same subtask.
    
    Could we also initialize a `transient GlobalIndexFileReadWrite` once in 
`BuildIndexOperator#open()` and pass it to `createIndexWriter(fileReadWrite, 
...)` (just like `SortedGlobalIndexWriter`)?



##########
paimon-api/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -4188,17 +4189,19 @@ public boolean indexFileInDataFileDir() {
         return options.get(INDEX_FILE_IN_DATA_FILE_DIR);
     }
 
-    public Path globalIndexExternalPath() {
+    public List<Path> globalIndexExternalPaths() {
         String pathString = options.get(GLOBAL_INDEX_EXTERNAL_PATH);
         if (pathString == null || pathString.isEmpty()) {
-            return null;
+            return Collections.emptyList();
         }
-        Path path = new Path(pathString);
-        String scheme = path.toUri().getScheme();
-        if (scheme == null) {
-            throw new IllegalArgumentException("scheme should not be null: " + 
path);
+        List<Path> paths = new ArrayList<>();
+        for (String value : pathString.split(",")) {
+            Path path = new Path(value.trim());
+            checkArgument(path.toUri().getScheme() != null, "scheme should not 
be null: %s", path);

Review Comment:
   `CoreOptions#globalIndexExternalPaths` trims each comma-separated path 
(`value.trim()`) and ignores `""` (and `CoreOptionsTest` explicitly tests 
values with spaces and trailing commas like `"file:///a, file:///b"`, `" 
file:///a/ "`, and `""`).
    
    However, during `dropTable`, `AbstractCatalog#getSchemaExternalPaths` 
splits the raw option value directly via:
    ```java
    .flatMap(externalPath -> Arrays.stream(externalPath.split(",")))
    .map(Path::new)
   ```
   without trimming or filtering empty strings. If global-index.external-path 
is configured with spaces around the comma (e.g., "file:///a, file:///b") or 
"", new Path(" file:///b") / new Path("") in 
AbstractCatalog#getSchemaExternalPaths will throw IllegalArgumentException 
(URISyntaxException: Illegal character in scheme name at index 0 / Can not 
create a Path from an empty string) when dropping the table.
   
   Should we update AbstractCatalog#getSchemaExternalPaths to 
.map(String::trim).filter(s -> !s.isEmpty()).map(Path::new) so dropTable 
handles the same inputs accepted by CoreOptions#globalIndexExternalPaths?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to