JingsongLi commented on code in PR #11248:
URL: https://github.com/apache/paimon/pull/11248#discussion_r4623681072


##########
paimon-common/src/main/java/org/apache/paimon/globalindex/io/GlobalIndexFileWriter.java:
##########
@@ -25,7 +25,9 @@
 /** Global index writer io operators. */
 public interface GlobalIndexFileWriter {
 
-    String newFileName(String prefix);
+    /** Allocates and returns the full path for a new index file. */
+    String newFilePath(String prefix);

Review Comment:
   [P1] Keep the R-tree writer and existing index plugins buildable
   
   The clean textual merge into master `79b8ec4788` fails JDK 11 reactor 
compilation. `RTreeGlobalIndexWriter.java:142` calls `newFileName(String)`, and 
the R-tree tests still implement or call the old signatures. Rebase and migrate 
the current writer and fixtures. Also check the extension boundary before 
removing the old methods. A previously built Lumina index plugin invokes 
`GlobalIndexFileWriter.newFileName(String)` and would fail linkage against this 
interface even for a single root. Preserve the basename-based SPI with a 
compatibility adapter or define the coordinated plugin upgrade required by the 
change.



##########
paimon-api/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -4188,17 +4189,19 @@ public boolean indexFileInDataFileDir() {
         return options.get(INDEX_FILE_IN_DATA_FILE_DIR);
     }
 
-    public Path globalIndexExternalPath() {
+    public List<Path> globalIndexExternalPaths() {
         String pathString = options.get(GLOBAL_INDEX_EXTERNAL_PATH);
         if (pathString == null || pathString.isEmpty()) {
-            return null;
+            return Collections.emptyList();
         }
-        Path path = new Path(pathString);
-        String scheme = path.toUri().getScheme();
-        if (scheme == null) {
-            throw new IllegalArgumentException("scheme should not be null: " + 
path);
+        List<Path> paths = new ArrayList<>();
+        for (String value : pathString.split(",")) {
+            Path path = new Path(value.trim());

Review Comment:
   [P2] Use the same path parser for writing and dropping tables
   
   The new parser and `CoreOptionsTest` accept `file:///tmp/a, file:///tmp/b`, 
but `AbstractCatalog.getSchemaExternalPaths` still splits and constructs `Path` 
from the untrimmed tokens. I created and dropped a table with those two URI 
forms using `FileSystemCatalog` on this head. Without the space, both roots 
were removed. With the space, parsing the roots for the store succeeded and 
`dropTable` failed with `Illegal character in scheme name`. Please share the 
parser and cover whitespace in the catalog drop test. Also preserve the 
existing single-root interpretation for a legal URI containing a comma, such as 
`file:///tmp/index,archive`, which the old reader accepts and the new parser 
rejects.



##########
docs/docs/multimodal-table/global-index/manage-indexes.mdx:
##########
@@ -475,7 +475,7 @@ These table options affect global index build and read 
behavior:
 | `scalar-index.search-mode` | `fast` | Search mode for BTree, Bitmap, 
Multivalue, and FM queries. |
 | `vector-index.search-mode` | `fast` | Search mode for vector queries. |
 | `full-text-index.search-mode` | `fast` | Search mode for full-text queries. |
-| `global-index.external-path` | Not set | Root directory for global index 
files. If not set, files are stored under the table index directory. |
+| `global-index.external-path` | Not set | Comma-separated root directories 
for global index files, for example `s3://bucket-a/index,s3://bucket-b/index`. 
New files use round-robin selection independently of data-file external path 
settings. Each file's actual path is saved in metadata, so changing this option 
does not relocate existing files. If not set, files are stored under the table 
index directory. |

Review Comment:
   [P1] Handle multiple roots in PyPaimon before enabling this table option
   
   I configured two `file:` roots on a real Data Evolution Parquet table and 
called PyPaimon's public `table.create_global_index('id')`. It built a B-tree 
and persisted a path of the form 
`file:///tmp/root-a,file:///tmp/root-b/btree-….index`. The file was outside 
both configured roots. Current Python builders still use the entire option as 
one directory, and Java drop cleanup enumerates the individual roots. Implement 
multi-root placement for the supported engines or reject multiple roots in 
unsupported writers before creating any artifacts. Add a cross-engine 
build/cleanup case.



##########
paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexBuilderUtils.java:
##########
@@ -609,11 +596,21 @@ private static List<IndexFileMeta> toIndexFileMetas(
     public static GlobalIndexWriter createIndexWriter(
             FileStoreTable table, String indexType, List<DataField> 
indexFields, Options options)
             throws IOException {
+        return createIndexWriter(
+                createGlobalIndexFileReadWrite(table), indexType, indexFields, 
options);

Review Comment:
   [P2] Keep round-robin state across generic shards
   
   `GenericIndexTopoBuilder` calls this overload per shard. Each call creates a 
new path factory and a randomly seeded round-robin provider. In a single Flink 
subtask, consecutive one-file shards can therefore all pick the same root. The 
same factory construction is used by the Spark generic builder. A local 
two-root probe alternated when reusing the provider and repeated roots when 
constructing one per allocation. Please reuse the allocator at the per-worker 
scope or document the actual per-writer placement semantics, with a multi-shard 
test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to