JingsongLi commented on code in PR #11248:
URL: https://github.com/apache/paimon/pull/11248#discussion_r4623681072
##########
paimon-common/src/main/java/org/apache/paimon/globalindex/io/GlobalIndexFileWriter.java:
##########
@@ -25,7 +25,9 @@
/** Global index writer io operators. */
public interface GlobalIndexFileWriter {
- String newFileName(String prefix);
+ /** Allocates and returns the full path for a new index file. */
+ String newFilePath(String prefix);
Review Comment:
[P1] Keep the R-tree writer and existing index plugins buildable
The clean textual merge into master `79b8ec4788` fails JDK 11 reactor
compilation. `RTreeGlobalIndexWriter.java:142` calls `newFileName(String)`, and
the R-tree tests still implement or call the old signatures. Rebase and migrate
the current writer and fixtures. Also check the extension boundary before
removing the old methods. A previously built Lumina index plugin invokes
`GlobalIndexFileWriter.newFileName(String)` and would fail linkage against this
interface even for a single root. Preserve the basename-based SPI with a
compatibility adapter or define the coordinated plugin upgrade required by the
change.
##########
paimon-api/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -4188,17 +4189,19 @@ public boolean indexFileInDataFileDir() {
return options.get(INDEX_FILE_IN_DATA_FILE_DIR);
}
- public Path globalIndexExternalPath() {
+ public List<Path> globalIndexExternalPaths() {
String pathString = options.get(GLOBAL_INDEX_EXTERNAL_PATH);
if (pathString == null || pathString.isEmpty()) {
- return null;
+ return Collections.emptyList();
}
- Path path = new Path(pathString);
- String scheme = path.toUri().getScheme();
- if (scheme == null) {
- throw new IllegalArgumentException("scheme should not be null: " +
path);
+ List<Path> paths = new ArrayList<>();
+ for (String value : pathString.split(",")) {
+ Path path = new Path(value.trim());
Review Comment:
[P2] Use the same path parser for writing and dropping tables
The new parser and `CoreOptionsTest` accept `file:///tmp/a, file:///tmp/b`,
but `AbstractCatalog.getSchemaExternalPaths` still splits and constructs `Path`
from the untrimmed tokens. I created and dropped a table with those two URI
forms using `FileSystemCatalog` on this head. Without the space, both roots
were removed. With the space, parsing the roots for the store succeeded and
`dropTable` failed with `Illegal character in scheme name`. Please share the
parser and cover whitespace in the catalog drop test. Also preserve the
existing single-root interpretation for a legal URI containing a comma, such as
`file:///tmp/index,archive`, which the old reader accepts and the new parser
rejects.
##########
docs/docs/multimodal-table/global-index/manage-indexes.mdx:
##########
@@ -475,7 +475,7 @@ These table options affect global index build and read
behavior:
| `scalar-index.search-mode` | `fast` | Search mode for BTree, Bitmap,
Multivalue, and FM queries. |
| `vector-index.search-mode` | `fast` | Search mode for vector queries. |
| `full-text-index.search-mode` | `fast` | Search mode for full-text queries. |
-| `global-index.external-path` | Not set | Root directory for global index
files. If not set, files are stored under the table index directory. |
+| `global-index.external-path` | Not set | Comma-separated root directories
for global index files, for example `s3://bucket-a/index,s3://bucket-b/index`.
New files use round-robin selection independently of data-file external path
settings. Each file's actual path is saved in metadata, so changing this option
does not relocate existing files. If not set, files are stored under the table
index directory. |
Review Comment:
[P1] Handle multiple roots in PyPaimon before enabling this table option
I configured two `file:` roots on a real Data Evolution Parquet table and
called PyPaimon's public `table.create_global_index('id')`. It built a B-tree
and persisted a path of the form
`file:///tmp/root-a,file:///tmp/root-b/btree-….index`. The file was outside
both configured roots. Current Python builders still use the entire option as
one directory, and Java drop cleanup enumerates the individual roots. Implement
multi-root placement for the supported engines or reject multiple roots in
unsupported writers before creating any artifacts. Add a cross-engine
build/cleanup case.
##########
paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexBuilderUtils.java:
##########
@@ -609,11 +596,21 @@ private static List<IndexFileMeta> toIndexFileMetas(
public static GlobalIndexWriter createIndexWriter(
FileStoreTable table, String indexType, List<DataField>
indexFields, Options options)
throws IOException {
+ return createIndexWriter(
+ createGlobalIndexFileReadWrite(table), indexType, indexFields,
options);
Review Comment:
[P2] Keep round-robin state across generic shards
`GenericIndexTopoBuilder` calls this overload per shard. Each call creates a
new path factory and a randomly seeded round-robin provider. In a single Flink
subtask, consecutive one-file shards can therefore all pick the same root. The
same factory construction is used by the Spark generic builder. A local
two-root probe alternated when reusing the provider and repeated roots when
constructing one per allocation. Please reuse the allocator at the per-worker
scope or document the actual per-writer placement semantics, with a multi-shard
test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]