Akash3121 commented on code in PR #11248:
URL: https://github.com/apache/paimon/pull/11248#discussion_r4619699409
##########
paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexBuilderUtils.java:
##########
@@ -609,11 +596,21 @@ private static List<IndexFileMeta> toIndexFileMetas(
public static GlobalIndexWriter createIndexWriter(
FileStoreTable table, String indexType, List<DataField>
indexFields, Options options)
throws IOException {
+ return createIndexWriter(
+ createGlobalIndexFileReadWrite(table), indexType, indexFields,
options);
Review Comment:
In `SortedGlobalIndexWriter`, `fileReadWrite` is cached across rotated files
so that `createGlobalIndexFileReadWrite(table)` is not called per file (since
`table.store().pathFactory().globalIndexFileFactory()` creates a new
`RoundRobinExternalPathProvider` whose starting cursor is randomized via
`ThreadLocalRandom`).
However, in `GenericIndexTopoBuilder.BuildIndexOperator#processElement`, a
single operator subtask can process multiple `IndexedSplit` shards and still
calls `createIndexWriter(table, ...)` on every shard. That recreates a new
`RoundRobinExternalPathProvider` with a fresh random starting position for each
shard instead of round-robining across shards processed by the same subtask.
Could we also initialize a `transient GlobalIndexFileReadWrite` once in
`BuildIndexOperator#open()` and pass it to `createIndexWriter(fileReadWrite,
...)` (just like `SortedGlobalIndexWriter`)?
##########
paimon-api/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -4188,17 +4189,19 @@ public boolean indexFileInDataFileDir() {
return options.get(INDEX_FILE_IN_DATA_FILE_DIR);
}
- public Path globalIndexExternalPath() {
+ public List<Path> globalIndexExternalPaths() {
String pathString = options.get(GLOBAL_INDEX_EXTERNAL_PATH);
if (pathString == null || pathString.isEmpty()) {
- return null;
+ return Collections.emptyList();
}
- Path path = new Path(pathString);
- String scheme = path.toUri().getScheme();
- if (scheme == null) {
- throw new IllegalArgumentException("scheme should not be null: " +
path);
+ List<Path> paths = new ArrayList<>();
+ for (String value : pathString.split(",")) {
+ Path path = new Path(value.trim());
+ checkArgument(path.toUri().getScheme() != null, "scheme should not
be null: %s", path);
Review Comment:
`CoreOptions#globalIndexExternalPaths` trims each comma-separated path
(`value.trim()`) and ignores `""` (and `CoreOptionsTest` explicitly tests
values with spaces and trailing commas like `"file:///a, file:///b"`, `"
file:///a/ "`, and `""`).
However, during `dropTable`, `AbstractCatalog#getSchemaExternalPaths`
splits the raw option value directly via:
```java
.flatMap(externalPath -> Arrays.stream(externalPath.split(",")))
.map(Path::new)
```
without trimming or filtering empty strings. If global-index.external-path
is configured with spaces around the comma (e.g., "file:///a, file:///b") or
"", new Path(" file:///b") / new Path("") in
AbstractCatalog#getSchemaExternalPaths will throw IllegalArgumentException
(URISyntaxException: Illegal character in scheme name at index 0 / Can not
create a Path from an empty string) when dropping the table.
Should we update AbstractCatalog#getSchemaExternalPaths to
.map(String::trim).filter(s -> !s.isEmpty()).map(Path::new) so dropTable
handles the same inputs accepted by CoreOptions#globalIndexExternalPaths?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]