aakashofficial-k01 opened a new issue, #4379:
URL: https://github.com/apache/amoro/issues/4379
### What happened?
In `amoro-format-iceberg`, table location URIs and target file paths are
constructed using `File.separator` instead of Iceberg's URI path standards:
1. `TableFileUtil.getNewFilePath()`:
```java
return newDirectory + File.separator + getFileName(filePath);
```
2. `IcebergTableMaintainer.clearInternalTableContentsFiles()`:
```java
String dataLocation = table.location() + File.separator +
DATA_FOLDER_NAME;
```
3. `IcebergTableMaintainer.clearInternalTableMetadata()`:
```java
String metadataLocation = table.location() + File.separator +
METADATA_FOLDER_NAME;
```
Table locations and object storage locations in Iceberg are URIs (RFC 3986),
where hierarchical path segments are separated strictly by forward slash `/`.
Using the host OS `File.separator` introduces two defects:
1. **URI Syntax Error on Windows Environments**:
On Windows (`File.separator == "\\"`), calling
`TableFileUtil.getNewFilePath("hdfs://nn:8020/warehouse/db/tbl",
"00000-0.parquet")` returns `hdfs://nn:8020/warehouse/db/tbl\00000-0.parquet`.
When this path is later passed to `TableFileUtil.getUriPath()` or
`java.net.URI.create()` during table optimizing commit
(`UnKeyedTableCommit.moveTargetFiles`), Java immediately fails:
```text
java.lang.IllegalArgumentException: Illegal character in path at index
37: hdfs://nn:8020/warehouse/db/tbl\00000-0.parquet
at java.base/java.net.URI.create(URI.java:883)
Caused by: java.net.URISyntaxException: Illegal character in path at
index 37: hdfs://nn:8020/warehouse/db/tbl\00000-0.parquet
```
Similarly, for cloud object stores (S3/OSS/GCS), `table.location() +
File.separator + METADATA_FOLDER_NAME` produces `s3://bucket/tbl\metadata`.
Because S3 keys use forward slashes, prefix listing (`pio.listPrefix()`) fails
to match object keys and returns 0 files, silently skipping orphan metadata
cleanup.
2. **Double Slashes with Trailing Slashes**:
On Linux, if a table location or directory ends with a trailing slash
(`/`), concatenation produces redundant double slashes (`...//metadata`).
### Affects Versions
master (0.9-SNAPSHOT)
### How to reproduce
1. Run `TableFileUtil.getNewFilePath("s3://bucket/warehouse/db/tbl",
"00000-0.parquet")` on Windows.
2. Pass the result to `TableFileUtil.getUriPath(path)` or `URI.create(path)`.
- **Expected**: `s3://bucket/warehouse/db/tbl/00000-0.parquet` and URI path
`/warehouse/db/tbl/00000-0.parquet`.
- **Actual**: returns `s3://bucket/warehouse/db/tbl\00000-0.parquet` and
throws `java.lang.IllegalArgumentException: Illegal character in path`.
### Proposed Solution
1. In `TableFileUtil.getNewFilePath()`, use Iceberg's standard
`LocationUtil.stripTrailingSlash(newDirectory)` and format with `/`.
2. In `IcebergTableMaintainer`, construct `dataLocation` and
`metadataLocation` via `String.format("%s/%s",
LocationUtil.stripTrailingSlash(table.location()), ...)` adhering to Iceberg's
`DefaultLocationProvider` convention.
3. Add unit test coverage in `TestFileUtil` verifying URI formatting with
and without trailing slashes across S3 and HDFS paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
