djouallah opened a new issue, #2159: URL: https://github.com/apache/iceberg-go/issues/2159
### Apache Iceberg version 0.7.0 (latest release) ### Please describe the bug 🐞 On storage with a hierarchical namespace, `PlanOrphanFiles` / `DeleteOrphanFiles` (and so `iceberg clean-orphan-files`) never list the table's files. They return the table location itself as the only orphan: a 0-byte file whose path is the table location. Seen on Microsoft Fabric OneLake (`abfss://<workspace>@onelake.dfs.fabric.microsoft.com/<lakehouse>/Tables/<ns>/<table>`, `adls.endpoint=blob.fabric.microsoft.com`, REST catalog `https://onelake.table.fabric.microsoft.com/iceberg`). I expect the same on any ADLS Gen2 account with a hierarchical namespace, but have only tested OneLake. **Cause.** A hierarchical namespace stores a directory as a 0-byte blob named after the directory, with no trailing slash and with metadata `hdi_isfolder=true`. - `blobfs.FileIO.WalkDir` (`io/gocloud/blobfs/blob.go`) calls `fs.WalkDir(bucket, "<prefix>/Tables/ns/table", ...)`. - `fs.WalkDir` stats the root through gocloud's `Bucket.Open`, which decides file-or-directory with `b.Exists(ctx, path)` (`gocloud.dev/blob/blob_fs.go`). - `Exists` is true for the directory blob, so the root is opened as a file. `WalkDir` calls the callback once for it and never descends. - `isDirectoryMarker` only skips `<root>/` (with the trailing slash), so it doesn't catch this blob. **Result:** every table reports `1 orphan file, 0 bytes`, which is the table directory itself. Without `--dry-run`, cleanup would try to delete the table's directory entry. What it should report is the unreferenced data and metadata files under it. **Reproduce.** Dry run against any table on OneLake: ```go tbl, _ := cat.LoadTable(ctx, catalog.ToIdentifier("ns", "table")) plan, _ := tbl.PlanOrphanFiles(ctx, table.WithFilesOlderThan(72*time.Hour)) for _, f := range plan.OrphanFiles() { fmt.Println(f.Path, f.SizeBytes) // prints only tbl.Location(), 0 } ``` Output on 25 tables. 22 report exactly this one entry; the other 3 report nothing: ``` landing.fct_price_today location abfss://<ws>@onelake.dfs.fabric.microsoft.com/<lh>/Tables/landing/fct_price_today snapshots 47, oldest 35h19m0s old orphans 1 files, 0.0 MiB, older than 72h0m0s path == location ``` **Possible fix:** for the walk root, list by prefix `key + "/"` instead of letting `fs.WalkDir` stat it. Or treat a 0-byte blob with `hdi_isfolder=true` metadata as a directory, both for the root and for subdirectories met during the walk. Snapshot expiry (`ExpireSnapshots`) works against the same catalog and storage. Only the directory walk is affected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
