C-Loftus commented on code in PR #1350:
URL: https://github.com/apache/iceberg-go/pull/1350#discussion_r3507823385
##########
catalog/hadoop/io.go:
##########
@@ -29,59 +28,15 @@ type HadoopCatalogFS interface {
icebergio.ListableIO
icebergio.ReadFileIO
icebergio.WriteFileIO
- StatIO
- RenameIO
- RenameNoReplaceIO
- RemoveAllIO
- MkdirAllIO
+ icebergio.StatIO
+ icebergio.RenameIO
+ icebergio.RenameNoReplaceIO
+ icebergio.RemoveAllIO
+ icebergio.MkdirAllIO
}
+// LocalFS can be used to implement a Hadoop catalog with a local filesystem.
var _ HadoopCatalogFS = (*icebergio.LocalFS)(nil)
-// StatIO is an extension of IO interface that includes the Stat
-// method for retrieving file or directory information
-type StatIO interface {
- icebergio.IO
-
- // The Stat method returns a FileInfo describing the named file or
directory,
- // or an error satisfying errors.Is(err, fs.ErrNotExist) if the file
does not exist
- Stat(path string) (fs.FileInfo, error)
-}
-
-// RenameIO is an extension of IO interface that includes the Rename
-// method for renaming (moving) files or directories; this must be
-// atomic and can be used for committing metadata updates
-type RenameIO interface {
- icebergio.IO
-
- Rename(oldpath, newpath string) error
-}
-
-// RenameNoReplaceIO is an extension of IO interface that includes the
-// RenameNoReplace method for atomically moving files only when the destination
-// path does not already exist. Implementations may require oldpath and newpath
-// to live on the same filesystem.
-type RenameNoReplaceIO interface {
- icebergio.IO
-
- RenameNoReplace(oldpath, newpath string) error
-}
-
-// RemoveAllIO is an extension of IO interface that includes the RemoveAll
-// method for removing a file path and any children recursively
-type RemoveAllIO interface {
- icebergio.IO
-
- RemoveAll(name string) error
-}
-
-// MkdirAllIO is an extension of IO interface that includes the MkdirAll
-// method for creating a directory path recursively
-type MkdirAllIO interface {
- icebergio.IO
-
- // MkdirAll should not raise an error if the directory already exists,
- // and should create any parent directories in the path if they
- // do not already exist.
- MkdirAll(path string) error
-}
+// BlobFileIO can be used to implement a Hadoop catalog with a blob storage
bucket.
+var _ HadoopCatalogFS = (*gocloud.BlobFileIO)(nil)
Review Comment:
I have made `BlobFileIO` a public export now so that we can have a compile
time interface assertion in the `hadoop` package.
##########
io/gocloud/blob.go:
##########
@@ -256,10 +319,178 @@ func (bfs *blobFileIO) DeleteFiles(ctx context.Context,
paths []string) ([]strin
return deleted, errs
}
+// MkdirAll mimics creating a directory by creating a zero-length object for
each component of the path
+func (bfs *BlobFileIO) MkdirAll(path string) error {
Review Comment:
There was a fair bit of discussion in #1096 regarding how to implement
MkdirAll in a way that makes sense for blob stores that don't have a notion of
a directory. Worth reviewing this function a bit closer to ensure that even if
it passes all tests, it performs as desired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
