laskoviymishka commented on code in PR #2392:
URL: https://github.com/apache/iceberg-go/pull/2392#discussion_r4577810732


##########
io/gocloud/blobfs/blob.go:
##########
@@ -37,26 +37,131 @@ import (
        "gocloud.dev/gcerrors"
 )
 
+// wholeObjectReadMaxSize is the largest object ReadAt downloads whole on its
+// first call. Reading a Parquet file issues a ReadAt for its footer and for
+// every column chunk; for a small file one GET of the object is far cheaper
+// than one GET per column. The downloaded object is held until Close.

Review Comment:
   `Close` never touches `f.data`, so the buffer lives as long as the file 
value is referenced, not until `Close`. Clearing it in `Close` would need a 
lock against concurrent `ReadAt`, so fixing the comment seems simpler:
   
   ```suggestion
   // than one GET per column. The downloaded object stays in memory for as long
   // as the file is referenced; Close does not release it.
   ```



##########
io/gocloud/blobfs/blob.go:
##########
@@ -37,26 +37,131 @@ import (
        "gocloud.dev/gcerrors"
 )
 
+// wholeObjectReadMaxSize is the largest object ReadAt downloads whole on its
+// first call. Reading a Parquet file issues a ReadAt for its footer and for
+// every column chunk; for a small file one GET of the object is far cheaper
+// than one GET per column. The downloaded object is held until Close.
+const wholeObjectReadMaxSize = 8 << 20
+
 // blobOpenFile describes a single open blob as a File.
 // It implements the iceberg-go/io.File interface.
-// It is based on gocloud.dev/blob.iofsOpenFile which:
-// - Doesn't support the `io.ReaderAt` interface
-// - Is not externally accessible, so copied here
+//
+// Opening a blobOpenFile reads only the object's attributes. Read starts a
+// streaming GET from the current offset on first use, and ReadAt issues a
+// ranged GET per call, or downloads an object of at most
+// wholeObjectReadMaxSize once and serves every ReadAt from memory.
 type blobOpenFile struct {
-       *blob.Reader
-
        name, key string
+       size      int64
+       modTime   time.Time
        b         *FileIO
        ctx       context.Context
+
+       mu  sync.Mutex // guards r and pos, used by Read and Seek
+       r   io.ReadCloser
+       pos int64
+
+       whole    sync.Once // downloads data for ReadAt of a small object
+       data     []byte
+       wholeErr error
 }
 
-func (f *blobOpenFile) ReadAt(p []byte, off int64) (n int, err error) {
-       var rdr io.ReadCloser
+func (f *blobOpenFile) rangeReader(off, length int64) (io.ReadCloser, error) {
        if f.b.newRangeReader != nil {
-               rdr, err = f.b.newRangeReader(f.ctx, f.key, off, int64(len(p)))
-       } else {
-               rdr, err = f.b.NewRangeReader(f.ctx, f.key, off, int64(len(p)), 
nil)
+               return f.b.newRangeReader(f.ctx, f.key, off, length)
+       }
+
+       return f.b.NewRangeReader(f.ctx, f.key, off, length, nil)
+}
+
+func (f *blobOpenFile) Read(p []byte) (int, error) {
+       f.mu.Lock()
+       defer f.mu.Unlock()
+
+       if f.pos >= f.size {
+               return 0, io.EOF
        }
+       if f.r == nil {

Review Comment:
   Should `Read` after `Close` return `fs.ErrClosed` instead of starting a new 
GET? 
[`Close`](https://github.com/apache/iceberg-go/blob/7c4352e8b0bd46a18ded27a42571cf170e2bfa3e/io/gocloud/blobfs/blob.go#L125-L136)
 only nils `f.r`, so the next `Read` lands here and opens a fresh streaming 
reader that stays open until `Close` is called again. With a counting 
`newRangeReader`, that `Read` sends one GET on this head and none on `main`. A 
`closed` flag checked here would cover it, plus a `Read`-after-`Close` test.



##########
io/gocloud/blobfs/blob.go:
##########
@@ -37,26 +37,131 @@ import (
        "gocloud.dev/gcerrors"
 )
 
+// wholeObjectReadMaxSize is the largest object ReadAt downloads whole on its
+// first call. Reading a Parquet file issues a ReadAt for its footer and for
+// every column chunk; for a small file one GET of the object is far cheaper
+// than one GET per column. The downloaded object is held until Close.
+const wholeObjectReadMaxSize = 8 << 20
+
 // blobOpenFile describes a single open blob as a File.
 // It implements the iceberg-go/io.File interface.
-// It is based on gocloud.dev/blob.iofsOpenFile which:
-// - Doesn't support the `io.ReaderAt` interface
-// - Is not externally accessible, so copied here
+//
+// Opening a blobOpenFile reads only the object's attributes. Read starts a
+// streaming GET from the current offset on first use, and ReadAt issues a
+// ranged GET per call, or downloads an object of at most
+// wholeObjectReadMaxSize once and serves every ReadAt from memory.
 type blobOpenFile struct {
-       *blob.Reader
-
        name, key string
+       size      int64
+       modTime   time.Time
        b         *FileIO
        ctx       context.Context
+
+       mu  sync.Mutex // guards r and pos, used by Read and Seek
+       r   io.ReadCloser
+       pos int64
+
+       whole    sync.Once // downloads data for ReadAt of a small object
+       data     []byte
+       wholeErr error
 }
 
-func (f *blobOpenFile) ReadAt(p []byte, off int64) (n int, err error) {
-       var rdr io.ReadCloser
+func (f *blobOpenFile) rangeReader(off, length int64) (io.ReadCloser, error) {
        if f.b.newRangeReader != nil {
-               rdr, err = f.b.newRangeReader(f.ctx, f.key, off, int64(len(p)))
-       } else {
-               rdr, err = f.b.NewRangeReader(f.ctx, f.key, off, int64(len(p)), 
nil)
+               return f.b.newRangeReader(f.ctx, f.key, off, length)
+       }
+
+       return f.b.NewRangeReader(f.ctx, f.key, off, length, nil)
+}
+
+func (f *blobOpenFile) Read(p []byte) (int, error) {
+       f.mu.Lock()
+       defer f.mu.Unlock()
+
+       if f.pos >= f.size {
+               return 0, io.EOF
        }
+       if f.r == nil {
+               r, err := f.rangeReader(f.pos, -1)
+               if err != nil {
+                       return 0, err
+               }
+               f.r = r
+       }
+       n, err := f.r.Read(p)
+       f.pos += int64(n)
+
+       return n, err
+}
+
+func (f *blobOpenFile) Seek(offset int64, whence int) (int64, error) {
+       f.mu.Lock()
+       defer f.mu.Unlock()
+
+       switch whence {
+       case io.SeekCurrent:
+               offset += f.pos
+       case io.SeekEnd:
+               offset += f.size
+       case io.SeekStart:
+       default:
+               return 0, fmt.Errorf("blobfs: invalid whence %d", whence)
+       }
+       if offset < 0 {
+               return 0, fmt.Errorf("blobfs: negative position %d", offset)
+       }
+       if offset != f.pos && f.r != nil {
+               err := f.r.Close()
+               f.r = nil
+               if err != nil {
+                       return 0, err
+               }
+       }
+       f.pos = offset
+
+       return offset, nil
+}
+
+func (f *blobOpenFile) Close() error {
+       f.mu.Lock()
+       defer f.mu.Unlock()
+
+       if f.r == nil {
+               return nil
+       }
+       err := f.r.Close()
+       f.r = nil
+
+       return err
+}
+
+func (f *blobOpenFile) ReadAt(p []byte, off int64) (int, error) {
+       if f.size > wholeObjectReadMaxSize {
+               return f.readRange(p, off)
+       }
+
+       f.whole.Do(func() {
+               data := make([]byte, f.size)
+               _, f.wholeErr = f.readRange(data, 0)
+               f.data = data
+       })
+       if f.wholeErr != nil {
+               return 0, f.wholeErr
+       }
+       if off >= int64(len(f.data)) {

Review Comment:
   A negative offset now crashes the reader instead of failing the read. On 
objects of 8 MiB or less, `ReadAt(p, -1)` panics with `slice bounds out of 
range [-1:]`; `main` passed the offset to `NewRangeReader`, which [returns 
InvalidArgument](https://github.com/google/go-cloud/blob/115ae339a43f37c568480ec6cbafda1ffc7863c0/blob/blob.go#L1037-L1038),
 and objects over 8 MiB still do. A repro panics on this head and returns the 
error on the base. Could we guard it and add an `off = -1` case to the `ReadAt` 
tests?
   
   ```suggestion
        if off < 0 {
                return 0, fmt.Errorf("blobfs: negative offset %d", off)
        }
        if off >= int64(len(f.data)) {
   ```



##########
io/gocloud/blobfs/blob.go:
##########
@@ -37,26 +37,131 @@ import (
        "gocloud.dev/gcerrors"
 )
 
+// wholeObjectReadMaxSize is the largest object ReadAt downloads whole on its
+// first call. Reading a Parquet file issues a ReadAt for its footer and for
+// every column chunk; for a small file one GET of the object is far cheaper
+// than one GET per column. The downloaded object is held until Close.
+const wholeObjectReadMaxSize = 8 << 20
+
 // blobOpenFile describes a single open blob as a File.
 // It implements the iceberg-go/io.File interface.
-// It is based on gocloud.dev/blob.iofsOpenFile which:
-// - Doesn't support the `io.ReaderAt` interface
-// - Is not externally accessible, so copied here
+//
+// Opening a blobOpenFile reads only the object's attributes. Read starts a
+// streaming GET from the current offset on first use, and ReadAt issues a
+// ranged GET per call, or downloads an object of at most
+// wholeObjectReadMaxSize once and serves every ReadAt from memory.
 type blobOpenFile struct {
-       *blob.Reader
-
        name, key string
+       size      int64
+       modTime   time.Time
        b         *FileIO
        ctx       context.Context
+
+       mu  sync.Mutex // guards r and pos, used by Read and Seek
+       r   io.ReadCloser
+       pos int64
+
+       whole    sync.Once // downloads data for ReadAt of a small object
+       data     []byte
+       wholeErr error
 }
 
-func (f *blobOpenFile) ReadAt(p []byte, off int64) (n int, err error) {
-       var rdr io.ReadCloser
+func (f *blobOpenFile) rangeReader(off, length int64) (io.ReadCloser, error) {
        if f.b.newRangeReader != nil {
-               rdr, err = f.b.newRangeReader(f.ctx, f.key, off, int64(len(p)))
-       } else {
-               rdr, err = f.b.NewRangeReader(f.ctx, f.key, off, int64(len(p)), 
nil)
+               return f.b.newRangeReader(f.ctx, f.key, off, length)
+       }
+
+       return f.b.NewRangeReader(f.ctx, f.key, off, length, nil)
+}
+
+func (f *blobOpenFile) Read(p []byte) (int, error) {
+       f.mu.Lock()
+       defer f.mu.Unlock()
+
+       if f.pos >= f.size {
+               return 0, io.EOF
        }
+       if f.r == nil {
+               r, err := f.rangeReader(f.pos, -1)
+               if err != nil {
+                       return 0, err
+               }
+               f.r = r
+       }
+       n, err := f.r.Read(p)

Review Comment:
   Should `Close` be able to interrupt a stalled `Read`? `f.mu` is held across 
this network read and 
[`Close`](https://github.com/apache/iceberg-go/blob/7c4352e8b0bd46a18ded27a42571cf170e2bfa3e/io/gocloud/blobfs/blob.go#L125-L136)
 takes the same lock, so with a hung stream `Close` waited over 500 ms in a 
repro. `main` didn't block, but under `-race` its `Close` races 
`blob.Reader.Read`, so the lock is an improvement. Closing `f.r` outside the 
lock would let it cancel; or we accept the wait and document it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to