zeroshade commented on a change in pull request #9862:
URL: https://github.com/apache/arrow/pull/9862#discussion_r606820653
##########
File path: go/arrow/internal/arrdata/ioutil.go
##########
@@ -172,6 +173,47 @@ func WriteFile(t *testing.T, f *os.File, mem
memory.Allocator, schema *arrow.Sch
}
}
+// WriteFile writes a list of records to the given file descriptor, as an
ARROW file.
+func WriteFileCompressed(t *testing.T, f *os.File, mem memory.Allocator,
schema *arrow.Schema, recs []array.Record, codec flatbuf.CompressionType) {
+ t.Helper()
+
+ opts := []ipc.Option{ipc.WithSchema(schema), ipc.WithAllocator(mem)}
+ switch codec {
+ case flatbuf.CompressionTypeLZ4_FRAME:
+ opts = append(opts, ipc.WithLZ4())
+ case flatbuf.CompressionTypeZSTD:
+ opts = append(opts, ipc.WithZstd())
+ }
+
+ w, err := ipc.NewFileWriter(f, opts...)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer w.Close()
+
+ for i, rec := range recs {
+ err = w.Write(rec)
+ if err != nil {
+ t.Fatalf("could not write record[%d]: %v", i, err)
+ }
+ }
+
+ err = w.Close()
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ err = f.Sync()
+ if err != nil {
+ t.Fatalf("could not sync data to disk: %v", err)
+ }
+
+ _, err = f.Seek(0, io.SeekStart)
Review comment:
So, I actually copied the existing WriteFile test function and then just
added the compression to it, but looking at the code my guess is that it's
because the file isn't being opened here, it's passed in. So doing the seek to
start after writing just preps the file handle to immediately be able to read
from it for the test rather than leaving it at the end of the file after doing
the writes, which makes sense since this is only used for testing and can't be
called externally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
