It was parquet-go indeed, which works well but as pure Go has some performance limits.
In our volume and patterns, Python using Arrow is actually faster apparently due to the C++ code for writing Parquet via Arrow. Thus using Arrow in Go gives us another performance path. On Wed, Dec 5, 2018 at 3:45 PM Sebastien Binet <[email protected]> wrote: > hi, > > On Wed, Dec 5, 2018 at 5:29 PM MattK <[email protected]> wrote: > >> Pure Go Parquet libraries are currently a bit slower than we need, and >> using Arrow might be an option. >> >> Are there any examples of using Arrow in Go to create and write Parquet >> in code where the data has already been read and typed? >> > > like Wes, I am not aware of any body of work around this. > > out of curiosity, what were the Parquet packages you tried? > > at some point I tried to merge go-parquet with Apache Arrow: > - https://github.com/xitongsys/parquet-go/issues/91 > > was it this package? > > -s >
