Hi Paul, Yes, we are planning to support it eventually but I estimate there is at least 100 hours of developer time involved in doing it thoroughly and with both write and read support, and with good performance. There is a patch available implementing reads (but not writes yet), but it has performance regressions and so hasn't been merged until those are sorted out.
If you are able to contribute in some way it would be appreciated. Thanks On Fri, Nov 9, 2018, 4:18 AM Paul Wais <p...@getcruise.com.invalid wrote: > Dear List, > > I'm writing about the ability to write nested data via pyarrow, as > discussed in this StackOverflow posting: > > https://stackoverflow.com/questions/45341182/nested-data-in-parquet-with-python > > I looked around JIRA and some of the PRs and I'm a bit confused: is > pyarrow supposed to support writing nested data? Or is there planned > support? I have tried writing such data using pyarrow and I get an > exception; if I write the same data to Parquet using pyspark, the job > completes successfully. I was hoping to use pyarrow for my task > though. > > Cheers, > -Paul > > -- > > > *Confidentiality Note:* We care about protecting our proprietary > information, confidential material, and trade secrets. This message may > contain some or all of those things. Cruise will suffer material harm if > anyone other than the intended recipient disseminates or takes any action > based on this message. If you have received this message (including any > attachments) in error, please delete it immediately and notify the sender > promptly. >
