Thank you Wes and Joshua for the super fast response! This clarification helps immensely. I'm gonna go with Spark in the interim since I need Spark SQL for my problem anyways. On Fri, Nov 9, 2018 at 5:52 AM Joshua Storck <joshuasto...@gmail.com> wrote: > > I hope to get around to finishing my PR for reading in the next month or > so. The work is mostly complete, it's just that it didn't get merged before > the monorepo consolidation. I expect the write side of things to be easier, > though it will probably take a month or so of work with testing. > > On Fri, Nov 9, 2018 at 7:26 AM Wes McKinney <wesmck...@gmail.com> wrote: > > > Hi Paul, > > > > Yes, we are planning to support it eventually but I estimate there is at > > least 100 hours of developer time involved in doing it thoroughly and with > > both write and read support, and with good performance. There is a patch > > available implementing reads (but not writes yet), but it has performance > > regressions and so hasn't been merged until those are sorted out. > > > > If you are able to contribute in some way it would be appreciated. > > > > Thanks > > > > On Fri, Nov 9, 2018, 4:18 AM Paul Wais <p...@getcruise.com.invalid wrote: > > > > > Dear List, > > > > > > I'm writing about the ability to write nested data via pyarrow, as > > > discussed in this StackOverflow posting: > > > > > > > > https://stackoverflow.com/questions/45341182/nested-data-in-parquet-with-python > > > > > > I looked around JIRA and some of the PRs and I'm a bit confused: is > > > pyarrow supposed to support writing nested data? Or is there planned > > > support? I have tried writing such data using pyarrow and I get an > > > exception; if I write the same data to Parquet using pyspark, the job > > > completes successfully. I was hoping to use pyarrow for my task > > > though. > > > > > > Cheers, > > > -Paul > > > > > > -- > > > > > > > > > *Confidentiality Note:* We care about protecting our proprietary > > > information, confidential material, and trade secrets. This message may > > > contain some or all of those things. Cruise will suffer material harm if > > > anyone other than the intended recipient disseminates or takes any action > > > based on this message. If you have received this message (including any > > > attachments) in error, please delete it immediately and notify the sender > > > promptly. > > > > >
-- *Confidentiality Note:* We care about protecting our proprietary information, confidential material, and trade secrets. This message may contain some or all of those things. Cruise will suffer material harm if anyone other than the intended recipient disseminates or takes any action based on this message. If you have received this message (including any attachments) in error, please delete it immediately and notify the sender promptly.
