hi Mike, I think you want to use WriteBatch on TypedColumnWriter:
https://github.com/apache/parquet-cpp/blob/master/src/parquet/column/writer.h#L166 For a flat table with an optional repetition type, the definition levels are a sequence of 1's and 0's, where 1 is for non-null values. The array of values does not include nulls. We have an additional API WriteBatchSpaced for value arrays that include null slots. We use this in the Arrow writer: https://github.com/apache/parquet-cpp/blob/master/src/parquet/arrow/writer.cc#L391 You can also create arrow::DoubleArray and use the Arrow writer interface. Hope this helps. Wes On Mon, May 15, 2017 at 2:01 PM, Julien Le Dem <[email protected]> wrote: > Hi Mike, > Is this a C++ question? > Optional in the schema means it can be null/missing. It usually translate > into the definition level being 0 (null) or 1 (defined) which is also how > it is represented in an Arrow validity vector. > Definition levels explanation here: > https://blog.twitter.com/2013/dremel-made-simple-with-parquet > > > On Mon, May 15, 2017 at 7:39 AM, Katelman, Michael < > [email protected]> wrote: > >> Hi, >> >> I was wondering if someone could help me understand what the correct way >> is to represent and write out NA values using a TypedColumnWriter. >> Basically, what I want to do is, e.g., repeatedly write out batches of >> doubles where some of them may arbitrarily be None's. It seemed like maybe >> some combination of OPTIONAL, definition levels, and valid bits might work, >> but it wasn't exactly clear to me. So, if someone could point me in the >> right direction I would appreciate it. >> >> -Mike >> >> >> >> >> >> DISCLAIMER: This e-mail message and any attachments are intended solely >> for the use of the individual or entity to which it is addressed and may >> contain information that is confidential or legally privileged. If you are >> not the intended recipient, you are hereby notified that any dissemination, >> distribution, copying or other use of this message or its attachments is >> strictly prohibited. If you have received this message in error, please >> notify the sender immediately and permanently delete this message and any >> attachments. >> >> >> >> > > > -- > Julien
