Hi Mike, Is this a C++ question? Optional in the schema means it can be null/missing. It usually translate into the definition level being 0 (null) or 1 (defined) which is also how it is represented in an Arrow validity vector. Definition levels explanation here: https://blog.twitter.com/2013/dremel-made-simple-with-parquet
On Mon, May 15, 2017 at 7:39 AM, Katelman, Michael < [email protected]> wrote: > Hi, > > I was wondering if someone could help me understand what the correct way > is to represent and write out NA values using a TypedColumnWriter. > Basically, what I want to do is, e.g., repeatedly write out batches of > doubles where some of them may arbitrarily be None's. It seemed like maybe > some combination of OPTIONAL, definition levels, and valid bits might work, > but it wasn't exactly clear to me. So, if someone could point me in the > right direction I would appreciate it. > > -Mike > > > > > > DISCLAIMER: This e-mail message and any attachments are intended solely > for the use of the individual or entity to which it is addressed and may > contain information that is confidential or legally privileged. If you are > not the intended recipient, you are hereby notified that any dissemination, > distribution, copying or other use of this message or its attachments is > strictly prohibited. If you have received this message in error, please > notify the sender immediately and permanently delete this message and any > attachments. > > > > -- Julien
