Hi Julien, Thanks for the response. Yes, the context here is c++ and an otherwise flat table where each field is intuitively an option type and therefore might be null. I'll take a look at the link you sent, but in the meantime is there any reason I would need the valid_bits args from WriteBatchSpaced?
-Mike -----Original Message----- From: Julien Le Dem [mailto:[email protected]] Sent: Monday, May 15, 2017 14:02 To: [email protected] Subject: Re: representing NA values Hi Mike, Is this a C++ question? Optional in the schema means it can be null/missing. It usually translate into the definition level being 0 (null) or 1 (defined) which is also how it is represented in an Arrow validity vector. Definition levels explanation here: https://urldefense.proofpoint.com/v2/url?u=https-3A__blog.twitter.com_2013_dremel-2Dmade-2Dsimple-2Dwith-2Dparquet&d=DwIBaQ&c=f5Q7ov8zryUUIGT55zpGgw&r=p7uiAfJkXEwbVhZPqB-VxtsgxuGNpO5tGgnMUX3wqrPAIvdxhcKmn9kvZiXDziBQ&m=NNFEsYUgGnQHXAMWu_9a4omzJBkc5l-CtKlUWtzILgc&s=dfPqlmKc5MUdMOdmn22lxpMtf3J3NMcsGMirSWI44aA&e= On Mon, May 15, 2017 at 7:39 AM, Katelman, Michael < [email protected]> wrote: > Hi, > > I was wondering if someone could help me understand what the correct > way is to represent and write out NA values using a TypedColumnWriter. > Basically, what I want to do is, e.g., repeatedly write out batches of > doubles where some of them may arbitrarily be None's. It seemed like > maybe some combination of OPTIONAL, definition levels, and valid bits > might work, but it wasn't exactly clear to me. So, if someone could > point me in the right direction I would appreciate it. > > -Mike > > > > > > DISCLAIMER: This e-mail message and any attachments are intended > solely for the use of the individual or entity to which it is > addressed and may contain information that is confidential or legally > privileged. If you are not the intended recipient, you are hereby > notified that any dissemination, distribution, copying or other use of > this message or its attachments is strictly prohibited. If you have > received this message in error, please notify the sender immediately > and permanently delete this message and any attachments. > > > > -- Julien DISCLAIMER: This e-mail message and any attachments are intended solely for the use of the individual or entity to which it is addressed and may contain information that is confidential or legally privileged. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, copying or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately and permanently delete this message and any attachments.
