Re: [DISCUSS][Java] Should null values in VariableWidthVector/ListVector always takes 0 space?

2019-09-03 Thread Fan Liya
Hi Wes, Thanks for the effort. I will add clarifications. Best, Liya Fan On Wed, Sep 4, 2019 at 11:06 AM Wes McKinney wrote: > I opened https://issues.apache.org/jira/browse/ARROW-6451 > > On Sun, Sep 1, 2019 at 9:59 PM Fan Liya wrote: > > > > Hi Wes, > > > > Thanks for the information. > >

Re: [DISCUSS][Java] Should null values in VariableWidthVector/ListVector always takes 0 space?

2019-09-03 Thread Wes McKinney
I opened https://issues.apache.org/jira/browse/ARROW-6451 On Sun, Sep 1, 2019 at 9:59 PM Fan Liya wrote: > > Hi Wes, > > Thanks for the information. > I agree with you that we had better make this clear in the document, to > help users avoid unexpected behaviors. > > Best, > Liya Fan > > On Sun,

Re: [DISCUSS][Java] Should null values in VariableWidthVector/ListVector always takes 0 space?

2019-09-01 Thread Fan Liya
Hi Wes, Thanks for the information. I agree with you that we had better make this clear in the document, to help users avoid unexpected behaviors. Best, Liya Fan On Sun, Sep 1, 2019 at 7:17 AM Wes McKinney wrote: > Option 3 is the what the columnar specification currently intends, for > the

Re: [DISCUSS][Java] Should null values in VariableWidthVector/ListVector always takes 0 space?

2019-08-31 Thread Wes McKinney
Option 3 is the what the columnar specification currently intends, for the reasons that Jacques cites. In particular, a value can be made null only by altering the validity bitmap. It might be helpful to add some language to make clear that the contents "underneath" a null can be anything. The

Re: [DISCUSS][Java] Should null values in VariableWidthVector/ListVector always takes 0 space?

2019-08-28 Thread Fan Liya
Hi Jacques and Ravindra, Thanks for your valuable feedback. Please let me talk more about contiguous memory: For some operations (like memory segment comparison, hash code computation, etc.), if we we chose option 1 or 2, we can get the result with a single call, without any reference to the

Re: [DISCUSS][Java] Should null values in VariableWidthVector/ListVector always takes 0 space?

2019-08-28 Thread Ravindra Pindikura
On Wed, Aug 28, 2019 at 12:32 PM Fan Liya wrote: > Dear all, > > In the discussion of this PR (https://github.com/apache/arrow/pull/5073), > we are faced with a problem: > > Normally, in a VariableWidthVector (e.g. VarCharVector), a null value is > supposed to take no space in the data buffer.

Re: [DISCUSS][Java] Should null values in VariableWidthVector/ListVector always takes 0 space?

2019-08-28 Thread Jacques Nadeau
#3 is the correct behavior and how the code was meant to be written. I don't see any problems with that pattern. This allows someone to (if they so decide) to null a value without having to rewrite the data. #3 is also a consistent behavior with all other vectors. Null values can use up space but

[DISCUSS][Java] Should null values in VariableWidthVector/ListVector always takes 0 space?

2019-08-28 Thread Fan Liya
Dear all, In the discussion of this PR (https://github.com/apache/arrow/pull/5073), we are faced with a problem: Normally, in a VariableWidthVector (e.g. VarCharVector), a null value is supposed to take no space in the data buffer. In particular, for a null value, we have start index == end