On Fri, Feb 18, 2022 at 2:06 PM Antoine Pitrou <anto...@python.org> wrote:
> > Le 18/02/2022 à 20:01, Phillip Cloud a écrit : > > I think I'm confused by where this appended value lives. Is it only a > > logical value or does the value show up in memory? > > The logical value is null. The appended value is only a physical value > that shows up in memory but doesn't have any bearing on the logical value. > Yes, but where does that value reside? Does it depend on the array type? Is it garbage in the values buffer? Something else? > > > For example, appending another null to the name field is only going to > > change the validity map, offsets array and length and there will not be > any > > changes the values buffer. > > I may be missing some context, but what is the "name field" here? > The field in the example in the spec: https://arrow.apache.org/docs/format/Columnar.html#struct-layout > (if we are talking about a child string array, then the rules are the > same as for any string array) > This seems to conflict with what you said above, or whether there's actually any additional memory used depends on the array's element type: > The appended value is only a physical value that shows up in memory but doesn't have any bearing on the logical value. If that were the case, I would expect garbage in between "joe" and "mark" in the values array from the example (the garbage being the physical value not having any bearing on the logical value). > > Regards > > Antoine. >
