Hi Wes,
Thanks for your good suggestion, we will think about it.
Hi Jacques,
Thanks for your feedback. Please allow me to address some of your concerns:
> This change is undesirable as it optimizes one path and makes several
others behave in unintended ways.
Could you please elaborate more on
I agree with Jacques here. Perhaps what is needed is an unsafe
non-nullable array accessor layer, then there is no need for flags
etc. We've already been writing a lot of such code in C++ (splitting
between no-nulls and some-nulls paths, see also the BitBlockCounter
stuff we've been doing, is such
This change is undesirable as it optimizes one path and makes several
others behave in unintended ways. What happens if a vector with nulls
shows up? What happens if a user sets a position to a null value in user
code when this flag set?
If the answer to the above questions is the use is an advan
Hi all,
Thanks a lot for your previous feedback.
Now we have made some investigation and prepared an initial PR supporting
the non-nullable IntVector [1], as this represents a common scenario.
Some initial observations and conclusions can be made.
The basic idea of the PR is to provide a global
Hi Jacques,
Thanks a lot for your valuable comments.
I agree with you that collapsing nullable and non-nullable implementations
is a good idea, and it does not contradict with the idea of introducing a
fast code path, if it does not introduce much cost or code complexity.
The idea of word level
Generally Ive found that this isnt an important optimization in the use
cases we see. Memory overhead, especially with our Java shared allocation
scheme is nominal. Optimizing null checks at the word level usually is much
more impactful since non null and null runs are much more common on a
shorter
> And there is a "nullable" metadata-only flag at the
> Field level. Could the same kinds of optimizations be implemented in
> Java without introducing a "nullable" concept?
Note Liya Fan did suggest pulling the nullable flag from the Field when the
vector is created in item (1) of the proposed ch
Hi Micah,
Thanks a lot for your valuable comments. Please see my comments inline.
> I'm a little concerned that this will change assumptions for at least some
> of the clients using the library (some might always rely on the validity
> buffer being present).
I can understand your concern and I a
Hi Liya Fan,
I'm a little concerned that this will change assumptions for at least some
of the clients using the library (some might always rely on the validity
buffer being present).
I think this is a good feature to have for the reasons you mentioned. It
seems like there would need to be some so
Hi Wes,
Thanks a lot for your quick reply.
I think what you mentioned is almost exactly what we want to do in Java.The
concept is not important.
Maybe there are only some minor differences:
1. In C++, the null_count is mutable, while for Java, once a vector is
constructed as non-nullable, its nul
hi Liya,
In C++ we elect certain faster code paths when the null count is 0 or
computed to be zero. When the null count is 0, we do not allocate a
validity bitmap. And there is a "nullable" metadata-only flag at the
Field level. Could the same kinds of optimizations be implemented in
Java without
11 matches
Mail list logo
