Agreed, that sounds like a great solution to this problem - the layout
information is redundant and it doesn't make sense to include it in
every schema.
Although I would argue we should write down exactly what buffers are
supposed to go on the wire in the dictionary batches (i.e. value
vectors) as well. This should be largely the same as what goes on the
wire in a record batch for a non dictionary-encoded vector of the same
type, but there could be a difference. For example, if a dictionary
vector is nullable, do we really need a validity buffer in both the
index and in the values? I think that's the current behavior, but maybe
it would make sense to assert that a dictionary's value vector should be
non-nullable, and nulls should be handled in the index vector?
Brian
On 11/08/2017 05:24 PM, Wes McKinney wrote:
Per Jacques' comment in ARROW-1693
https://issues.apache.org/jira/browse/ARROW-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244812#comment-16244812,
I think we should remove the buffer layout from the metadata. It would
be a good idea to do this for 0.8.0 since we're breaking the metadata
anyway.
In addition to bloating the size of the schemas on the wire, the
buffer layout metadata provides redundant information which should be
a strict part of the Arrow specification. I agree with Jacques that it
would be better to write down exactly what buffers are supposed to go
on the wire for each logical type. In the case of the dictionary
vectors, it is the buffers for the indices, so the issue under
discussion resolves itself if we nix the metadata.
If writers are emitting possibly different buffer layouts (like
omitting a null or zero-length buffer), it will introduce brittleness
and cause much special casing to trickle down into the reader
implementations. This seems like undue complexity.
- Wes
On Mon, Nov 6, 2017 at 9:33 AM, Brian Hulette <brian.hule...@ccri.com> wrote:
We've been having some integration issues with reading Dictionary Vectors in
the JS implementation - our current implementation can read arrow files and
streams generated by Java, but not by C++. Most of this discussion is
captured in ARROW-1693 [1].
It looks like ultimately the issue is that there are inconsistencies in the
way the various implementations handle buffer layouts for dictionary-encoded
vectors in the Schema message. Some places write/read the buffer layout for
the value vector (the vector found in the dictionary batch), and others
expect the layout for the index vector (the int vector found in the record
batch). Both the Java and C++ IPC readers don't seem to care about this
portion of the Schema, which explains why the integration tests are passing.
Here's a fun ASCII table of how I think the Java/C++/JS IPC readers and
writers handle those buffers layouts right now:
| Writer | Reader
-----+--------------+-------------
Java | value vector | doesn't care
C++ | index vector | doesn't care
JS | N/A | value vector
Note that I can only really speak with authority about the JS
implementation. I'd appreciate it if people more familiar with the other two
could validate my claims.
As far as I can tell the expected behavior isn't stated anywhere in the
documentation, which I suppose explains the inconsistency. Paul Taylor is
currently working on resolving ARROW-1693 by making the JS reader ambivalent
to buffer layout, but I think ultimately the correct solution is to agree on
a consistent standard, and make the reader implementations opinionated about
the Schema buffer layouts (i.e. ARROW-1362 [2]).
Personally, I don't really have an opinion either way about which vector's
layout should be in the Schema. Either way we'll be missing some layout
information though, so we should also consider where the information for the
"other" vector might go.
I know there's a release coming up, and now is probably not the time to
tackle this problem, but I wanted to write it up while its fresh in my mind.
I'm fine shelving it until after 0.8.
Brian
[1] https://issues.apache.org/jira/browse/ARROW-1693
[2] https://issues.apache.org/jira/browse/ARROW-1362
