Wes, I could do the Java side for dictionary integration testing. It looks like ARROW-693 might have stalled, I'll ping to see if any work has been started.
Thanks, Bryan On Mon, May 22, 2017 at 2:26 PM, Wes McKinney <wesmck...@gmail.com> wrote: > hi folks, > > We've come a long way in our integration testing in the last 6 months. > For those who haven't followed actively, the integration tests verify > wire compatibility between Java and C++ streaming and file binary > formats. We run these tests continuously in Travis CI, which makes it > safe for developers to refactor and improve the IPC / wire format code > without breaking cross-implementation compatibility. I expect in the > near future we will also see some JavaScript integration tests. > > The integration test suite works as follows: > > * Test data is specified in an Arrow-specific JSON format > * Each application reads the JSON and converts it to the Arrow binary > format according to its implementation > * The other application reads the binary and compares the binary to > the JSON "point of truth" file > > What is still missing from the current test suite: > > - Dictionary tests > - Fixed size list > - Decimal tests (first available in C++ in 0.3.0) > - Unions > - Intervals > > Otherwise we have pretty good coverage of the rest of the data types > in https://github.com/apache/arrow/blob/master/format/Schema.fbs; a > lot of work in Arrow 0.3.0 was reconciling the date and time types. > > For the next release 0.5.0 the most important thing to test is > dictionaries. It would be nice to test decimals also, but we have some > specification work to do as the in-memory format for Decimals in C++ > and Java is currently different. > > I should have time to implement the C++ side of the dictionary > integration tests (i.e. converting to/from the JSON format), but > someone will need to implement the Java side. It would be great if we > could get this done in the next few weeks if anyone has the time to do > it. > > Thanks > Wes >
