Alex Black created ARROW-3175: --------------------------------- Summary: Arrow Java: Upgrade to official FlatBuffers release (Flatbuffers incompatibility) Key: ARROW-3175 URL: https://issues.apache.org/jira/browse/ARROW-3175 Project: Apache Arrow Issue Type: Bug Components: Java Affects Versions: 0.10.0 Reporter: Alex Black
Arrow Java currently uses an unofficial flatbuffers dependency - com.vlkan:flatbuffers: [https://github.com/apache/arrow/blob/master/java/pom.xml#L481-L485] The likely motivation here is that previously, no Java flatbuffers implementation was available on maven central. [https://github.com/vy/flatbuffers] > Unfortunately, FlatBuffers project does not publish any artifacts to the > Maven Central Repository However, this is no longer the case: [https://search.maven.org/search?q=g:com.google.flatbuffers%20AND%20a:flatbuffers-java&core=gav] The flatbuffers version used in Arrow java is a nearly 3-year-old snapshot, not even a version of an official release: [https://github.com/vy/flatbuffers#usage] The main problem is that this version of flatbuffers is not compatible with the official releases of flatbuffers. For example, we use the official flatbuffers releases in ND4J and Deeplearning4j: [https://github.com/deeplearning4j/deeplearning4j] Running Arrow with an official flatbuffers library on the classpath results in issues such as: {noformat} java.lang.NoSuchMethodError: com.google.flatbuffers.FlatBufferBuilder.createString(Ljava/lang/String;)I at org.apache.arrow.vector.types.pojo.Field.getField(Field.java:154) at org.apache.arrow.vector.types.pojo.Schema.getSchema(Schema.java:145) at org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:124) at org.apache.arrow.vector.ipc.ArrowWriter.ensureStarted(ArrowWriter.java:136) at org.apache.arrow.vector.ipc.ArrowWriter.start(ArrowWriter.java:97) at FlatBuffersDependencyIssue.test(FlatBuffersDependencyIssue.java:56) {noformat} Simply excluding the com.vlkan:flatbuffers dependency in lieu of an official flatbuffers release is not a solution (same exception as above) and we aren't prepared to downgrade all of our projects to use the flatbuffers version that Arrow currently requires. Consequently, this is a major issue that prevents us using Arrow in our libraries. I have prepared a simple repository to reproduce this issue, if required: [https://github.com/AlexDBlack/arrowflatbufferstest] Is there a reason for using this particular version of flatbuffers, and if not, can Arrow java use an official release of flatbuffers instead? -- This message was sent by Atlassian JIRA (v7.6.3#76005)