Appreciate the help Jacques. Unfortunately calling setPosition(0) on the writer for the list did not solve it.
I put the entirety of the code up on pastebin so it should be an easy copy/paste if anybody else wants to try to reproduce it. I suppose it could also be a bug in VectorAppender, but again I'm not sure if the error is in my code or in Arrow. https://pastebin.com/vwvnYY40 Thanks in advance. On Mon, Feb 15, 2021 at 1:33 PM Jacques Nadeau <[email protected]> wrote: > I think you need to call setPosition(0) before you start writing the list. > (This is from memory when I wrote the code 6-7 years ago so I may be off.) > > On Sun, Feb 14, 2021 at 6:20 PM John Peterson < > [email protected]> wrote: > >> Hi Bryan, >> >> This is the stacktrace I get: >> >> java.lang.IndexOutOfBoundsException: index: 0, length: 4 (expected: >> range(0, 0)) >> at org.apache.arrow.memory.ArrowBuf.checkIndexD(ArrowBuf.java:318) >> at org.apache.arrow.memory.ArrowBuf.chk(ArrowBuf.java:305) >> at org.apache.arrow.memory.ArrowBuf.getInt(ArrowBuf.java:424) >> at >> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:97) >> at >> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:45) >> at >> org.apache.arrow.vector.BaseVariableWidthVector.accept(BaseVariableWidthVector.java:1402) >> at >> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:233) >> at >> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:45) >> at org.apache.arrow.vector.complex.ListVector.accept(ListVector.java:449) >> at >> org.apache.arrow.vector.util.VectorSchemaRootAppender.append(VectorSchemaRootAppender.java:67) >> at >> org.apache.arrow.vector.util.VectorSchemaRootAppender.append(VectorSchemaRootAppender.java:81) >> >> Thanks for your help. >> >> On Thu, Jan 14, 2021 at 2:23 PM Bryan Cutler <[email protected]> wrote: >> >>> Hi John, could you include the error with stacktrace? >>> >>> On Sat, Jan 9, 2021 at 9:34 PM John Peterson < >>> [email protected]> wrote: >>> >>>> I believe I'm running into a bug with Flight but I'd like to confirm >>>> and get some advice on a potential fix. I'm not sure where to look or what >>>> could be causing it. >>>> >>>> The code in question simply uploads a one-element List<Integer> to the >>>> example server, fetches it from the server, and attempts to append the data >>>> from the server to a new VectorSchemaRoot. It fails in the same way >>>> regardless of whether or not I construct a VectorSchemaRoot instance. >>>> >>>> Likewise, the data from the server can't be written out with the JSON >>>> writer, it'll fail in the same way. However, changing the data from a >>>> ListVector to an IntVector causes it to succeed. >>>> >>>> Any help would be appreciated. >>>> >>>> Thanks, >>>> John >>>> >>>> Code in question: >>>> // Set up the server and client >>>> BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE); >>>> Location l = Location.forGrpcInsecure(FlightTestUtil.LOCALHOST, 12233); >>>> ExampleFlightServer server = new ExampleFlightServer(allocator, l); >>>> server.start(); >>>> FlightClient client = FlightClient.builder(allocator, l).build(); >>>> >>>> // Write a one-element List<Integer> >>>> ListVector listVector = ListVector.empty("list", allocator); >>>> UnionListWriter writer = listVector.getWriter(); >>>> writer.startList(); >>>> writer.integer().writeInt(1); >>>> writer.endList(); >>>> writer.setValueCount(1); >>>> >>>> // Send that data to the server >>>> VectorSchemaRoot root = VectorSchemaRoot.of(listVector); >>>> ClientStreamListener listener = >>>> client.startPut(FlightDescriptor.path("test"), root, new >>>> AsyncPutListener()); >>>> root.setRowCount(1); >>>> listener.putNext(); >>>> root.clear(); >>>> listener.completed(); >>>> >>>> // wait for ack to avoid memory leaks. >>>> listener.getResult(); >>>> >>>> // Attempt to read it back >>>> FlightInfo info = client.getInfo(FlightDescriptor.path("test")); >>>> try (final FlightStream stream = >>>> client.getStream(info.getEndpoints().get(0).getTicket())) { >>>> VectorSchemaRoot newRoot = stream.getRoot(); >>>> while (stream.next()) { >>>> // Copying into an entirely new VectorSchemaRoot fails >>>> try { >>>> ListVector newList = ListVector.empty("list", allocator); >>>> >>>> newList.addOrGetVector(FieldType.nullable(Types.MinorType.INT.getType())); >>>> VectorSchemaRoot copyRoot = VectorSchemaRoot.of(newList); >>>> VectorSchemaRootAppender.append(copyRoot, newRoot); >>>> } catch (IndexOutOfBoundsException e) { >>>> System.err.println("Expected IOOBE caught"); >>>> } >>>> >>>> // The same is true if we try to copy the data from the server to >>>> our VectorSchemaRoot >>>> try { >>>> VectorSchemaRootAppender.append(root, newRoot); >>>> } catch (IndexOutOfBoundsException e) { >>>> System.err.println("Expected IOOBE caught again"); >>>> throw e; >>>> } >>>> >>>> root.clear(); >>>> newRoot.clear(); >>>> } >>>> } >>>> >>>
