phogh opened a new issue, #1383: URL: https://github.com/apache/incubator-fury/issues/1383
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-fury/issues) and found no similar issues. ### Version Fury v0.4.1, Java JDK 8, Ubuntu 20.04 ### Component(s) Java ### Minimal reproduce step test.dat.gz is a gzipped file containing many serialized objects of type MyClass. I don't know how many objects are in the file so I loop through the file while input is available to deserialize them as below. However as `Fury#readToBufferFromStream` assumes complete reads of the serialized size, an exception is thrown. This "slow" inputstream issue was partially resolved in PR https://github.com/apache/incubator-fury/pull/1034. ```java int i = 0; try (InputStream input = new GZIPInputStream(new FileInputStream("test.dat.gz"))) { while (input.available() > 0) { ++i; tick = (MyClass) fury.deserialize(input); } } catch (Exception e) { System.out.printf("Exception for object #%d%n", i); throw e; } ``` ### What did you expect to see? All objects in the gzipped file should deserialize. ### What did you see instead? java.lang.IllegalArgumentException at io.fury.util.Preconditions.checkArgument(Preconditions.java:47) at io.fury.Fury.readToBufferFromStream(Fury.java:1209) at io.fury.Fury.deserialize(Fury.java:766) at io.fury.Fury.deserialize(Fury.java:760) [...] ### Anything Else? The slow InputStream issue addressed in PR https://github.com/apache/incubator-fury/pull/1034 could be extended to allow partial reads of the serialized size in the same way the PR fixed this for the serialized object read. Here's my attempt to fix `Fury#readToBufferFromStream`: ```java private static void readToBufferFromStream(InputStream inputStream, MemoryBuffer buffer) throws IOException { buffer.readerIndex(0); int read = readBytes(inputStream, buffer.getHeapMemory(), 0, 4); Preconditions.checkArgument(read == 4); int size = buffer.readInt(); buffer.ensure(4 + size); read = readBytes(inputStream, buffer.getHeapMemory(), 4, size); Preconditions.checkArgument(read == size); } private static int readBytes(InputStream inputStream, byte[] buffer, int offset, int size) throws IOException { int read = 0; int count = 0; while (read < size) { if ((count = inputStream.read(buffer, offset + read, size - read)) == -1) { break; } read += count; } return (read == 0 && count == -1) ? -1 : read; } ``` ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
