I am taking in a JSON file that is an array of objects. I am trying to
unnest it and serialize it however memory management is very import,
meaning that the json being read and unested needs to be done through
through a stream. The given input would look like an array of json objects
looking like this:
{"a": "some \"string\" here"}
With the desired outcome looking like
{"data": "{\"a\":\"some \\\"string\\\" here\"}"}
for the reading and writing of the data object i am using the jackson
streaming library as it is the least memory intensive library fitting with
my requirements. To serialize the data i am use an chararray library called
fastUtils however the problem persists without this so it is not the
library or the serialization itself. I have also increased the buffer size
of both the pipedReader object and the char buffer but again there is no
benefit.
Current code:
Reader reader = new InputStreamReader(inputStream, StandardCharsets.UTF_8);
Writer writer = new OutputStreamWriter(outputStream,
StandardCharsets.UTF_8);
final JsonFactory jsonFactory = new JsonFactory();
final JsonGenerator jsonGenerator = jsonFactory.createGenerator(writer);
final JsonParser jsonParser = jsonFactory.createParser(reader);
jsonParser.nextValue();
jsonGenerator.writeStartObject();
jsonGenerator.writeFieldName("dataItem");
jsonGenerator.flush();
writer.flush();
writer.write(58)
PipedReader pipedReader = new PipedReader(4096);
PipedWriter pipedWriter = new PipedWriter(pipedReader);
JsonGenerator jsonGeneratorPiped = jsonFactory.createGenerator(pipedWriter);
new Thread(
new Runnable(){
public void run() {
jsonGeneratorPiped.copyCurrentStructure(jsonParser);
jsonGeneratorPiped.flush();
pipedWriter.close();
}
}
).start();
char[] buffer = new char[2048];
int len = reader.read(buffer);
char quote = '"';
char escape = '\\';
CharArrayList listBuffer;
//listBuffer = new CharArrayList();
while (len != -1) {
listBuffer = new CharArrayList(buffer,0,len);
//listBuffer.addElements(0,buffer,0,len);
for (CharListIterator iter = listBuffer.iterator(); iter.hasNext();) {
if (iter.nextChar() == quote) {
iter.set(escape);
iter.add(quote);
}
}
writer.write(listBuffer.elements(), 0, listBuffer.size());
//toCharArray uses lot of memory
//writer.write(listBuffer.toCharArray(), 0, listBuffer.size());
//listBuffer.trim();
len = reader.read(buffer);
}
writer.flush();
pipedReader.close();
jsonGeneratorPiped.close();
jsonGenerator.writeEndObject();
jsonGenerator.flush();
jsonGenerator.close();
jsonParser.close();
that is the current code i have implemented and it works fine for files
that are small, however with larger files >10mb from my testing, the stream
closes mid way through resulting in this error
java.io.IOexception: stream is closed
Any help would be greatly appreciated
--
You received this message because you are subscribed to the Google Groups
"jackson-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jackson-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jackson-user/a49847a4-be2a-4c87-9157-948fc6f387afn%40googlegroups.com.
