zratkai commented on code in PR #1431:
URL: https://github.com/apache/orc/pull/1431#discussion_r1129262725
##########
java/core/src/java/org/apache/orc/impl/TreeReaderFactory.java:
##########
@@ -2292,10 +2292,12 @@ private void readDictionaryStream(InStream in) throws
IOException {
int dictionaryBufferSize =
dictionaryOffsets[dictionaryOffsets.length - 1];
dictionaryBuffer = new byte[dictionaryBufferSize];
int pos = 0;
- int chunkSize = in.available();
- byte[] chunkBytes = new byte[chunkSize];
+ //check if dictionary size is smaller than available stream size
+ // to avoid ArrayIndexOutOfBoundsException
+ int readSize = Math.min(in.available(), dictionaryBufferSize);
+ byte[] chunkBytes = new byte[readSize];
while (pos < dictionaryBufferSize) {
- int currentLength = in.read(chunkBytes, 0, chunkSize);
+ int currentLength = in.read(chunkBytes, 0, readSize);
System.arraycopy(chunkBytes, 0, dictionaryBuffer, pos,
currentLength);
pos += currentLength;
Review Comment:
@guiyanakuang ok, I understand it now. I combined my changes with your
suggestions. Combining this two reduces the memory usage also. With that this
cases are covered:
Case 1:
in.available() = 4096
dictionaryBuffer = 900
readSize = 900
It creates a chunkBytes array only with 900 bytes (instead of 4096 bytes)
and reads 900 bytes in one run.
Case2:
in.available() = 4k
dictionaryBufferSize = 5k
readSize = 4k
It creates a chunkBytes with 4kB and reads 4kB at first run, then it reads
1kB and copies the remaining 1kB to the dictionaryBuffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
