[
https://issues.apache.org/jira/browse/HIVE-25190?focusedWorklogId=613503&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-613503
]
ASF GitHub Bot logged work on HIVE-25190:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 22/Jun/21 14:10
Start Date: 22/Jun/21 14:10
Worklog Time Spent: 10m
Work Description: pgaref commented on a change in pull request #2408:
URL: https://github.com/apache/hive/pull/2408#discussion_r656260598
##########
File path:
storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/BytesColumnVector.java
##########
@@ -121,30 +125,30 @@ public void setRef(int elementNum, byte[] sourceBuf, int
start, int length) {
* Provide the estimated number of bytes needed to hold
* a full column vector worth of byte string data.
*
- * @param estimatedValueSize Estimated size of buffer space needed
+ * @param estimatedValueSize Estimated size of buffer space needed per row
*/
public void initBuffer(int estimatedValueSize) {
- nextFree = 0;
smallBufferNextFree = 0;
// if buffer is already allocated, keep using it, don't re-allocate
- if (buffer != null) {
+ if (smallBuffer != null) {
// Free up any previously allocated buffers that are referenced by vector
if (bufferAllocationCount > 0) {
for (int idx = 0; idx < vector.length; ++idx) {
vector[idx] = null;
length[idx] = 0;
}
- buffer = smallBuffer; // In case last row was a large bytes value
}
} else {
// allocate a little extra space to limit need to re-allocate
- int bufferSize = this.vector.length * (int)(estimatedValueSize *
EXTRA_SPACE_FACTOR);
+ long bufferSize = (long) (this.vector.length * estimatedValueSize *
EXTRA_SPACE_FACTOR);
if (bufferSize < DEFAULT_BUFFER_SIZE) {
bufferSize = DEFAULT_BUFFER_SIZE;
}
- buffer = new byte[bufferSize];
- smallBuffer = buffer;
+ if (bufferSize > MAX_SIZE_FOR_SMALL_BUFFER) {
Review comment:
is this condition valid? Would we have enough space to store the new
element when > MAX_SIZE_FOR_SMALL_BUFFER?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 613503)
Time Spent: 0.5h (was: 20m)
> BytesColumnVector fails when the aggregate size is > 1gb
> --------------------------------------------------------
>
> Key: HIVE-25190
> URL: https://issues.apache.org/jira/browse/HIVE-25190
> Project: Hive
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently, BytesColumnVector will allocate a buffer for small values (< 1mb),
> but fail with:
> {code:java}
> new RuntimeException("Overflow of newLength. smallBuffer.length="
> + smallBuffer.length + ", nextElemLength=" + nextElemLength);
> {code:java}
> if the aggregate size of the buffer crosses over 1gb.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
