[
https://issues.apache.org/jira/browse/TAJO-1250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249354#comment-14249354
]
ASF GitHub Bot commented on TAJO-1250:
--------------------------------------
Github user jihoonson commented on a diff in the pull request:
https://github.com/apache/tajo/pull/303#discussion_r21949532
--- Diff: tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java ---
@@ -272,6 +272,10 @@ public static int setDateOrder(int dateOrder) {
// Geo IP
GEOIP_DATA("tajo.function.geoip-database-location", ""),
+ // Storage IO BUFFER
+ STORAGE_IO_WRITE_BUFFER_SIZE("tajo.io.write.buffer.size", 128 * 1024),
+ STORAGE_IO_READ_BUFFER_SIZE("tajo.io.read.buffer.size", 128 * 1024),
--- End diff --
They seem buffer sizes for HDFS.
I think that we will support various types of storages.
In addition, it would be more intuititive if it represents the size unit
such as KB.
So, how about change the property name such as
tajo.hdfs.write.buffer.size.KB?
> RawFileAppender occasionally causes BufferOverflowException
> -----------------------------------------------------------
>
> Key: TAJO-1250
> URL: https://issues.apache.org/jira/browse/TAJO-1250
> Project: Tajo
> Issue Type: Bug
> Components: data shuffle, storage
> Affects Versions: 0.8.0, 0.9.0
> Reporter: Jinho Kim
> Assignee: Jinho Kim
> Fix For: 0.9.1
>
>
> The main problem is the maximum bytes size of variant int64.
> if a value is long.max, varint64 length is 10 bytes .
> * Variant int32 : 1~5 bytes
> * Variant int64 : 1~10 bytes
> {noformat}
> 2014-12-15 13:25:47,908 ERROR
> org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec:
> java.nio.BufferOverflowException
> at java.nio.Buffer.nextPutIndex(Buffer.java:513)
> at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:291)
> at
> org.apache.tajo.storage.RawFile$RawFileAppender.writeRawVarint64(RawFile.java:604)
> at
> org.apache.tajo.storage.RawFile$RawFileAppender.addTuple(RawFile.java:659)
> at
> org.apache.tajo.storage.HashShuffleAppender.addTuples(HashShuffleAppender.java:92)
> at
> org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.next(HashShuffleFileWriteExec.java:137)
> at org.apache.tajo.worker.Task.run(Task.java:407)
> at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:276)
> at java.lang.Thread.run(Thread.java:745)
> 2014-12-15 13:25:47,910 ERROR org.apache.tajo.worker.Task:
> java.nio.BufferOverflowException
> java.io.IOException: java.nio.BufferOverflowException
> at
> org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.next(HashShuffleFileWriteExec.java:152)
> at org.apache.tajo.worker.Task.run(Task.java:407)
> at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:276)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.BufferOverflowException
> at java.nio.Buffer.nextPutIndex(Buffer.java:513)
> at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:291)
> at
> org.apache.tajo.storage.RawFile$RawFileAppender.writeRawVarint64(RawFile.java:604)
> at
> org.apache.tajo.storage.RawFile$RawFileAppender.addTuple(RawFile.java:659)
> at
> org.apache.tajo.storage.HashShuffleAppender.addTuples(HashShuffleAppender.java:92)
> at
> org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.next(HashShuffleFileWriteExec.java:137)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
