[
https://issues.apache.org/jira/browse/TAJO-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238798#comment-14238798
]
ASF GitHub Bot commented on TAJO-1235:
--------------------------------------
Github user hyunsik commented on a diff in the pull request:
https://github.com/apache/tajo/pull/289#discussion_r21501490
--- Diff:
tajo-storage/src/main/java/org/apache/tajo/storage/text/ByteBufLineReader.java
---
@@ -77,23 +74,25 @@ private void fillBuffer() throws IOException {
int tailBytes = 0;
if (this.readBytes > 0) {
+ //startIndex = 0, readIndex = tailBytes length, writable = (buffer
capacity - tailBytes)
this.buffer.markReaderIndex();
- this.buffer.discardSomeReadBytes(); // compact the buffer
+ this.buffer.discardReadBytes(); // compact the buffer
tailBytes = this.buffer.writerIndex();
if (!this.buffer.isWritable()) {
// a line bytes is large than the buffer
- BufferPool.ensureWritable(buffer, bufferSize);
+ BufferPool.ensureWritable(buffer, bufferSize * 2);
this.bufferSize = buffer.capacity();
}
+ this.startIndex = 0;
}
boolean release = true;
try {
int readBytes = tailBytes;
for (; ; ) {
- int localReadBytes = buffer.writeBytes(channel, bufferSize -
readBytes);
+ int localReadBytes = buffer.writeBytes(channel, this.bufferSize -
readBytes);
if (localReadBytes < 0) {
- if (tailBytes == readBytes) {
+ if (buffer.isWritable()) {
--- End diff --
It seems that we can ensure EOF if there is no read bytes and there are
write buffer. But, it may be hard for code readers to imagine this logic. Could
you add some comments about this logic?
> ByteBufLineReader can not read text line with CRLF
> --------------------------------------------------
>
> Key: TAJO-1235
> URL: https://issues.apache.org/jira/browse/TAJO-1235
> Project: Tajo
> Issue Type: Bug
> Components: storage
> Affects Versions: 0.9.1
> Reporter: Jinho Kim
> Assignee: Jinho Kim
> Fix For: 0.9.1
>
> Attachments: TAJO-1235.patch, TAJO-1235_2.patch
>
>
> Here is the problem example
> {noformat}
> [text .. \r] [\n text ..]
> buffer buffer
> {noformat}
> {noformat}
> 2014-12-05 13:46:35,355 ERROR org.apache.tajo.worker.Task: maxCapacity: -1
> (expected: >= 0)
> java.lang.IllegalArgumentException: maxCapacity: -1 (expected: >= 0)
> at io.netty.buffer.AbstractByteBuf.<init>(AbstractByteBuf.java:51)
> at
> io.netty.buffer.AbstractDerivedByteBuf.<init>(AbstractDerivedByteBuf.java:28)
> at io.netty.buffer.SlicedByteBuf.<init>(SlicedByteBuf.java:40)
> at io.netty.buffer.AbstractByteBuf.slice(AbstractByteBuf.java:938)
> at
> org.apache.tajo.storage.text.ByteBufLineReader.readLineBuf(ByteBufLineReader.java:168)
> at
> org.apache.tajo.storage.text.DelimitedLineReader.readLine(DelimitedLineReader.java:125)
> at
> org.apache.tajo.storage.text.DelimitedTextFile$DelimitedTextFileScanner.init(DelimitedTextFile.java:328)
> at
> org.apache.tajo.engine.planner.physical.SeqScanExec.initScanner(SeqScanExec.java:226)
> at
> org.apache.tajo.engine.planner.physical.SeqScanExec.init(SeqScanExec.java:200)
> at
> org.apache.tajo.engine.planner.physical.UnaryPhysicalExec.init(UnaryPhysicalExec.java:53)
> at
> org.apache.tajo.engine.planner.physical.UnaryPhysicalExec.init(UnaryPhysicalExec.java:53)
> at
> org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.init(HashShuffleFileWriteExec.java:83)
> at org.apache.tajo.worker.Task.run(Task.java:447)
> at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:276)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
