[
https://issues.apache.org/jira/browse/TAJO-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238799#comment-14238799
]
ASF GitHub Bot commented on TAJO-1235:
--------------------------------------
Github user hyunsik commented on a diff in the pull request:
https://github.com/apache/tajo/pull/289#discussion_r21501517
--- Diff:
tajo-storage/src/main/java/org/apache/tajo/storage/text/ByteBufLineReader.java
---
@@ -120,24 +118,34 @@ private void fillBuffer() throws IOException {
* Read a line terminated by one of CR, LF, or CRLF.
*/
public ByteBuf readLineBuf(AtomicInteger reads) throws IOException {
- if(eof) return null;
-
- int startIndex = buffer.readerIndex();
- int readBytes;
+ int readBytes = 0;
+ int newlineLength = 0; //length of terminating newline
int readable;
- int newlineLength; //length of terminating newline
+
+ this.startIndex = buffer.readerIndex();
loop:
while (true) {
readable = buffer.readableBytes();
if (readable <= 0) {
- buffer.readerIndex(startIndex);
+ buffer.readerIndex(this.startIndex);
fillBuffer(); //compact and fill buffer
- if (!buffer.isReadable()) {
+ if (!buffer.isReadable() && buffer.writerIndex() == 0) {
--- End diff --
Could you add comments about this case?
> ByteBufLineReader can not read text line with CRLF
> --------------------------------------------------
>
> Key: TAJO-1235
> URL: https://issues.apache.org/jira/browse/TAJO-1235
> Project: Tajo
> Issue Type: Bug
> Components: storage
> Affects Versions: 0.9.1
> Reporter: Jinho Kim
> Assignee: Jinho Kim
> Fix For: 0.9.1
>
> Attachments: TAJO-1235.patch, TAJO-1235_2.patch
>
>
> Here is the problem example
> {noformat}
> [text .. \r] [\n text ..]
> buffer buffer
> {noformat}
> {noformat}
> 2014-12-05 13:46:35,355 ERROR org.apache.tajo.worker.Task: maxCapacity: -1
> (expected: >= 0)
> java.lang.IllegalArgumentException: maxCapacity: -1 (expected: >= 0)
> at io.netty.buffer.AbstractByteBuf.<init>(AbstractByteBuf.java:51)
> at
> io.netty.buffer.AbstractDerivedByteBuf.<init>(AbstractDerivedByteBuf.java:28)
> at io.netty.buffer.SlicedByteBuf.<init>(SlicedByteBuf.java:40)
> at io.netty.buffer.AbstractByteBuf.slice(AbstractByteBuf.java:938)
> at
> org.apache.tajo.storage.text.ByteBufLineReader.readLineBuf(ByteBufLineReader.java:168)
> at
> org.apache.tajo.storage.text.DelimitedLineReader.readLine(DelimitedLineReader.java:125)
> at
> org.apache.tajo.storage.text.DelimitedTextFile$DelimitedTextFileScanner.init(DelimitedTextFile.java:328)
> at
> org.apache.tajo.engine.planner.physical.SeqScanExec.initScanner(SeqScanExec.java:226)
> at
> org.apache.tajo.engine.planner.physical.SeqScanExec.init(SeqScanExec.java:200)
> at
> org.apache.tajo.engine.planner.physical.UnaryPhysicalExec.init(UnaryPhysicalExec.java:53)
> at
> org.apache.tajo.engine.planner.physical.UnaryPhysicalExec.init(UnaryPhysicalExec.java:53)
> at
> org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.init(HashShuffleFileWriteExec.java:83)
> at org.apache.tajo.worker.Task.run(Task.java:447)
> at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:276)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
