[
https://issues.apache.org/jira/browse/IO-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210212#comment-13210212
]
Lars Kolb edited comment on IO-288 at 2/17/12 12:50 PM:
--------------------------------------------------------
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
Text str= new Text();
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
ReversedLinesReader reader= new ReversedLinesReader(is);
int bytesConsumed;
while((bytesConsumed=reader.readLine(str))>0)
{
...
}
public class ReversedLinesReader
{
//constructor takes (positioned) Input stream
public ReversedLinesReader(InputStream is)
{
//do not seek to end of file, simply start reading from is
}
//current behaviour
public ReversedLinesReader(File file)
{
//seek to end of file
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair<String,Integer> to not depend on
org.apache.hadoop.io.Text
}
}
{code}
was (Author: weihnachtsmann):
*
[Web|http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=log]
* [SVN|http://hadoop.apache.org/common/version_control.html]
Basically, it would be required to support:
{code}
Text str= new Text();
FSDataInputStream is= FileSystem.get(conf);
is.seek(offset);
ReversedLinesReader reader= new ReversedLinesReader(is);
int bytesConsumed;
while((bytesConsumed=reader.readLine(str))>0)
{
...
}
public class ReversedLinesReader
{
//constructor takes (positioned) Input stream
public ReversedLinesReader(InputStream is)
{
//do not seek to end of file, simply start reading from is
}
//constructor takes (positioned) Input stream
public ReversedLinesReader(File file)
{
//seek to end of file
}
public int readLine(Text text)
{
//return bytes read and store line in text
//alternatively one could return a Pair<String,Integer> to not depend on
org.apache.hadoop.io.Text
}
}
{code}
> Supply a ReversedLinesFileReader
> ---------------------------------
>
> Key: IO-288
> URL: https://issues.apache.org/jira/browse/IO-288
> Project: Commons IO
> Issue Type: New Feature
> Components: Utilities
> Reporter: Georg Henzler
> Fix For: 2.2
>
> Attachments: ReversedLinesFileReader0.3.zip
>
>
> I needed to analyse a log file today and I was looking for a
> ReversedLinesFileReader: A class that behaves exactly like BufferedReader
> except that it goes from bottom to top when readLine() is called. I didn't
> find it in IOUtils and the internet didn't help a lot either, e.g.
> http://www.java2s.com/Tutorial/Java/0180__File/ReversingaFile.htm is a fairly
> inefficient - the log files I'm analysing are huge and it is not a good idea
> to load the whole content in the memory.
> So I ended up writing an implementation myself using little memory and the
> class RandomAccessFile - see attached file. It's used as follows:
> int blockSize = 4096; // only that much memory is needed, no matter how big
> the file is
> ReversedLinesFileReader reversedLinesFileReader = new ReversedLinesFileReader
> (myFile, blockSize, "UTF-8"); // encoding is supported
> String line = null;
> while((line=reversedLinesFileReader.readLine())!=null) {
> ... // use the line
> if(enoughLinesSeen) {
> break;
> }
> }
> reversedLinesFileReader.close();
> I believe this could be useful for other people as well!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
