Allow CR as a line ending Allow for any tab interval from 2 to 80 Allow for preservation of TAB chars in string and character literals Process files as character streams, rather than as byte streams
This is not tested against any files sufficiently large to cause memory problems. In addition, the handling of CTRL-Z EOF markers is not tested on any systems but linux. This is potentially significant in this version because lines are being handled by the ReadLine method of BufferedReader. This handles ^Z as text both within a line and at the end-of-file.
I have determined that the handling of CTRL-Z in and at the end of files is the same under NT4, but do not know if this behaviour is consistent across Win 95/8 and 2000.
The test cases that I posted to this list with the previous version of the patch are handled identically in this version.
The changes are backward compatible with the behaviour and XML syntax of the previous version, but extend in for the new EOL handling.
Patches to FixCRLF.java are attached as patch-java.txt Patches to fixcrlf.html are attached as patch-html.txt.
Peter -- Peter B. West [EMAIL PROTECTED] http://powerup.com.au/~pbwest "Lord, to whom shall we go?"
Index: FixCRLF.java =================================================================== RCS file: /home/cvspublic/jakarta-ant/src/main/org/apache/tools/ant/taskdefs/FixCRLF.java,v retrieving revision 1.14 diff -u -u -r1.14 FixCRLF.java --- FixCRLF.java 2001/04/03 11:26:26 1.14 +++ FixCRLF.java 2001/07/12 14:19:13 @@ -53,17 +53,20 @@ */ package org.apache.tools.ant.taskdefs; - +import org.apache.tools.ant.Project; import org.apache.tools.ant.BuildException; import org.apache.tools.ant.DirectoryScanner; -import org.apache.tools.ant.Project; import org.apache.tools.ant.types.EnumeratedAttribute; import java.io.*; import java.util.*; -import java.text.*; /** + * FixCRLF.java + * + * Based on FixCR.java + * by Sam Ruby <a href="mailto:[EMAIL PROTECTED]">[EMAIL PROTECTED]</a>. + * * Task to convert text source files to local OS formatting conventions, as * well as repair text files damaged by misconfigured or misguided editors or * file transfer programs. @@ -74,7 +77,7 @@ * <li>destdir * <li>include * <li>exclude - * <li>cr + * <li>eol * <li>tab * <li>eof * </ul> @@ -82,42 +85,94 @@ * <p> * When this task executes, it will scan the srcdir based on the include * and exclude properties. + * <p> + * This version generalises the handling of EOL characters, and allows + * for CR-only line endings (which I suspect is the standard on Macs.) + * Tab handling has also been generalised to accommodate any tabwidth + * from 2 to 80, inclusive. Importantly, it will leave untouched any + * literal TAB characters embedded within string or character constants. * <p> - * <em>Warning:</em> do not run on binary or carefully formatted files. - * this may sound obvious, but if you don't specify asis, presume that - * your files are going to be modified. If you want tabs to be fixed, + * <em>Warning:</em> do not run on binary files. + * <em>Caution:</em> run with care on carefully formatted files. + * This may sound obvious, but if you don't specify asis, presume that + * your files are going to be modified. If "tabs" is "add" or "remove", * whitespace characters may be added or removed as necessary. Similarly, - * for CR's - in fact cr="add" can result in cr characters being removed. - * (to handle cases where other programs have converted CRLF into CRCRLF). + * for CR's - in fact "eol"="crlf" can result in cr characters being removed + * in one special case accommodated from the comments in the original version + * of this program (FixCRLF), i.e., CRCRLF is regarded as a single EOL + * to handle cases where other programs have converted CRLF into CRCRLF. + * + * Created: Wed Jun 6 23:38:54 2001 * * @author Sam Ruby <a href="mailto:[EMAIL PROTECTED]">[EMAIL PROTECTED]</a> + * @author <a href="mailto:[EMAIL PROTECTED]">Peter B. West</a> + * @version $Revision: 1.4 $ $Name: $ */ -public class FixCRLF extends MatchingTask { +public class FixCRLF extends MatchingTask +{ + private static final int UNDEF = -1; + private static final int NOTJAVA = 0; + private static final int LOOKING = 1; + private static final int IN_CHAR_CONST = 2; + private static final int IN_STR_CONST = 3; + private static final int IN_SINGLE_COMMENT = 4; + private static final int IN_MULTI_COMMENT = 5; + + private static final int ASIS = 0; + private static final int CR = 1; + private static final int LF = 2; + private static final int CRLF = 3; + private static final int ADD = 1; + private static final int REMOVE = -1; + private static final int SPACES = -1; + private static final int TABS = 1; + + private static final int INBUFLEN = 8192; + private static final int LINEBUFLEN = 200; + + private int tablength = 8; + private StringBuffer spaces = new StringBuffer(8); + private StringBuffer linebuf = new StringBuffer(1024); + private StringBuffer linebuf2 = new StringBuffer(1024); + private int eol = UNDEF; + private int addcr = UNDEF; + private String eolstr; + private int ctrlz; + private String ctrlzstr; + private int tabs; + private boolean javafiles = false; + private int defaultState = NOTJAVA; - private int addcr; // cr: -1 => remove, 0 => asis, +1 => add - private int addtab; // tab: -1 => remove, 0 => asis, +1 => add - private int ctrlz; // eof: -1 => remove, 0 => asis, +1 => add - private int tablength = 8; // length of tab in spaces - private File srcDir; private File destDir = null; + public FixCRLF () { + spaces.append(" "); + if (System.getProperty("path.separator").equals(":")) { + ctrlz = REMOVE; + ctrlzstr = new String(""); + } + else { + ctrlz = ASIS; + } + } + /** - * Defaults the properties based on the system type. - * <ul><li>Unix: cr="remove" tab="asis" eof="remove" - * <li>DOS: cr="add" tab="asis" eof="asis"</ul> - */ - public FixCRLF() { - if (System.getProperty("path.separator").equals(":")) { - addcr = -1; // remove - ctrlz = -1; // remove - } else { - addcr = +1; // add - ctrlz = 0; // asis + * Enumerated attribute with the values "asis", "add" and "remove". + */ + public static class AddAsisRemove extends EnumeratedAttribute { + public String[] getValues() { + return new String[] {"add", "asis", "remove"}; } } + public static class CrLf extends EnumeratedAttribute { + public String[] getValues() { + return new String[] {"asis", "cr", "lf", "crlf"}; + } + } + /** * Set the source dir to find the source text files. */ @@ -133,8 +188,49 @@ this.destDir = destDir; } + /** + * Fixing Java source files? + */ + public void setJavafiles(boolean javafiles) { + this.javafiles = javafiles; + defaultState = javafiles ? LOOKING : NOTJAVA; + } + + + /** + * Specify how EndOfLine characters are to be handled + * + * @param option valid values: + * <ul> + * <li>asis: leave line endings alone + * <li>cr: convert line endings to CR + * <li>lf: convert line endings to LF + * <li>crlf: convert line endings to CRLF + * </ul> + */ + public void setEol(CrLf attr) { + String option = attr.getValue(); + if (option.equals("asis")) { + eol = ASIS; + } else if (option.equals("cr")) { + eol = CR; + eolstr = new String("\r"); + } else if (option.equals("lf")) { + eol = LF; + eolstr = new String("\n"); + } else { + // Must be "crlf" + eol = CRLF; + eolstr = new String("\r\n"); + } + } + /** - * Specify how carriage return (CR) charaters are to be handled + * Deprecated alternative method of specifying EOL handling. + * This method is for backward compatibility with previous + * versions of FixCRLF.java. + * + * Specify how carriage return (CR) characters are to be handled * * @param option valid values: * <ul> @@ -146,17 +242,18 @@ public void setCr(AddAsisRemove attr) { String option = attr.getValue(); if (option.equals("remove")) { - addcr = -1; + addcr = REMOVE; } else if (option.equals("asis")) { - addcr = 0; + addcr = ASIS; } else { // must be "add" - addcr = +1; + addcr = ADD; } } /** - * Specify how tab charaters are to be handled + * Specify how tab characters outside string constants are to be handled + * Note that this includes tabs within comments * * @param option valid values: * <ul> @@ -168,26 +265,30 @@ public void setTab(AddAsisRemove attr) { String option = attr.getValue(); if (option.equals("remove")) { - addtab = -1; + tabs = SPACES; } else if (option.equals("asis")) { - addtab = 0; + tabs = ASIS; } else { // must be "add" - addtab = +1; + tabs = TABS; } } /** * Specify tab length in characters * - * @param tlength specify the length of tab in spaces, has to be a power of 2 + * @param tlength specify the length of tab in spaces, has to be a + * power of 2 */ public void setTablength(int tlength) throws BuildException { - if (tlength < 2 || (tlength & (tlength-1)) != 0) { - throw new BuildException("tablength must be a positive power of 2", - location); + if (tlength < 2 || tlength >80) { + throw new BuildException("tablength must be between 2 and 80", + location); } tablength = tlength; + for (int i = 0; i < tablength; i++) { + spaces.append(' '); + } } /** @@ -203,21 +304,21 @@ public void setEof(AddAsisRemove attr) { String option = attr.getValue(); if (option.equals("remove")) { - ctrlz = -1; + ctrlz = REMOVE; } else if (option.equals("asis")) { - ctrlz = 0; + ctrlz = ASIS; } else { // must be "add" - ctrlz = +1; + ctrlz = ADD; } } - /** * Executes the task. */ public void execute() throws BuildException { // first off, make sure that we've got a srcdir and destdir - + System.out.println("Entering execute()"); + if (srcDir == null) { throw new BuildException("srcdir attribute must be set!"); } @@ -236,11 +337,63 @@ } } + // Set up the correct EOL values + if (eol == UNDEF) { + if (addcr == UNDEF) { + // Neither eol not addcr has been defined + // go for the system defaults + if (System.getProperty("line.separator").equals("\r")) { + eol = CR; + } else if (System.getProperty("line.separator").equals("\n")) { + eol = LF; + } else { + eol = CRLF; + } + + } // end of if (addcr == UNDEF) + else { + // addcr has been defined - translate to eol values + switch (addcr) { + case ADD: + eol = CRLF; + break; + + case REMOVE: + eol = LF; + break; + + case ASIS: + eol = ASIS; + break; + + } // end of switch (addcr) + + } // end of if (addcr == UNDEF)else + + } // end of if (eol == UNDEF) + + switch (eol) { + // set eolstr value unless ASIS + case CR: + eolstr = new String("\r"); + break; + + case LF: + eolstr = new String("\n"); + break; + + case CRLF: + eolstr = new String("\r\n"); + break; + + } // end of switch (eol) + // log options used log("options:" + - " cr=" + (addcr==1 ? "add" : addcr==0 ? "asis" : "remove") + - " tab=" + (addtab==1 ? "add" : addtab==0 ? "asis" : "remove") + - " eof=" + (ctrlz==1 ? "add" : ctrlz==0 ? "asis" : "remove") + + " eol=" + + (eol==ASIS ? "asis" : eol==CR ? "cr" : eol==LF ? "lf" : "crlf") + + " tab=" + (tabs==TABS ? "add" : tabs==ASIS ? "asis" : "remove") + + " eof=" + (ctrlz==ADD ? "add" : ctrlz==ASIS ? "asis" : "remove") + " tablength=" + tablength, Project.MSG_VERBOSE); @@ -248,182 +401,592 @@ String[] files = ds.getIncludedFiles(); for (int i = 0; i < files.length; i++) { - File srcFile = new File(srcDir, files[i]); + processFile(files[i]); + } + + } + + private void processFile(String file) + throws BuildException + { + File srcFile = new File(srcDir, file); + File destFile = srcFile; + BufferedWriter outWriter; + OneLiner.BufferLine line; + + // read the contents of the file + OneLiner lines = new OneLiner(srcFile); + + // Set up the output Writer + try { + if (destDir != null) destFile = new File(destDir, file); + FileWriter writer = new FileWriter(destFile); + outWriter = new BufferedWriter(writer); + } catch (IOException e) { + throw new BuildException(e); + } + + while (lines.hasNext()) { + // In-line states + int endComment; - // read the contents of the file - int count = (int)srcFile.length(); - byte indata[] = new byte[count]; try { - FileInputStream inStream = new FileInputStream(srcFile); - inStream.read(indata); - inStream.close(); - } catch (IOException e) { + line = (OneLiner.BufferLine)lines.next(); + } catch (NoSuchElementException e) { throw new BuildException(e); } - - // count the number of cr, lf, and tab characters - int cr = 0; - int lf = 0; - int tab = 0; - - for (int k=0; k<count; k++) { - byte c = indata[k]; - if (c == '\r') cr++; - if (c == '\n') lf++; - if (c == '\t') tab++; + + String lineString = line.getLineString(); + int linelen = line.length(); + + // Note - all of the following processing NOT done for tabs ASIS + + if (tabs == ASIS) { + // Just copy the body of the line across + try { + outWriter.write(lineString); + } catch (IOException e) { + throw new BuildException(e); + } // end of try-catch + } + else { // (tabs != ASIS) + int ptr; + + while ((ptr = line.getNext()) < linelen) { + + switch (lines.getState()) { + + case NOTJAVA: + notInConstant(line, line.length(), outWriter); + break; + + case IN_MULTI_COMMENT: + if ((endComment = + lineString.indexOf("*/", line.getNext()) + ) >= 0) + { + // End of multiLineComment on this line + endComment += 2; // Include the end token + lines.setState(LOOKING); + } + else { + endComment = linelen; + } + + notInConstant(line, endComment, outWriter); + break; + + case IN_SINGLE_COMMENT: + notInConstant(line, line.length(), outWriter); + lines.setState(LOOKING); + break; + + case IN_CHAR_CONST: + case IN_STR_CONST: + // Got here from LOOKING by finding an opening "\'" + // next points to that quote character. + // Find the end of the constant. Watch out for + // backslashes. Literal tabs are left unchanged, and + // the column is adjusted accordingly. + + int begin = line.getNext(); + char terminator = (lines.getState() == IN_STR_CONST + ? '\"' + : '\''); + endOfCharConst(line, terminator); + while (line.getNext() < line.getLookahead()) { + if (line.getNextCharInc() == '\t') { + line.setColumn( + line.getColumn() + + tablength - + line.getColumn() % tablength); + } + else { + line.incColumn(); + } + } + + // Now output the substring + try { + outWriter.write( + line.substring(begin, line.getNext())); + } catch (IOException e) { + throw new BuildException(e); + } + + lines.setState(LOOKING); + + break; + + + case LOOKING: + nextStateChange(line); + notInConstant(line, line.getLookahead(), outWriter); + break; + + } // end of switch (state) + + } // end of while (line.getNext() < linelen) + + } // end of else (tabs != ASIS) + + // Handle end of line now + if (eol == ASIS) { + eolstr = line.getEol(); } - // check for trailing eof - boolean eof = ((count>0) && (indata[count-1] == 0x1A)); + try { + outWriter.write(eolstr); + } catch (IOException e) { + throw new BuildException(e); + } // end of try-catch + + } // end of while (lines.hasNext()) + // Handle CTRLZ + if (ctrlz == REMOVE) { + ctrlzstr = new String(""); + } else if (ctrlz == ASIS) { + ctrlzstr = lines.getEofStr(); + } else { + // must be "add" + ctrlzstr = new String("\u001A"); + } + try { + outWriter.write(ctrlzstr); + outWriter.close(); + } catch (IOException e) { + throw new BuildException(e); + } // end of try-catch + + } - // log stats (before fixes) - log(srcFile + ": size=" + count + " cr=" + cr + - " lf=" + lf + " tab=" + tab + " eof=" + eof, - Project.MSG_VERBOSE); - - // determine the output buffer size (slightly pessimisticly) - int outsize = count; - if (addcr != 0) outsize-=cr; - if (addcr == +1) outsize+=lf; - if (addtab == -1) outsize+=tab*(tablength-1); - if (ctrlz == +1) outsize+=1; - - // copy the data - byte outdata[] = new byte[outsize]; - int o = 0; // output offset - int line = o; // beginning of line - int col = 0; // desired column - - for (int k=0; k<count; k++) { - switch (indata[k]) { - case (byte)' ': - // advance column - if (addtab == 0) outdata[o++]=(byte)' '; - col++; - break; - case (byte)'\t': - if (addtab == 0) { - // treat like any other character - outdata[o++]=(byte)'\t'; - col++; - } else { - // advance column to next tab stop - col = (col|(tablength-1))+1; - } - break; - case (byte)'\r': - if (addcr == 0) { - // treat like any other character - outdata[o++]=(byte)'\r'; - col++; - } - break; + /** + * Scan a BufferLine for the next state changing token: the beginning + * of a single or multi-line comment, a character or a string constant. + * + * As a side-effect, sets the buffer state to the next state, and sets + * field lookahead to the first character of the state-changing token, or + * to the next eol character. + * + * @param BufferLine bufline BufferLine containing the string + * to be processed + * @exception org.apache.tools.ant.BuildException + * Thrown when end of line is reached + * before the terminator is found. + */ + private void nextStateChange(OneLiner.BufferLine bufline) + throws BuildException + { + int eol = bufline.length(); + int ptr = bufline.getNext(); + + + // Look for next single or double quote, double slash or slash star + while (ptr < eol) { + switch (bufline.getChar(ptr++)) { + case '\'': + bufline.setState(IN_CHAR_CONST); + bufline.setLookahead(--ptr); + return; + case '\"': + bufline.setState(IN_STR_CONST); + bufline.setLookahead(--ptr); + return; + case '/': + if (ptr < eol) { + if (bufline.getChar(ptr) == '*') { + bufline.setState(IN_MULTI_COMMENT); + bufline.setLookahead(--ptr); + return; + } + else if (bufline.getChar(ptr) == '/') { + bufline.setState(IN_SINGLE_COMMENT); + bufline.setLookahead(--ptr); + return; + } + } + break; + } // end of switch (bufline.getChar(ptr++)) + + } // end of while (ptr < eol) + // Eol is the next token + bufline.setLookahead(ptr); + } - case (byte)'\n': - // start a new line (optional CR followed by LF) - if (addcr == +1) outdata[o++]=(byte)'\r'; - outdata[o++]=(byte)'\n'; - line=o; - col=0; - break; - default: - // add tabs if two or more spaces are required - if (addtab>0 && o+1<line+col) { - // determine logical column - int diff=o-line; - - // add tabs until this column would be passed - // note: the start of line is adjusted to match - while ((diff|(tablength-1))<col) { - outdata[o++]=(byte)'\t'; - line-=(tablength-1)-(diff&(tablength-1)); - diff=o-line; - }; - }; - - // space out to desired column - while (o<line+col) outdata[o++]=(byte)' '; - - // append desired character - outdata[o++]=indata[k]; - col++; + /** + * Scan a BufferLine forward from the 'next' pointer + * for the end of a character constant. Set 'lookahead' pointer to the + * character following the terminating quote. + * + * @param BufferLine bufline BufferLine containing the string + * to be processed + * @param char terminator The constant terminator + * + * @exception org.apache.tools.ant.BuildException + * Thrown when end of line is reached + * before the terminator is found. + */ + private void endOfCharConst(OneLiner.BufferLine bufline, char terminator) + throws BuildException + { + int ptr = bufline.getNext(); + int eol = bufline.length(); + char c; + ptr++; // skip past initial quote + while (ptr < eol) { + if ((c = bufline.getChar(ptr++)) == '\\') { + ptr++; + } + else { + if (c == terminator) { + bufline.setLookahead(ptr); + return; } } + } // end of while (ptr < eol) + // Must have fallen through to the end of the line + throw new BuildException("endOfCharConst: unterminated char constant"); + } - // add or remove an eof character as required - if (ctrlz == +1) { - if (outdata[o-1]!=0x1A) outdata[o++]=0x1A; - } else if (ctrlz == -1) { - if (o>2 && outdata[o-1]==0x0A && outdata[o-2]==0x1A) o--; - if (o>1 && outdata[o-1]==0x1A) o--; - } - // output the data + /** + * Process a BufferLine string which is not part of of a string constant. + * The start position of the string is given by the 'next' field. + * Sets the 'next' and 'column' fields in the BufferLine. + * + * @param BufferLine bufline BufferLine containing the string + * to be processed + * @param int end Index just past the end of the + * string + * @param BufferedWriter outWriter Sink for the processed string + */ + private void notInConstant(OneLiner.BufferLine bufline, int end, + BufferedWriter outWriter) + { + // N.B. both column and string index are zero-based + // Process a string not part of a constant; + // i.e. convert tabs<->spaces as required + // This is NOT called for ASIS tab handling + int nextTab; + int nextStop; + int tabspaces; + String line = bufline.substring(bufline.getNext(), end); + int place = 0; // Zero-based + int col = bufline.getColumn(); // Zero-based + + // process sequences of white space + // first convert all tabs to spaces + linebuf.setLength(0); + while ((nextTab = line.indexOf((int) '\t', place)) >= 0) { + linebuf.append(line.substring(place, nextTab)); // copy to the TAB + col += nextTab - place; + tabspaces = tablength - (col % tablength); + linebuf.append(spaces.substring(0, tabspaces)); + col += tabspaces; + place = nextTab + 1; + } // end of while + linebuf.append(line.substring(place, line.length())); + // if converting to spaces, all finished + String linestring = new String(linebuf.toString()); + if (tabs == REMOVE) { try { - // Determine whether it should be written, - // that is if it is different than the potentially already existing file - boolean write = false; - byte[] existingdata = indata; - File destFile = srcFile; - if (destDir != null) { - destFile = new File(destDir, files[i]); - if(destFile.isFile()) { - int len = (int)destFile.length(); - if(len != o) { - write = true; - } else { - existingdata = new byte[len]; - try { - FileInputStream in = new FileInputStream(destFile); - in.read(existingdata); - in.close(); - } catch (IOException e) { - throw new BuildException(e); - } - } - } else { - write = true; - } + outWriter.write(linestring); + } catch (IOException e) { + throw new BuildException(e); + } // end of try-catch + } + else { // tabs == ADD + int tabCol; + linebuf2.setLength(0); + place = 0; + col = bufline.getColumn(); + int placediff = col - 0; + // for the length of the string, cycle through the tab stop + // positions, checking for a space preceded by at least one + // other space at the tab stop. if so replace the longest possible + // preceding sequence of spaces with a tab. + nextStop = col + (tablength - col % tablength); + if (nextStop - col < 2) { + linebuf2.append(linestring.substring( + place, nextStop - placediff)); + place = nextStop - placediff; + nextStop += tablength; + } + + for ( ; nextStop - placediff <= linestring.length() + ; nextStop += tablength) + { + for (tabCol = nextStop; + --tabCol - placediff >= place + && linestring.charAt(tabCol - placediff) == ' ' + ;) + { + ; // Loop for the side-effects } - - if(!write) { - if(existingdata.length != o) { - write = true; - } else { - for(int j = 0; j < o; ++j) { - if(existingdata[j] != outdata[j]) { - write = true; - break; - } - } - } + // tabCol is column index of the last non-space character + // before the next tab stop + if (nextStop - tabCol > 2) { + linebuf2.append(linestring.substring( + place, ++tabCol - placediff)); + linebuf2.append('\t'); } + else { + linebuf2.append(linestring.substring( + place, nextStop - placediff)); + } // end of else - if(write) { - log(destFile + " is being written", Project.MSG_VERBOSE); - FileOutputStream outStream = new FileOutputStream(destFile); - outStream.write(outdata,0,o); - outStream.close(); - } else { - log(destFile + " is not written, as the contents are identical", - Project.MSG_VERBOSE); - } + place = nextStop - placediff; + } // end of for (nextStop ... ) + + // pick up that last bit, if any + linebuf2.append(linestring.substring(place, linestring.length())); + + try { + outWriter.write(linebuf2.toString()); } catch (IOException e) { throw new BuildException(e); - } + } // end of try-catch + + } // end of else tabs == ADD + + // Set column position as modified by this method + bufline.setColumn(bufline.getColumn() + linestring.length()); + bufline.setNext(end); - } /* end for */ } - /** - * Enumerated attribute with the values "asis", "add" and "remove". - */ - public static class AddAsisRemove extends EnumeratedAttribute { - public String[] getValues() { - return new String[] {"add", "asis", "remove"}; + + class OneLiner implements Iterator + { + private static final char CTRLZ = '\u001A'; + + private int state = defaultState; + + private StringBuffer eolStr = new StringBuffer(LINEBUFLEN); + private StringBuffer eofStr = new StringBuffer(); + + private BufferedReader reader; + private String line; + + public OneLiner(File srcFile) + throws BuildException + { + try { + reader = new BufferedReader + (new FileReader(srcFile), INBUFLEN); + nextLine(); + } catch (IOException e) { + throw new BuildException(e); + } + } + + protected void nextLine() + throws BuildException { + int ch; + int eolcount = 0; + + eolStr.setLength(0); + + try { + int linelen; + + reader.mark(INBUFLEN); + line = reader.readLine(); + if (line == null) { + // Eof has been reached + linelen = 0; + } + else { + linelen = line.length(); + } + + + // Find the EOL character(s) + + reader.reset(); + + // an IOException will be thrown + reader.skip((long)linelen); + reader.mark(INBUFLEN); + ch = reader.read(); + switch ((char) ch) { + case '\r': + // Check for \r, \r\n and \r\r\n + // Regard \r\r not followed by \n as two lines + ++eolcount; + eolStr.append('\r'); + switch ((char)(ch = reader.read())) { + case '\r': + if ((char)(ch = reader.read()) == '\n') { + eolcount += 2; + eolStr.append("\r\n"); + } + break; + case '\n': + ++eolcount; + eolStr.append('\n'); + break; + } // end of switch ((char)(ch = reader.read())) + break; + + case '\n': + ++eolcount; + eolStr.append('\n'); + break; + + } // end of switch ((char) ch) + + // Reset the position of the file reader + reader.reset(); + reader.skip((long)eolcount); + + // if at eolcount == 0 and trailing characters of string + // are CTRL-Zs, set eofStr + if (line != null && eolcount == 0) { + int i = linelen; + while (--i >= 0 && line.charAt(i) == '\u001A') {} + if (i < linelen - 1) { + // Trailing characters are ^Zs + // Construct new line and eofStr + eofStr.append(line.substring(i + 1)); + line = i < 0 ? null : line.substring(0, i + 1); + } + + } // end of if (eolcount == 0) + + } catch (IOException e) { + throw new BuildException(e); + } + } + + public String getEofStr() { + return eofStr.toString(); } + + public int getState() { + return state; + } + + public void setState(int state) { + this.state = state; + } + + public boolean hasNext() + { + return line != null; + } + + public Object next() + throws NoSuchElementException + { + if (! hasNext()) { + throw new NoSuchElementException("OneLiner"); + } + BufferLine tmpLine = + new BufferLine(line, eolStr.toString()); + nextLine(); + return tmpLine; + } + + public void remove() + throws UnsupportedOperationException + { + throw new UnsupportedOperationException("OneLiner"); + } + + + class BufferLine { + private int next = 0; + private int column = 0; + private int lookahead = UNDEF; + private String line; + private String eolStr; + + + public BufferLine(String line, String eolStr) + throws BuildException + { + next = 0; + column = 0; + this.line = line; + this.eolStr = eolStr; + } + + public int getNext() { + return next; + } + + public void setNext(int next) { + this.next = next; + } + + public int getLookahead() { + return lookahead; + } + + public void setLookahead(int lookahead) { + this.lookahead = lookahead; + } + + public char getChar(int i) { + return line.charAt(i); + } + + public char getNextChar() { + return getChar(next); + } + + public char getNextCharInc() { + return getChar(next++); + } + + public int getColumn() { + return column; + } + + public void setColumn(int col) { + column = col; + } + + public int incColumn() { + return column++; + } + + public int length() { + return line.length(); + } + + public int getEolLength() { + return eolStr.length(); + } + + public String getLineString() { + return line; + } + + public String getEol() { + return eolStr; + } + + public String substring(int begin) { + return line.substring(begin); + } + + public String substring(int begin, int end) { + return line.substring(begin, end); + } + + public void setState(int state) { + OneLiner.this.setState(state); + } + + public int getState() { + return OneLiner.this.getState(); + } + } } + }
Index: fixcrlf.html
===================================================================
RCS file: /home/cvspublic/jakarta-ant/docs/manual/CoreTasks/fixcrlf.html,v
retrieving revision 1.2
diff -u -u -r1.2 fixcrlf.html
--- fixcrlf.html 2001/02/13 12:31:51 1.2
+++ fixcrlf.html 2001/07/12 14:17:25
@@ -1,168 +1,309 @@
-<html>
+ <html>
+
+ <head>
+ <meta http-equiv="Content-Language" content="en-us">
+ <title>Ant User Manual</title>
+ </head>
-<head>
-<meta http-equiv="Content-Language" content="en-us">
-<title>Ant User Manual</title>
-</head>
-
-<body>
-
-<h2><a name="fixcrlf">FixCRLF</a></h2>
-<h3>Description</h3>
-<p>Adjusts a text file to local.</p>
-<p>It is possible to refine the set of files that are being adjusted. This can
be
-done with the <i>includes</i>, <i>includesfile</i>, <i>excludes</i>,
<i>excludesfile</i> and <i>defaultexcludes</i>
-attributes. With the <i>includes</i> or <i>includesfile</i> attribute you
specify the files you want to
-have included by using patterns. The <i>exclude</i> or <i>excludesfile</i>
attribute is used to specify
-the files you want to have excluded. This is also done with patterns. And
-finally with the <i>defaultexcludes</i> attribute, you can specify whether you
-want to use default exclusions or not. See the section on <a
-href="../dirtasks.html#directorybasedtasks">directory based tasks</a>, on how
the
-inclusion/exclusion of files works, and how to write patterns.</p>
-<p>This task forms an implicit <a href="../CoreTypes/fileset.html">FileSet</a>
and
-supports all attributes of <code><fileset></code>
-(<code>dir</code> becomes <code>srcdir</code>) as well as the nested
-<code><include></code>, <code><exclude></code> and
-<code><patternset></code> elements.</p>
-<h3>Parameters</h3>
-<table border="1" cellpadding="2" cellspacing="0">
- <tr>
- <td valign="top"><b>Attribute</b></td>
- <td valign="top"><b>Description</b></td>
- <td align="center" valign="top"><b>Required</b></td>
- </tr>
- <tr>
- <td valign="top">srcDir</td>
- <td valign="top">Where to find the files to be fixed up.</td>
- <td valign="top" align="center">Yes</td>
- </tr>
- <tr>
- <td valign="top">destDir</td>
- <td valign="top">Where to place the corrected files. Defaults to
- srcDir (replacing the original file)</td>
- <td valign="top" align="center">No</td>
- </tr>
- <tr>
- <td valign="top">includes</td>
- <td valign="top">comma separated list of patterns of files that must be
- included. All files are included when omitted.</td>
- <td valign="top" align="center">No</td>
- </tr>
- <tr>
- <td valign="top">includesfile</td>
- <td valign="top">the name of a file. Each line of this file is
- taken to be an include pattern</td>
- <td valign="top" align="center">No</td>
- </tr>
- <tr>
- <td valign="top">excludes</td>
- <td valign="top">comma separated list of patterns of files that must be
- excluded. No files (except default excludes) are excluded when
omitted.</td>
- <td valign="top" align="center">No</td>
- </tr>
- <tr>
- <td valign="top">excludesfile</td>
- <td valign="top">the name of a file. Each line of this file is
- taken to be an exclude pattern</td>
- <td valign="top" align="center">No</td>
- </tr>
- <tr>
- <td valign="top">defaultexcludes</td>
- <td valign="top">indicates whether default excludes should be used or not
- ("yes"/"no"). Default excludes are used when
omitted.</td>
- <td valign="top" align="center">No</td>
- </tr>
- <tr>
- <td valign="top">cr</td>
- <td valign="top">Specifies how carriage return (CR) characters are to
- be handled. Valid values for this property are:
- <ul>
- <li>add: ensure that there is a CR before every LF</li>
- <li>asis: leave CR characters alone</li>
- <li>remove: remove all CR characters</li>
- </ul>
- Default is based on the platform on which you are running this task.
- For Unix platforms, the default is remove. For DOS based systems
- (including Windows), the default is add.
- <p>
- Note: Unless this property is specified as "asis", extra CR
characters
- which do not precede a LF will be removed.</p>
- </td>
- <td valign="top" align="center">No</td>
- </tr>
- <tr>
- <td valign="top">tab</td>
- <td valign="top">Specifies how tab characters are to be handled. Valid
- values for this property are:
- <ul>
- <li>add: convert sequences of spaces which span a tab stop to tabs</li>
- <li>asis: leave tab and space characters alone</li>
- <li>remove: convert tabs to spaces</li>
- </ul>
- Default for this parameter is "asis".
- <p>
- Note: Unless this property is specified as "asis", extra
spaces and
- tabs after the last non-whitespace character on the line will be
removed.</p>
- </td>
- <td valign="top" align="center">No</td>
- </tr>
- <tr>
- <td valign="top">tablength</td>
- <td valign="top">The number of characters a TAB stop corresponds to.
- Must be a positive power of 2, default for this parameter is 8.</td>
- <td valign="top" align="center">No</td>
- </tr>
- <tr>
- <td valign="top">eof</td>
- <td valign="top">Specifies how DOS end of file (control-Z) characters are
- to be handled. Valid values for this property are:
- <ul>
- <li>add: ensure that there is an EOF character at the end of the
file</li>
- <li>asis: leave EOF characters alone</li>
- <li>remove: remove any EOF character found at the end</li>
- </ul>
- Default is based on the platform on which you are running this task.
- For Unix platforms, the default is remove. For DOS based systems
- (including Windows), the default is asis.
- </td>
- <td valign="top" align="center">No</td>
- </tr>
-</table>
-<h3>Examples</h3>
-<pre> <fixcrlf srcdir="${src}"
- cr="remove" eof="remove"
+ <body>
+
+ <h2><a name="fixcrlf">FixCRLF</a></h2>
+
+ <h3>Description</h3>
+
+ <p>
+ Adjusts a text file to local conventions.
+ </p>
+
+ <p>
+ The set of files to be adjusted can be refined with the
+ <i>includes</i>, <i>includesfile</i>, <i>excludes</i>,
+ <i>excludesfile</i> and <i>defaultexcludes</i>
+ attributes. Patterns provided through the <i>includes</i> or
+ <i>includesfile</i> attributes specify files to be
+ included. Patterns provided throught the <i>exclude</i> or
+ <i>excludesfile</i> attribute specify files to be
+ excluded. Additionally, default exclusions can be specified with
+ the <i>defaultexcludes</i> attribute. See the section on <a
+ href="../dirtasks.html#directorybasedtasks">directory based
+ tasks</a>, for details of file inclusion/exclusion patterns
+ and their usage.
+ </p>
+
+ <p>
+ This task forms an implicit <a href =
+ "../CoreTypes/fileset.html" >FileSet</a> and supports all
+ attributes of <code><fileset></code> (<code>dir</code>
+ becomes <code>srcdir</code>) as well as the nested
+ <code><include></code>, <code><exclude></code> and
+ <code><patternset></code> elements.
+ </p>
+
+ <p>
+ The output file is only written if it is a new file, or if it
+ differs from the existing file. This prevents spurious
+ rebuilds based on unchanged files which have been regenerated
+ by this task.
+ </p>
+
+ <h3>Parameters</h3>
+
+ <table border="1" cellpadding="2" cellspacing="0">
+ <tr>
+ <td valign="top"><b>Attribute</b></td>
+ <td valign="top"><b>Description</b></td>
+ <td align="center" valign="top"><b>Required</b></td>
+ </tr>
+ <tr>
+ <td valign="top">srcDir</td>
+ <td valign="top">Where to find the files to be fixed up.</td>
+ <td valign="top" align="center">Yes</td>
+ </tr>
+ <tr>
+ <td valign="top">destDir</td>
+ <td valign="top">Where to place the modified files. Defaults to
+ srcDir (replacing the original file)</td>
+ <td valign="top" align="center">No</td>
+ </tr>
+ <tr>
+ <td valign="top">includes</td>
+ <td valign="top">
+ Comma separated list of patterns of files that must be
+ included. All files are included when omitted.
+ </td>
+ <td valign="top" align="center">No</td>
+ </tr>
+ <tr>
+ <td valign="top">includesfile</td>
+ <td valign="top">
+ The name of a file. Each line of this file is taken to be
+ an include pattern
+ </td>
+ <td valign="top" align="center">No</td>
+ </tr>
+ <tr>
+ <td valign="top">excludes</td>
+ <td valign="top">
+ Comma separated list of patterns of files that must be
+ excluded. No files (except default excludes) are excluded
+ when omitted.
+ </td>
+ <td valign="top" align="center">No</td>
+ </tr>
+ <tr>
+ <td valign="top">excludesfile</td>
+ <td valign="top">
+ The name of a file. Each line of this file is taken to be
+ an exclude pattern
+ </td>
+ <td valign="top" align="center">No</td>
+ </tr>
+ <tr>
+ <td valign="top">defaultexcludes</td>
+ <td valign="top">
+ Indicates whether default excludes should be used or not
+ ("yes"/"no"). Default excludes are
+ used when omitted.
+ </td>
+ <td valign="top" align="center">No</td>
+ </tr>
+ <tr>
+ <td valign="top">eol</td>
+ <td valign="top">
+ Specifies how end-of-line (EOL) characters are to be
+ handled. The EOL characters are CR, LF and the pair CRLF.
+ Valid values for this property are:
+ <ul>
+ <li>asis: leave EOL characters alone</li>
+ <li>cr: convert all EOLs to a single CR</li>
+ <li>lf: convert all EOLs to a single LF</li>
+ <li>crlf: convert all EOLs to the pair CRLF</li>
+ </ul>
+ Default is based on the platform on which you are running
+ this task. For Unix platforms, the default is "lf".
+ For DOS based systems (including Windows), the default is
+ "crlf". Defaults may be added later for other systems,
+ e.g. Mac OS.
+ <p>
+ This is the preferred method for specifying EOL. The
+ "<i><b>addcr</b></i>" attribute (see below) is
+ now deprecated. If both are specified, "eol"
+ takes precedence.
+ </p>
+ <p>
+ <i>N.B.</i>: One special case is recognized. The three
+ characters CR-CR-LF are regarded as a single EOL.
+ Unless this property is specified as "asis",
+ this sequence will be converted into the specified EOL
+ type.
+ </p>
+ </td>
+ <td valign="top" align="center">No</td>
+ </tr>
+ <tr>
+ <td valign="top">addcr</td>
+ <td valign="top">
+ <i><b>Deprecated.</b></i> Specifies how CR characters are
+ to be handled at end-of-line (EOL). Valid values for this
+ property are:
+ <ul>
+ <li>asis: leave EOL characters alone.</li>
+ <li>
+ add: add a CR before any single LF characters. The
+ intent is to convert all EOLs to the pair CRLF.
+ </li>
+ <li>
+ remove: remove all CRs from the file. The intent is
+ to convert all EOLs to a single LF.
+ </li>
+ </ul>
+ Default is based on the platform on which you are running
+ this task. For Unix platforms, the default is "remove".
+ For DOS based systems (including Windows), the default is
+ "add".
+ <p>
+ <i>N.B.</i>: One special case is recognized. The three
+ characters CR-CR-LF are regarded as a single EOL.
+ Unless this property is specified as "asis",
+ this sequence will be converted into the specified EOL
+ type.
+ </p>
+ </td>
+ <td valign="top" align="center">No</td>
+ </tr>
+ <tr>
+ <td valign="top">javafiles</td>
+ <td valign="top">
+ Used only in association with the
+ "<i><b>tab</b></i>" attribute (see below), this
+ boolean attribute indicates whether the fileset is a set
+ of java source files
+ ("yes"/"no"). Defaults to
+ "no". See notes in section on "tab".
+ </td>
+ <td valign="top" align="center">No</td>
+ </tr>
+ <tr>
+ <td valign="top">tab</td>
+ <td valign="top">
+ Specifies how tab characters are to be handled. Valid
+ values for this property are:
+ <ul>
+ <li>
+ add: convert sequences of spaces which span a tab stop
+ to tabs
+ </li>
+ <li>asis: leave tab and space characters alone</li>
+ <li>remove: convert tabs to spaces</li>
+ </ul>
+ Default for this parameter is "asis".
+ <p>
+ <i>N.B.</i>: When the attribute
+ "<i><b>javafiles</b></i>" (see above) is
+ "true", literal TAB characters occurring
+ within Java string or character constants are never
+ modified. This functionality also requires the
+ recognition of Java-style comments.
+ </p>
+ </td>
+ <td valign="top" align="center">No</td>
+ </tr>
+ <tr>
+ <td valign="top">tablength</td>
+ <td valign="top">
+ TAB character interval. Valid values are between 2
+ and 80 inclusive. The default for this parameter is 8.
+ </td>
+ <td valign="top" align="center">No</td>
+ </tr>
+ <tr>
+ <td valign="top">eof</td>
+ <td valign="top">
+ Specifies how DOS end of file (control-Z) characters are
+ to be handled. Valid values for this property are:
+ <ul>
+ <li>
+ add: ensure that there is an EOF character at the end
+ of the file
+ </li>
+ <li>asis: leave EOF characters alone</li>
+ <li>
+ remove: remove any EOF character(s) found at the end
+ of the file
+ </li>
+ </ul>
+ Default is based on the platform on which you are running
+ this task. For Unix platforms, the default is remove.
+ For DOS based systems (including Windows), the default is
+ asis.
+ </td>
+ <td valign="top" align="center">No</td>
+ </tr>
+ </table>
+
+ <h3>Examples</h3>
+
+ <pre> <fixcrlf srcdir="${src}"
+ eol="lf" eof="remove"
includes="**/*.sh"
/></pre>
-<p>Removes carriage return and eof characters from the shell scripts. Tabs and
-spaces are left as is.</p>
-<pre> <fixcrlf srcdir="${src}"
- cr="add"
+ <p>
+ Replaces EOLs with LF characters and removes eof characters
+ from the shell scripts. Tabs and spaces are left as is.
+ </p>
+ <pre> <fixcrlf srcdir="${src}"
+ eol="crlf"
includes="**/*.bat"
/></pre>
-<p>Ensures that there are carriage return characters prior to evey line feed.
-Tabs and spaces are left as is.
-EOF characters are left alone if run on
-DOS systems, and are removed if run on Unix systems.</p>
-<pre> <fixcrlf srcdir="${src}"
+ <p>
+ Replaces all EOLs with cr-lf pairs in the batch files. Tabs
+ and spaces are left as is. EOF characters are left alone if
+ run on DOS systems, and are removed if run on Unix systems.
+ </p>
+ <pre> <fixcrlf srcdir="${src}"
tabs="add"
includes="**/Makefile"
/></pre>
-<p>Adds or removes CR characters to match local OS conventions, and
-converts spaces to tabs when appropriate. EOF characters are left alone if
-run on DOS systems, and are removed if run on Unix systems.
-Many versions of make require tabs prior to commands.</p>
-<pre> <fixcrlf srcdir="${src}"
+ <p>
+ Sets EOLs according to local OS conventions, and converts
+ sequences of spaces and tabs to the minimal set of spaces and
+ tabs which will maintain spacing within the line. Tabs are
+ set at 8 character intervals. EOF characters are left alone
+ if run on DOS systems, and are removed if run on Unix systems.
+ Many versions of make require tabs prior to commands.
+ </p>
+ <pre> <fixcrlf srcdir="${src}"
tabs="remove"
+ tablength="3"
+ eol="lf"
+ javafiles="yes"
+ includes="**/*.java"
+ /></pre>
+ <p>
+ Converts all EOLs in the included java source files to a
+ single LF. Replace all TAB characters except those in string
+ or character constants with spaces, assuming a tab width of 3.
+ If run on a unix system, any CTRL-Z EOF characters at the end
+ of the file are removed. On DOS/Windows, any such EOF
+ characters will be left untouched.
+ </p>
+ <pre> <fixcrlf srcdir="${src}"
+ tabs="remove"
includes="**/README*"
/></pre>
-<p>Adds or removes CR characters to match local OS conventions, and
-converts all tabs to spaces. EOF characters are left alone if run on
-DOS systems, and are removed if run on Unix systems.
-You never know what editor a user will use to browse README's.</p>
-<hr>
-<p align="center">Copyright © 2000,2001 Apache Software Foundation. All
rights
-Reserved.</p>
+ <p>
+ Sets EOLs according to local OS conventions, and converts all
+ tabs to spaces, assuming a tab width of 8. EOF characters are
+ left alone if run on DOS systems, and are removed if run on
+ Unix systems. You never know what editor a user will use to
+ browse README's.
+ </p>
+ <hr>
+ <p align="center">
+ Copyright © 2000,2001 Apache Software Foundation. All rights
+ Reserved.
+ </p>
-</body>
-</html>
+ </body>
+ </html>
