> I don't in general disagree with this sort of optimization, but I think
> a good fix is a bit more complicated than what you posted.
Hi Doug, I had some time to redo my fixes...
> This forces a flush() each time a byte array of any size is written.
> That could be much slower when lots of small byte arrays are written,
> since flush() invokes a system call. What would be best is, if there is
Yes, I made a better version in an exact way you have described above.
The patch replaces writeBytes method with the whole new implementation:
public void writeBytes(byte[] b, int length) throws IOException {
int bytesLeft = BUFFER_SIZE - bufferPosition;
// is there enough space in the buffer?
if (bytesLeft >= length) {
// we add the data to the end of the buffer
System.arraycopy(b, 0, buffer, bufferPosition, length);
bufferPosition += length;
// if the buffer is full, flush it
if (BUFFER_SIZE - bufferPosition == 0)
flush();
} else {
// we fill/flush the buffer (until the input is written)
int pos = 0; // position in the input data
int pieceLength;
while (pos < length) {
pieceLength = (length - pos < bytesLeft) ?
length - pos : bytesLeft;
System.arraycopy(b, pos, buffer,
bufferPosition, pieceLength);
pos += pieceLength;
bufferPosition += pieceLength;
// if the buffer is full, flush it
bytesLeft = BUFFER_SIZE - bufferPosition;
if (bytesLeft == 0) {
flush();
bytesLeft = BUFFER_SIZE;
}
}
}
}
This is much smarter and it is not slow for lots of one-byte writes. I
agree with the assession of my "fix" of readBytes:
> Again, this could be much slower when lots of small arrays are written,
> since each call forces seek and read system calls. However this could
> be optimized for the case where the desired data resides entirely in the
> current buffer to use System.arraycopy.
So I tried to optimize the reads from buffer, but the method was so
complicated I dropped this thought. Readbytes is fast enough.
I have modified the StoreTest, now you can test the *byte or *bytes
methods (3rd parameter). Because of creating and testing the buffers,
there is not speed improvement here but when you write from some
prepared buffer (e.g. from a file), the speed increases.
--
S pozdravem / Best regards
Lukas Zapletal
sefredaktor / editor-in-chief
LinuxEXPRES - opravdovy linuxovy magazin
www.LinuxEXPRES.cz
tel.:+420 777 003 843
Index: src/test/org/apache/lucene/StoreTest.java
===================================================================
--- src/test/org/apache/lucene/StoreTest.java (revision 290385)
+++ src/test/org/apache/lucene/StoreTest.java (working copy)
@@ -28,14 +28,13 @@
class StoreTest {
public static void main(String[] args) {
try {
- test(1000, true);
+ test(1000, true, true);
} catch (Exception e) {
- System.out.println(" caught a " + e.getClass() +
- "\n with message: " + e.getMessage());
+ e.printStackTrace();
}
}
- public static void test(int count, boolean ram)
+ public static void test(int count, boolean ram, boolean buffered)
throws Exception {
Random gen = new Random(1251971);
int i;
@@ -51,6 +50,8 @@
final int LENGTH_MASK = 0xFFF;
+ final byte[] buffer = new byte[LENGTH_MASK];
+
for (i = 0; i < count; i++) {
String name = i + ".dat";
int length = gen.nextInt() & LENGTH_MASK;
@@ -59,8 +60,14 @@
IndexOutput file = store.createOutput(name);
- for (int j = 0; j < length; j++)
- file.writeByte(b);
+ if (buffered) {
+ for (int j = 0; j < length; j++)
+ buffer[j] = b;
+ file.writeBytes(buffer, length);
+ } else {
+ for (int j = 0; j < length; j++)
+ file.writeByte(b);
+ }
file.close();
}
@@ -89,9 +96,18 @@
if (file.length() != length)
throw new Exception("length incorrect");
- for (int j = 0; j < length; j++)
- if (file.readByte() != b)
- throw new Exception("contents incorrect");
+ byte[] content = new byte[length];
+ if (buffered) {
+ file.readBytes(content, 0, length);
+ // check the buffer
+ for (int j = 0; j < length; j++)
+ if (content[j] != b)
+ throw new Exception("contents incorrect");
+ } else {
+ for (int j = 0; j < length; j++)
+ if (file.readByte() != b)
+ throw new Exception("contents incorrect");
+ }
file.close();
}
Index: src/java/org/apache/lucene/store/BufferedIndexOutput.java
===================================================================
--- src/java/org/apache/lucene/store/BufferedIndexOutput.java (revision 290385)
+++ src/java/org/apache/lucene/store/BufferedIndexOutput.java (working copy)
@@ -41,8 +41,32 @@
* @see IndexInput#readBytes(byte[],int,int)
*/
public void writeBytes(byte[] b, int length) throws IOException {
- for (int i = 0; i < length; i++)
- writeByte(b[i]);
+ int bytesLeft = BUFFER_SIZE - bufferPosition;
+ // is there enough space in the buffer?
+ if (bytesLeft >= length) {
+ // we add the data to the end of the buffer
+ System.arraycopy(b, 0, buffer, bufferPosition, length);
+ bufferPosition += length;
+ // if the buffer is full, flush it
+ if (BUFFER_SIZE - bufferPosition == 0)
+ flush();
+ } else {
+ // we fill/flush the buffer (until the input is written)
+ int pos = 0; // position in the input data
+ int pieceLength;
+ while (pos < length) {
+ pieceLength = (length - pos < bytesLeft) ? length - pos : bytesLeft;
+ System.arraycopy(b, pos, buffer, bufferPosition, pieceLength);
+ pos += pieceLength;
+ bufferPosition += pieceLength;
+ // if the buffer is full, flush it
+ bytesLeft = BUFFER_SIZE - bufferPosition;
+ if (bytesLeft == 0) {
+ flush();
+ bytesLeft = BUFFER_SIZE;
+ }
+ }
+ }
}
/** Forces any buffered output to be written. */
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
