Re: BufferedOutputStream
On 07.06.2008 03:13, Andy Stevens wrote: Out of interest, have you raised an enhancement request and/or a patch with the OpenJDK project[1]? It seems to me this increasing buffer behaviour would be useful more generally... Done. Joerg
Re: BufferedOutputStream
Joerg Heinicke wrote: On 02.06.2008 05:56, Sylvain Wallez wrote: Stupid question: why do we need a special BufferedOutputStream? For being able to reset the response buffer for error handling. This is also possible with java.io.BufferedOutputStream, if the buffer size is big enough (current default value is 1MB), but then the buffer byte[] is always that big rather than increasing. That's what's happening right now, if you don't specify -1 as buffer size. -1 means complete buffering which on the other hand might lead to OutOfMemoryError [1]. In addition our BOS counts the bytes so that we can use the value to set the content length header. Joerg [1] https://issues.apache.org/jira/browse/COCOON-2168 Out of interest, have you raised an enhancement request and/or a patch with the OpenJDK project[1]? It seems to me this increasing buffer behaviour would be useful more generally... Andy. [1] http://openjdk.java.net/ -- http://pseudoq.sourceforge.net/ Open source Sudoku solver
Re: BufferedOutputStream
Joerg Heinicke wrote:
Reinhard Pötz reinhard at apache.org writes:
Is my understanding right that the content length header can only be set
as long as you haven't written into the underlying output stream?
Yes, it is. That's why the buffering has to be big enough, i.e. nothing must
have been flushed yet.
What happens when the buffer size is exceeded? Does it mean that the
output is streamed and the content length parameter can't be set?
--
Reinhard Pötz Managing Director, {Indoqa} GmbH
http://www.indoqa.com/en/people/reinhard.poetz/
Member of the Apache Software Foundation
Apache Cocoon Committer, PMC member [EMAIL PROTECTED]
Re: BufferedOutputStream
Joerg Heinicke wrote: On 02.06.2008 05:56, Sylvain Wallez wrote: Only issue I want to solve before the release is the BufferedOutputStream issue. I planned to do it this weekend. Done. Please review the file attached. It's still completely untested. At the moment I need some sleep ;) I will write junit tests for it this week and eventually commit it. Stupid question: why do we need a special BufferedOutputStream? For being able to reset the response buffer for error handling. This is also possible with java.io.BufferedOutputStream, if the buffer size is big enough (current default value is 1MB), but then the buffer byte[] is always that big rather than increasing. That's what's happening right now, if you don't specify -1 as buffer size. -1 means complete buffering which on the other hand might lead to OutOfMemoryError [1]. In addition our BOS counts the bytes so that we can use the value to set the content length header. Got it. I'm using a similar technique with a servlet filter providing the response with a ByteArrayOutputStream that gives the size and has the convenient writeTo(OutputStream) method. Now in my case [1] pages are always small and I don't need to put a high limit on the size of the buffer. Sylvain [1] http://bluxte.net/blog/2008-04/29-54-29.html -- Sylvain Wallez - http://bluxte.net
Re: BufferedOutputStream
Reinhard Pötz reinhard at apache.org writes: Is my understanding right that the content length header can only be set as long as you haven't written into the underlying output stream? Yes, it is. That's why the buffering has to be big enough, i.e. nothing must have been flushed yet. What happens when the buffer size is exceeded? Does it mean that the output is streamed and the content length parameter can't be set? If the buffer size is exceeded the output stream is flushed and eventually written to the response. A header like the content-length can no longer be set. That's why Cocoon's BufferedOutputStream was setup to buffer all and everything by default which lead to COCOON-2168 [1] with Cocoon's default setting. With the change to a buffer of 1 MB rather than endless buffering we switched back to java.io.BufferedOutputStream, but this allocates the memory for the buffer immediately. So even with 10 KB pages you always end up with 1 MB byte[]. I intend to fix this in our BufferedOutputStream with an increasing buffer and the two configuration parameters initial buffer size and flush buffer size. Just to make it clear: We talk about default configuration of Cocoon here. You still can enforce complete buffering by setting flush buffer size to -1 again. But it's up to the user to enforce this and potentially run into OOME. It's also up to the users to find a reasonable flush buffer size as it always used to be when they haven't used the endless buffering. Only thing that is going to change is in case of not endless buffering: The full buffer is not allocated immediately but step by step. Joerg [1] https://issues.apache.org/jira/browse/COCOON-2168
Re: BufferedOutputStream
Joerg Heinicke wrote: On 29.05.2008 00:06, Joerg Heinicke wrote: Only issue I want to solve before the release is the BufferedOutputStream issue. I planned to do it this weekend. Done. Please review the file attached. It's still completely untested. At the moment I need some sleep ;) I will write junit tests for it this week and eventually commit it. Stupid question: why do we need a special BufferedOutputStream? Sylvain -- Sylvain Wallez - http://bluxte.net
Re: BufferedOutputStream
On 02.06.2008 05:56, Sylvain Wallez wrote: Only issue I want to solve before the release is the BufferedOutputStream issue. I planned to do it this weekend. Done. Please review the file attached. It's still completely untested. At the moment I need some sleep ;) I will write junit tests for it this week and eventually commit it. Stupid question: why do we need a special BufferedOutputStream? For being able to reset the response buffer for error handling. This is also possible with java.io.BufferedOutputStream, if the buffer size is big enough (current default value is 1MB), but then the buffer byte[] is always that big rather than increasing. That's what's happening right now, if you don't specify -1 as buffer size. -1 means complete buffering which on the other hand might lead to OutOfMemoryError [1]. In addition our BOS counts the bytes so that we can use the value to set the content length header. Joerg [1] https://issues.apache.org/jira/browse/COCOON-2168
Re: BufferedOutputStream
Joerg Heinicke wrote:
On 02.06.2008 05:56, Sylvain Wallez wrote:
Only issue I want to solve before the release is the
BufferedOutputStream issue. I planned to do it this weekend.
Done. Please review the file attached. It's still completely
untested. At the moment I need some sleep ;) I will write junit tests
for it this week and eventually commit it.
Stupid question: why do we need a special BufferedOutputStream?
For being able to reset the response buffer for error handling. This is
also possible with java.io.BufferedOutputStream, if the buffer size is
big enough (current default value is 1MB), but then the buffer byte[] is
always that big rather than increasing. That's what's happening right
now, if you don't specify -1 as buffer size. -1 means complete buffering
which on the other hand might lead to OutOfMemoryError [1]. In addition
our BOS counts the bytes so that we can use the value to set the content
length header.
Is my understanding right that the content length header can only be set
as long as you haven't written into the underlying output stream?
--
Reinhard Pötz Managing Director, {Indoqa} GmbH
http://www.indoqa.com/en/people/reinhard.poetz/
Member of the Apache Software Foundation
Apache Cocoon Committer, PMC member [EMAIL PROTECTED]
Re: BufferedOutputStream
Reinhard Pötz reinhard at apache.org writes: Is my understanding right that the content length header can only be set as long as you haven't written into the underlying output stream? Yes, it is. That's why the buffering has to be big enough, i.e. nothing must have been flushed yet. Joerg
BufferedOutputStream (was: Releasing 2.1.12 - a new release manager?)
On 29.05.2008 00:06, Joerg Heinicke wrote:
Only issue I want to solve before the release is the
BufferedOutputStream issue. I planned to do it this weekend.
Done. Please review the file attached. It's still completely untested.
At the moment I need some sleep ;) I will write junit tests for it this
week and eventually commit it.
Joerg
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the License); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an AS IS BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cocoon.util;
import java.io.FilterOutputStream;
import java.io.IOException;
import java.io.OutputStream;
/**
* This class is similar to the [EMAIL PROTECTED]
java.io.BufferedOutputStream}. In
* addition it provides an increasing buffer, the possibility to reset the
* buffer and it counts the number of bytes written to the output stream.
*
* @author a href=mailto:[EMAIL PROTECTED]Carsten Ziegeler/a
* @version CVS $Id: BufferedOutputStream.java 433543 2006-08-22 06:22:54Z
crossley $
* @since 2.1
*/
public class BufferedOutputStream extends FilterOutputStream {
private byte buffer[];
private int count;
private int totalCount;
private final int flushBufferSize;
/**
* Creates a new buffered output stream to write data to the
* specified underlying output stream with a default 8192-byte
* buffer size.
*
* @param out the underlying output stream.
*/
public BufferedOutputStream(final OutputStream out) {
this(out, 8192, 32768);
}
/**
* Creates a new buffered output stream to write data to the specified
* underlying output stream with the specified buffer sizes.
*
* @param outthe underlying output stream.
* @param initialBufferSize the initial buffer size. Must be greater than
0.
* Will be limited to the flush buffer size.
* @param flushBufferSize the buffer size when the stream is flushed. Must
* be greater than 0 or -1 meaning the stream never
* flushes itself.
*/
public BufferedOutputStream(final OutputStream out,
final int initialBufferSize,
final int flushBufferSize) {
super(out);
if (initialBufferSize = 0) {
throw new IllegalArgumentException(Initial buffer size = 0);
}
if (flushBufferSize = 0 flushBufferSize != -1) {
throw new IllegalArgumentException(Flush buffer size = 0 !=
-1);
}
this.buffer =
new byte[initialBufferSize flushBufferSize ? flushBufferSize :
initialBufferSize];
this.flushBufferSize = flushBufferSize;
}
/**
* Writes the specified byte to this buffered output stream.
*
* @param b the byte to be written.
* @exception IOException if an I/O error occurs.
*/
public void write(int b) throws IOException {
if (this.count == this.buffer.length) {
// No need to check return value, can NEVER be 0.
this.increaseBuffer(1);
}
this.buffer[this.count++] = (byte)b;
this.totalCount++;
if (this.count == this.flushBufferSize) {
flush();
}
}
/**
* Writes codelen/code bytes from the specified byte array
* starting at offset codeoff/code to this buffered output stream.
*
* p Ordinarily this method stores bytes from the given array into this
* stream's buffer, flushing the buffer to the underlying output stream as
* needed. If the requested length is at least as large as this stream's
* buffer, however, then this method will flush the buffer and write the
* bytes directly to the underlying output stream. Thus redundant
* codeBufferedOutputStream/codes will not copy data unnecessarily.
*
* @param b the data.
* @param off the start offset in the data.
* @param len the number of bytes to write.
* @exception IOException if an I/O error occurs.
*/
public void write(final byte[] b, final int off, final int len) throws
IOException {
if (len this.buffer.length - this.count) {
int actualIncrease = this.increaseBuffer(len
