RE: EncodeRepresentation.write leaving blocked threads around?
Hi Tim, Thanks again for the suggestion about the queue. That makes total sense. I've just applied the changes to SVN. Best regards, Jerome -Message d'origine- De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] De la part de Tim Peierls Envoyé : mercredi 17 octobre 2007 23:41 À : discuss@restlet.tigris.org Objet : Re: EncodeRepresentation.write leaving blocked threads around? On 10/17/07, Jerome Louvel [EMAIL PROTECTED] wrote: In terms of end to end functionality, our use of GZip encoding works fine and the encoded representations make it from client to server OK, so the concern is simply that there end up being hundreds of these dead corpse threads left over as our software runs for longer. As suggested by Tim, I have added an explicit call to setDaemon(false) in the ByteUtils class, before starting the IO threads. I hope that this can workaround the issue. I think it's good practice to setDaemon(false) as you've done -- though I still would like to know *why* the threads in Matthew's example have daemon status -- but I don't think it will solve Matthew's problem, which is that large numbers of permanently blocked threads are accumulating over long server runs. (The only difference between daemon threads and user threads is that the former don't prevent a JVM from exiting.) I can't think of an explanation for the platform-specific behavior, but it appears that somehow the underlying queue is blocking forever on a put. A simple way to avoid this without having to understand why it is happening is to use queue.offer(byte, timeout, unit) instead of queue.put(byte), throwing IOE if offer fails. Pick the timeout to give the input stream consumer ample time to kick in; some number of seconds, say. Then at least these threads will eventually go away. Likewise, on the input stream side you could use queue.poll(timeout, unit) instead of queue.take(), throwing IOE if the poll fails. Again, pick a timeout to give ample time for the representation to be encoded and partly enqueued. A more robust approach might involve using an ExecutorService to execute the writer tasks asynchronously, instead of explicity creating a Thread. But that's a bigger project that should probably wait until Matthew's issue is resolved. --tim
RE: EncodeRepresentation.write leaving blocked threads around?
Hi Matthew, None of this is code we are writing, its all internal to Restlet, all we are doing is calling EncodeRepresentation.getText() and that method call causes all the other stuff to happen. I can't see how our code has any impact on the closing of the stream and shutting down of the thread. OK. It also appears that this is not a problem under linux, we will test under windows, but it may be its a Mac specific thing. Which HTTP server connector are you using? Have you tried another connector? In terms of end to end functionality, our use of GZip encoding works fine and the encoded representations make it from client to server OK, so the concern is simply that there end up being hundreds of these dead corpse threads left over as our software runs for longer. As suggested by Tim, I have added an explicit call to setDaemon(false) in the ByteUtils class, before starting the IO threads. I hope that this can workaround the issue. Could anyone test? This has been checked in SVN trunk and 1.0 branch. Best regards, Jerome
Re: EncodeRepresentation.write leaving blocked threads around?
Jerome Louvel wrote: Hi Matthew, None of this is code we are writing, its all internal to Restlet, all we are doing is calling EncodeRepresentation.getText() and that method call causes all the other stuff to happen. I can't see how our code has any impact on the closing of the stream and shutting down of the thread. OK. It also appears that this is not a problem under linux, we will test under windows, but it may be its a Mac specific thing. I have tested under windows and it is not an issue. Which HTTP server connector are you using? Have you tried another connector? we are using the org.restlet.Client. In terms of end to end functionality, our use of GZip encoding works fine and the encoded representations make it from client to server OK, so the concern is simply that there end up being hundreds of these dead corpse threads left over as our software runs for longer. As suggested by Tim, I have added an explicit call to setDaemon(false) in the ByteUtils class, before starting the IO threads. I hope that this can workaround the issue. Could anyone test? This has been checked in SVN trunk and 1.0 branch. someone here should be able to test. cheers /jima
Re: EncodeRepresentation.write leaving blocked threads around?
On 10/17/07, Jerome Louvel [EMAIL PROTECTED] wrote: In terms of end to end functionality, our use of GZip encoding works fine and the encoded representations make it from client to server OK, so the concern is simply that there end up being hundreds of these dead corpse threads left over as our software runs for longer. As suggested by Tim, I have added an explicit call to setDaemon(false) in the ByteUtils class, before starting the IO threads. I hope that this can workaround the issue. I think it's good practice to setDaemon(false) as you've done -- though I still would like to know *why* the threads in Matthew's example have daemon status -- but I don't think it will solve Matthew's problem, which is that large numbers of permanently blocked threads are accumulating over long server runs. (The only difference between daemon threads and user threads is that the former don't prevent a JVM from exiting.) I can't think of an explanation for the platform-specific behavior, but it appears that somehow the underlying queue is blocking forever on a put. A simple way to avoid this without having to understand why it is happening is to use queue.offer(byte, timeout, unit) instead of queue.put(byte), throwing IOE if offer fails. Pick the timeout to give the input stream consumer ample time to kick in; some number of seconds, say. Then at least these threads will eventually go away. Likewise, on the input stream side you could use queue.poll(timeout, unit) instead of queue.take(), throwing IOE if the poll fails. Again, pick a timeout to give ample time for the representation to be encoded and partly enqueued. A more robust approach might involve using an ExecutorService to execute the writer tasks asynchronously, instead of explicity creating a Thread. But that's a bigger project that should probably wait until Matthew's issue is resolved. --tim
Re: EncodeRepresentation.write leaving blocked threads around?
Tim Peierls wrote: On 10/17/07, *Jerome Louvel* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: In terms of end to end functionality, our use of GZip encoding works fine and the encoded representations make it from client to server OK, so the concern is simply that there end up being hundreds of these dead corpse threads left over as our software runs for longer. As suggested by Tim, I have added an explicit call to setDaemon(false) in the ByteUtils class, before starting the IO threads. I hope that this can workaround the issue. I think it's good practice to setDaemon(false) as you've done -- though I still would like to know *why* the threads in Matthew's example have daemon status -- but I don't think it will solve Matthew's problem, which is that large numbers of permanently blocked threads are accumulating over long server runs. (The only difference between daemon threads and user threads is that the former don't prevent a JVM from exiting.) I would agree with Tim here. I don't think that this is going to resolve the issue. cheers /jima
Re: EncodeRepresentation.write leaving blocked threads around?
In this case, as with a subsequent message from Jim Alateras, the problem appears to be that the representation size exceeds the capacity of the underlying pipe (1024), and no one is reading from the input stream side of the pipe. But who made these daemon threads? Mixing blocking data structures with daemon threads is a bad idea. To quote from Java Concurrency in Practice: Daemon threads should be used sparingly -- few processing activities can be safely abandoned at any time with no cleanup. In particular, it is dangerous to use daemon threads for tasks that might perform any sort of I/O. Daemon threads are best saved for housekeeping tasks, such as a background thread that periodically removes expired entries from an in-memory cache. Daemon threads are not a good substitute for properly managing the life-cycle of services within an application. --tim On 10/15/07, J. Matthew Pryor [EMAIL PROTECTED] wrote: My running app has about 15 threads this stack traces identical to this? Daemon Thread [Thread-44] (Suspended) Unsafe.park(boolean, long) line: not available [native method] LockSupport.park() line: 118 AbstractQueuedSynchronizer$ConditionObject.await() line: 1767 ArrayBlockingQueueE.put(E) line: 368 ByteUtils$PipeStream$2.write(int) line: 331 ByteUtils$PipeStream$2(OutputStream).write(byte[], int, int) line: 99 GZIPOutputStream.finish() line: 91 [local variables unavailable] EncodeRepresentation.write(OutputStream) line: 235 ByteUtils$2.run() line: 133 We are using EncodeRepresentation and the POST of our resources if working fine, not sure why these threads are staying around Thanks, Matthew
Re: EncodeRepresentation.write leaving blocked threads around?
The thread is created by the ByteUtils class in its getStream() method
/**
* Returns an input stream based on the given representation's
content and
* its write(OutputStream) method. Internally, it uses a writer
thread and a
* pipe stream.
*
* @return A stream with the representation's content.
*/
public static InputStream getStream(final Representation
representation)
throws IOException {
if (representation != null) {
final PipeStream pipe = new PipeStream();
// Creates a thread that will handle the task of
continuously
// writing the representation into the input side of the
pipe
Thread writer = new Thread() {
public void run() {
try {
OutputStream os = pipe.getOutputStream();
representation.write(os);
os.write(-1);
os.close();
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
};
// Starts the writer thread
writer.start();
return pipe.getInputStream();
} else {
return null;
}
}
The problem is that at least on my Mac OSX running Java 5, the above
call to representation.write(os) blocks at this line
encoderOutputStream.finish(); in the method below
/**
* Writes the representation to a byte stream.
*
* @param outputStream
*The output stream.
*/
public void write(OutputStream outputStream) throws IOException {
if (canEncode()) {
DeflaterOutputStream encoderOutputStream = null;
if (this.encoding.equals(Encoding.GZIP)) {
encoderOutputStream = new GZIPOutputStream
(outputStream);
} else if (this.encoding.equals(Encoding.DEFLATE)) {
encoderOutputStream = new DeflaterOutputStream
(outputStream);
} else if (this.encoding.equals(Encoding.ZIP)) {
encoderOutputStream = new ZipOutputStream
(outputStream);
} else if (this.encoding.equals(Encoding.IDENTITY)) {
// Encoder unecessary for identity encoding
}
if (encoderOutputStream != null) {
getWrappedRepresentation().write(encoderOutputStream);
encoderOutputStream.finish();
} else {
getWrappedRepresentation().write(outputStream);
}
} else {
getWrappedRepresentation().write(outputStream);
}
}
On 16/10/2007, at 11:05 PM, Tim Peierls wrote:
In this case, as with a subsequent message from Jim Alateras, the
problem appears to be that the representation size exceeds the
capacity of the underlying pipe (1024), and no one is reading from
the input stream side of the pipe.
But who made these daemon threads? Mixing blocking data structures
with daemon threads is a bad idea. To quote from Java Concurrency
in Practice:
Daemon threads should be used sparingly -- few processing
activities can be safely abandoned at any time with no cleanup. In
particular, it is dangerous to use daemon threads for tasks that
might perform any sort of I/O. Daemon threads are best saved for
housekeeping tasks, such as a background thread that periodically
removes expired entries from an in-memory cache. Daemon threads are
not a good substitute for properly managing the life-cycle of
services within an application.
--tim
On 10/15/07, J. Matthew Pryor [EMAIL PROTECTED] wrote: My
running app has about 15 threads this stack traces identical to this?
Daemon Thread [Thread-44] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park() line: 118
AbstractQueuedSynchronizer$ConditionObject.await() line: 1767
ArrayBlockingQueueE.put(E) line: 368
ByteUtils$PipeStream$2.write(int) line: 331
ByteUtils$PipeStream$2(OutputStream).write(byte[], int,
int) line: 99
GZIPOutputStream.finish() line: 91 [local variables
unavailable]
EncodeRepresentation.write(OutputStream) line: 235
ByteUtils$2.run() line: 133
We are using EncodeRepresentation and the POST of our resources if
working fine, not sure why these threads are staying around
Thanks,
Matthew
Re: EncodeRepresentation.write leaving blocked threads around?
On 10/16/07, J. Matthew Pryor [EMAIL PROTECTED] wrote: The thread is created by the ByteUtils class in its getStream() method Yes, but what are you doing with the InputStream returned by getStream()? I suspect that you aren't reading this stream completely, so the writing thread blocks forever. The writer thread is putting things in a capacity-bound queue; it might be better if the ByteUtils code used timed offer() instead of put(). Independent of the blocking issue, if the thread created by the call to ByteUtils.getStream isDaemon() without a call to setDaemon(), that implies that the calling thread isDaemon(). And that's bad. --tim
Re: EncodeRepresentation.write leaving blocked threads around?
Matt,
Interesting to see whether we get the same problem on Linux or whether
it is specific to the mac. On windows, at least, i don't see any of
these threads.
You said in the email that the post actually completes but the thread
doesn't terminate. Is this correct?
cheers
/jima
J. Matthew Pryor wrote:
The thread is created by the ByteUtils class in its getStream() method
/**
* Returns an input stream based on the given representation's
content and
* its write(OutputStream) method. Internally, it uses a writer
thread and a
* pipe stream.
*
* @return A stream with the representation's content.
*/
public static InputStream getStream(final Representation
representation)
throws IOException {
if (representation != null) {
final PipeStream pipe = new PipeStream();
// Creates a thread that will handle the task of continuously
// writing the representation into the input side of the pipe
Thread writer = new Thread() {
public void run() {
try {
OutputStream os = pipe.getOutputStream();
representation.write(os);
os.write(-1);
os.close();
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
};
// Starts the writer thread
writer.start();
return pipe.getInputStream();
} else {
return null;
}
}
The problem is that at least on my Mac OSX running Java 5, the above
call to representation.write(os) blocks at this line
encoderOutputStream.finish(); in the method below
/**
* Writes the representation to a byte stream.
*
* @param outputStream
*The output stream.
*/
public void write(OutputStream outputStream) throws IOException {
if (canEncode()) {
DeflaterOutputStream encoderOutputStream = null;
if (this.encoding.equals(Encoding.GZIP)) {
encoderOutputStream = new GZIPOutputStream(outputStream);
} else if (this.encoding.equals(Encoding.DEFLATE)) {
encoderOutputStream = new
DeflaterOutputStream(outputStream);
} else if (this.encoding.equals(Encoding.ZIP)) {
encoderOutputStream = new ZipOutputStream(outputStream);
} else if (this.encoding.equals(Encoding.IDENTITY)) {
// Encoder unecessary for identity encoding
}
if (encoderOutputStream != null) {
getWrappedRepresentation().write(encoderOutputStream);
encoderOutputStream.finish();
} else {
getWrappedRepresentation().write(outputStream);
}
} else {
getWrappedRepresentation().write(outputStream);
}
}
On 16/10/2007, at 11:05 PM, Tim Peierls wrote:
In this case, as with a subsequent message from Jim Alateras, the
problem appears to be that the representation size exceeds the
capacity of the underlying pipe (1024), and no one is reading from the
input stream side of the pipe.
But who made these daemon threads? Mixing blocking data structures
with daemon threads is a bad idea. To quote from Java Concurrency in
Practice:
Daemon threads should be used sparingly -- few processing activities
can be safely abandoned at any time with no cleanup. In particular, it
is dangerous to use daemon threads for tasks that might perform any
sort of I/O. Daemon threads are best saved for housekeeping tasks,
such as a background thread that periodically removes expired entries
from an in-memory cache. Daemon threads are not a good substitute for
properly managing the life-cycle of services within an application.
--tim
On 10/15/07, J. Matthew Pryor [EMAIL PROTECTED] wrote: My
running app has about 15 threads this stack traces identical to this?
Daemon Thread [Thread-44] (Suspended)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park() line: 118
AbstractQueuedSynchronizer$ConditionObject.await() line: 1767
ArrayBlockingQueueE.put(E) line: 368
ByteUtils$PipeStream$2.write(int) line: 331
ByteUtils$PipeStream$2(OutputStream).write(byte[], int, int)
line: 99
GZIPOutputStream.finish() line: 91 [local variables unavailable]
EncodeRepresentation.write(OutputStream) line: 235
ByteUtils$2.run() line: 133
We are using EncodeRepresentation and the POST of our resources if
working fine, not sure why these threads are staying around
Thanks,
Matthew
Re: EncodeRepresentation.write leaving blocked threads around?
None of this is code we are writing, its all internal to Restlet, all we are doing is calling EncodeRepresentation.getText() and that method call causes all the other stuff to happen. I can't see how our code has any impact on the closing of the stream and shutting down of the thread. It also appears that this is not a problem under linux, we will test under windows, but it may be its a Mac specific thing. In terms of end to end functionality, our use of GZip encoding works fine and the encoded representations make it from client to server OK, so the concern is simply that there end up being hundreds of these dead corpse threads left over as our software runs for longer. Thanks for your help Matthew On 17/10/2007, at 1:16 AM, Tim Peierls wrote: On 10/16/07, J. Matthew Pryor [EMAIL PROTECTED] wrote: The thread is created by the ByteUtils class in its getStream() method Yes, but what are you doing with the InputStream returned by getStream()? I suspect that you aren't reading this stream completely, so the writing thread blocks forever. The writer thread is putting things in a capacity-bound queue; it might be better if the ByteUtils code used timed offer() instead of put(). Independent of the blocking issue, if the thread created by the call to ByteUtils.getStream isDaemon() without a call to setDaemon (), that implies that the calling thread isDaemon(). And that's bad. --tim
EncodeRepresentation.write leaving blocked threads around?
My running app has about 15 threads this stack traces identical to this? Daemon Thread [Thread-44] (Suspended) Unsafe.park(boolean, long) line: not available [native method] LockSupport.park() line: 118 AbstractQueuedSynchronizer$ConditionObject.await() line: 1767 ArrayBlockingQueueE.put(E) line: 368 ByteUtils$PipeStream$2.write(int) line: 331 ByteUtils$PipeStream$2(OutputStream).write(byte[], int, int) line: 99 GZIPOutputStream.finish() line: 91 [local variables unavailable] EncodeRepresentation.write(OutputStream) line: 235 ByteUtils$2.run() line: 133 We are using EncodeRepresentation and the POST of our resources if working fine, not sure why these threads are staying around Thanks, Matthew
