RE: Cores with FlushableGzipOutputStream

2012-02-16 Thread Allen Reese 


 -Original Message-
 From: Allen Reese [mailto:are...@yahoo-inc.com]
 Sent: Thursday, February 09, 2012 1:28 PM
 To: Tomcat Users List
 Cc: Lars Anderson
 Subject: RE: Cores with FlushableGzipOutputStream
 
 
 
  -Original Message-
  From: Konstantin Kolinko [mailto:knst.koli...@gmail.com]
  Sent: Thursday, February 09, 2012 1:00 PM
  To: Tomcat Users List
  Subject: Re: Cores with FlushableGzipOutputStream
 
  2012/2/10 Allen Reese are...@yahoo-inc.com:
   Try again now that I'm subscribed.
  
  
   -Original Message-
   From: Allen Reese
   Sent: Thursday, February 09, 2012 12:03 PM
   To: 'users@tomcat.apache.org'
   Cc: Lars Anderson
   Subject: Cores with FlushableGzipOutputStream
  
   We've just upgraded from tomcat 6.0.33 to 6.0.35 and started
 having
   the JVM core on our production boxes.
  
   I'm trying to determine what the next course of action should be
  here.
   I have an Oracle Support contract, but they don't seem to see this
  as
   a JVM issue, and blame it on a native lib.
  
  
   Thanks.
  
   Allen Reese
   Core Platforms
   Yahoo!, Inc.
  
   Running on linux x86-64, jdk 6u27, 6u29, 6u30, 7u2
  
   We run several tests and the output is:
  
   Jdk   | Version       | flags
         |
   6u30  | 6.0.33        | compression enabled
         | works
   6u30  | 6.0.35        | compression enabled
         | cores
   6u30  | 6.0.35        | compression disabled
         | works
   6u30  | 6.0.35        | Remove changes to
 FlushableGzipOutputStream
   [1]
         | works
   6u30  | 6.0.35    | -Dsun.zip.disableMemoryMapping=true
         | cores
  
   7u2   | 6.0.35        | compression enabled
         | cores
   7u2   | 6.0.35        | compression disabled
         |
   7u2   | 6.0.35        | Remove changes to
 FlushableGzipOutputStream
   [1]
         |
   7u2   | 6.0.35    | -Dsun.zip.disableMemoryMapping=true
         | cores
  
   https://issues.apache.org/bugzilla/show_bug.cgi?id=52121
  
   I filed an SR with Oracle, as this looks like a JVM bug and got
 the
   following response:
  
   Generic Note
   
   Hi Allen,
  
   Thank you for sending the hotspot error logs (hs_err_pidpid).
   Each one of them has verbiage that indicates the problem is not
   with
  Java,
   but with native code:
  
   # Problematic frame:
   # C [libzip.so+0x77e3] char+0xa3
   #
   # If you would like to submit a bug report, please visit:
   # http://java.sun.com/webapps/bugreport/crash.jsp
   # The crash happened outside the Java Virtual Machine in native
  code.
   # See problematic frame for where to report the bug.
   #
  
   The case description also noted:
  
   Rolling back this patch to tomcat increases stability:
   http://svn.apache.org/viewvc?view=revisionrevision=1197382
  
   Again, this points to software other than Java. The Java defect
   mentioned, 4813885, was fixed in June of 2009.
   =
  
   Allen Reese
   Core Platforms
   Yahoo!, Inc.
  
   [1]: Patch to remove changes to FlushableGZIPOutputStream from
  6.0.35.
  
  (...)
  
 
  Hi!
 
  If you revert r1197382 you will be facing BZ 52121 [1].
 
 
  Overall, BZ 52121 itself was about a bug in Deflater implementation
 in
  JRE. The difference is that I tried to make the new implementation as
  safe as possible,  making sure that Deflater.setLevel() calls do not
  occur in a row, but there is some data between them.
 
  Either way Deflater API does not have such limitation on the use of
  setLevel(), so the root cause of BZ 52121 is a JRE bug.
 
 
  I think that the current code is safe and you hitting a different
  issue. Your better stability is likely a luck caused by a different
  state of Deflater thanks to different implementation at Tomcat side.
 
  Maybe you have some web resource that consistently produces a
 response
  that triggers this issue.
 
  From experience of BZ 52121 we were lucky that someone was able to
  provide sample data that reproduced the issue.  It looks that
  originally it was some product listing generated by JSP page. Maybe
  you can find something similar.
 
  That data is now part of Tomcat testsuite (see
  TestFlushableGZIPOutputStream class).  Note that the reproducer is
  rather simple and does not use Tomcat, but just the streams I/O.
 
 
  Anyway, a JRE should not fail fatally with a core dump regardless of
  what public API the program uses.
 Thanks.  This is what I was after, I'm going to put this thread in my
 SR with Oracle along with Mark's reply.
 
 I'll see if we can get the data that triggers it, it's a high volume
 system and the cores come after about 3 to 4 hours.

I'm looking at this again, and I'm trying do determine how to enhance the test 
case [2] to reproduce what I'm seeing. 

My question is, I have all of the data in pcap format, so I can extract the 
chunks, what is the best way to feed this through.
Do I decompress it all and write it all at once through the stream?
Or is this going to matter based

Re: Cores with FlushableGzipOutputStream

2012-02-16 Thread Mark Thomas 
On 16/02/2012 20:45, Allen Reese wrote:
 My question is, I have all of the data in pcap format, so I can extract the 
 chunks, what is the best way to feed this through.
 Do I decompress it all and write it all at once through the stream?
 Or is this going to matter based on what order the chunks got written in.

Based on the work we did on the previous JVM bug that we worked around,
you'll need to flush at the correct point. I would start by recreating
exactly what you see in the network trace and then start to merge chunks
to simplify as far as you can.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Cores with FlushableGzipOutputStream

2012-02-16 Thread Allen Reese 


 -Original Message-
 From: Mark Thomas [mailto:ma...@apache.org]
 Sent: Thursday, February 16, 2012 12:50 PM
 To: Tomcat Users List
 Subject: Re: Cores with FlushableGzipOutputStream
 
 On 16/02/2012 20:45, Allen Reese wrote:
  My question is, I have all of the data in pcap format, so I can
 extract the chunks, what is the best way to feed this through.
  Do I decompress it all and write it all at once through the stream?
  Or is this going to matter based on what order the chunks got written
 in.
 
 Based on the work we did on the previous JVM bug that we worked around,
 you'll need to flush at the correct point. I would start by recreating
 exactly what you see in the network trace and then start to merge
 chunks to simplify as far as you can.

That's what I was afraid of.  It looks like it's gzipped and then chunked.  I 
guess I need to extract each chunk and try flushing that.

--Allen

 Mark
 
 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Cores with FlushableGzipOutputStream

2012-02-09 Thread Allen Reese 
Try again now that I'm subscribed.


 -Original Message-
 From: Allen Reese
 Sent: Thursday, February 09, 2012 12:03 PM
 To: 'users@tomcat.apache.org'
 Cc: Lars Anderson
 Subject: Cores with FlushableGzipOutputStream
 
 We've just upgraded from tomcat 6.0.33 to 6.0.35 and started having the
 JVM core on our production boxes.
 
 I'm trying to determine what the next course of action should be here.
 I have an Oracle Support contract, but they don't seem to see this as a
 JVM issue, and blame it on a native lib.
 
 
 Thanks.
 
 Allen Reese
 Core Platforms
 Yahoo!, Inc.
 
 Running on linux x86-64, jdk 6u27, 6u29, 6u30, 7u2
 
 We run several tests and the output is:
 
 Jdk   | Version   | flags
   |
 6u30  | 6.0.33| compression enabled
   | works
 6u30  | 6.0.35| compression enabled
   | cores
 6u30  | 6.0.35| compression disabled
   | works
 6u30  | 6.0.35| Remove changes to FlushableGzipOutputStream [1]
   | works
 6u30  | 6.0.35| -Dsun.zip.disableMemoryMapping=true
   | cores
 
 7u2   | 6.0.35| compression enabled
   | cores
 7u2   | 6.0.35| compression disabled
   |
 7u2   | 6.0.35| Remove changes to FlushableGzipOutputStream [1]
   |
 7u2   | 6.0.35| -Dsun.zip.disableMemoryMapping=true
   | cores
 
 https://issues.apache.org/bugzilla/show_bug.cgi?id=52121
 
 I filed an SR with Oracle, as this looks like a JVM bug and got the
 following response:
 
 Generic Note
 
 Hi Allen,
 
 Thank you for sending the hotspot error logs (hs_err_pidpid). Each
 one of them has verbiage that indicates the problem is not with Java,
 but with native code:
 
 # Problematic frame:
 # C [libzip.so+0x77e3] char+0xa3
 #
 # If you would like to submit a bug report, please visit:
 # http://java.sun.com/webapps/bugreport/crash.jsp
 # The crash happened outside the Java Virtual Machine in native code.
 # See problematic frame for where to report the bug.
 #
 
 The case description also noted:
 
 Rolling back this patch to tomcat increases stability:
 http://svn.apache.org/viewvc?view=revisionrevision=1197382
 
 Again, this points to software other than Java. The Java defect
 mentioned, 4813885, was fixed in June of 2009.
 =
 
 Allen Reese
 Core Platforms
 Yahoo!, Inc.
 
 [1]: Patch to remove changes to FlushableGZIPOutputStream from 6.0.35.
 
 --- apache-tomcat-6.0.35-
 src/java/org/apache/coyote/http11/filters/FlushableGZIPOutputStream.jav
 a 2011-11-28 02:22:45.0 -0800
 +++ apache-tomcat-6.0.33-
 src/java/org/apache/coyote/http11/filters/FlushableGZIPOutputStream.jav
 a 2011-08-16 05:26:14.0 -0700
 @@ -35,93 +35,58 @@
  super(os);
  }
 
 +private static final byte[] EMPTYBYTEARRAY = new byte[0];
 +private boolean hasData = false;
 +
  /**
 - * It is used to reserve one byte of real data so that it can be
 used when
 - * flushing the stream.
 + * Here we make sure we have received data, so that the header has
 been for
 + * sure written to the output stream already.
   */
 -private byte[] lastByte = new byte[1];
 -private boolean hasLastByte = false;
 -
 -@Override
 -public void write(byte[] bytes) throws IOException {
 -write(bytes, 0, bytes.length);
 -}
 -
  @Override
 -public synchronized void write(byte[] bytes, int offset, int
 length)
 +public synchronized void write(byte[] bytes, int i, int i1)
  throws IOException {
 -if (length  0) {
 -flushLastByte();
 -if (length  1) {
 -super.write(bytes, offset, length - 1);
 -}
 -rememberLastByte(bytes[offset + length - 1]);
 -}
 +super.write(bytes, i, i1);
 +hasData = true;
  }
 
  @Override
  public synchronized void write(int i) throws IOException {
 -flushLastByte();
 -rememberLastByte((byte) i);
 +super.write(i);
 +hasData = true;
  }
 
  @Override
 -public synchronized void finish() throws IOException {
 -try {
 -flushLastByte();
 -} catch (IOException ignore) {
 -// If our write failed, then trailer write in finish()
 will fail
 -// with IOException as well, but it will leave Deflater in
 more
 -// consistent state.
 -}
 -super.finish();
 +public synchronized void write(byte[] bytes) throws IOException {
 +super.write(bytes);
 +hasData = true;
  }
 
  @Override
 -public synchronized void close() throws IOException {
 -try {
 -flushLastByte();
 -} catch (IOException ignored) {
 -// Ignore. As OutputStream#close() says, the contract of
 close()
 -// is to close the stream. It does not matter much if the
 -// stream is not writable any more.
 +public synchronized void flush()

RE: Cores with FlushableGzipOutputStream

2012-02-09 Thread ken dias 

unsubscribe
 

 From: are...@yahoo-inc.com
 To: users@tomcat.apache.org
 CC: laraa...@yahoo-inc.com
 Date: Thu, 9 Feb 2012 12:07:15 -0800
 Subject: RE: Cores with FlushableGzipOutputStream
 
 Try again now that I'm subscribed.
 
 
  -Original Message-
  From: Allen Reese
  Sent: Thursday, February 09, 2012 12:03 PM
  To: 'users@tomcat.apache.org'
  Cc: Lars Anderson
  Subject: Cores with FlushableGzipOutputStream
  
  We've just upgraded from tomcat 6.0.33 to 6.0.35 and started having the
  JVM core on our production boxes.
  
  I'm trying to determine what the next course of action should be here.
  I have an Oracle Support contract, but they don't seem to see this as a
  JVM issue, and blame it on a native lib.
  
  
  Thanks.
  
  Allen Reese
  Core Platforms
  Yahoo!, Inc.
  
  Running on linux x86-64, jdk 6u27, 6u29, 6u30, 7u2
  
  We run several tests and the output is:
  
  Jdk | Version | flags
  |
  6u30 | 6.0.33 | compression enabled
  | works
  6u30 | 6.0.35 | compression enabled
  | cores
  6u30 | 6.0.35 | compression disabled
  | works
  6u30 | 6.0.35 | Remove changes to FlushableGzipOutputStream [1]
  | works
  6u30 | 6.0.35 | -Dsun.zip.disableMemoryMapping=true
  | cores
  
  7u2 | 6.0.35 | compression enabled
  | cores
  7u2 | 6.0.35 | compression disabled
  |
  7u2 | 6.0.35 | Remove changes to FlushableGzipOutputStream [1]
  |
  7u2 | 6.0.35 | -Dsun.zip.disableMemoryMapping=true
  | cores
  
  https://issues.apache.org/bugzilla/show_bug.cgi?id=52121
  
  I filed an SR with Oracle, as this looks like a JVM bug and got the
  following response:
  
  Generic Note
  
  Hi Allen,
  
  Thank you for sending the hotspot error logs (hs_err_pidpid). Each
  one of them has verbiage that indicates the problem is not with Java,
  but with native code:
  
  # Problematic frame:
  # C [libzip.so+0x77e3] char+0xa3
  #
  # If you would like to submit a bug report, please visit:
  # http://java.sun.com/webapps/bugreport/crash.jsp
  # The crash happened outside the Java Virtual Machine in native code.
  # See problematic frame for where to report the bug.
  #
  
  The case description also noted:
  
  Rolling back this patch to tomcat increases stability:
  http://svn.apache.org/viewvc?view=revisionrevision=1197382
  
  Again, this points to software other than Java. The Java defect
  mentioned, 4813885, was fixed in June of 2009.
  =
  
  Allen Reese
  Core Platforms
  Yahoo!, Inc.
  
  [1]: Patch to remove changes to FlushableGZIPOutputStream from 6.0.35.
  
  --- apache-tomcat-6.0.35-
  src/java/org/apache/coyote/http11/filters/FlushableGZIPOutputStream.jav
  a 2011-11-28 02:22:45.0 -0800
  +++ apache-tomcat-6.0.33-
  src/java/org/apache/coyote/http11/filters/FlushableGZIPOutputStream.jav
  a 2011-08-16 05:26:14.0 -0700
  @@ -35,93 +35,58 @@
  super(os);
  }
  
  + private static final byte[] EMPTYBYTEARRAY = new byte[0];
  + private boolean hasData = false;
  +
  /**
  - * It is used to reserve one byte of real data so that it can be
  used when
  - * flushing the stream.
  + * Here we make sure we have received data, so that the header has
  been for
  + * sure written to the output stream already.
  */
  - private byte[] lastByte = new byte[1];
  - private boolean hasLastByte = false;
  -
  - @Override
  - public void write(byte[] bytes) throws IOException {
  - write(bytes, 0, bytes.length);
  - }
  -
  @Override
  - public synchronized void write(byte[] bytes, int offset, int
  length)
  + public synchronized void write(byte[] bytes, int i, int i1)
  throws IOException {
  - if (length  0) {
  - flushLastByte();
  - if (length  1) {
  - super.write(bytes, offset, length - 1);
  - }
  - rememberLastByte(bytes[offset + length - 1]);
  - }
  + super.write(bytes, i, i1);
  + hasData = true;
  }
  
  @Override
  public synchronized void write(int i) throws IOException {
  - flushLastByte();
  - rememberLastByte((byte) i);
  + super.write(i);
  + hasData = true;
  }
  
  @Override
  - public synchronized void finish() throws IOException {
  - try {
  - flushLastByte();
  - } catch (IOException ignore) {
  - // If our write failed, then trailer write in finish()
  will fail
  - // with IOException as well, but it will leave Deflater in
  more
  - // consistent state.
  - }
  - super.finish();
  + public synchronized void write(byte[] bytes) throws IOException {
  + super.write(bytes);
  + hasData = true;
  }
  
  @Override
  - public synchronized void close() throws IOException {
  - try {
  - flushLastByte();
  - } catch (IOException ignored) {
  - // Ignore. As OutputStream#close() says, the contract of
  close()
  - // is to close the stream. It does not matter much if the
  - // stream is not writable any more.
  + public synchronized void flush() throws IOException {
  + if (!hasData) {
  + return; // do not allow the gzip header to be flushed on
  its own
  }
  - super.close();
  - }
  
  - private void rememberLastByte(byte b

RE: Cores with FlushableGzipOutputStream

2012-02-09 Thread Caldarale, Charles R 
 From: Allen Reese [mailto:are...@yahoo-inc.com] 
 Subject: RE: Cores with FlushableGzipOutputStream

 I'm trying to determine what the next course of action should be here.
 I have an Oracle Support contract, but they don't seem to see this as a
 JVM issue, and blame it on a native lib.

 # Problematic frame:
 # C [libzip.so+0x77e3] char+0xa3

They conveniently forgot that's a library that comes with the JVM...

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Cores with FlushableGzipOutputStream

2012-02-09 Thread Mark Thomas 
On 09/02/2012 20:07, Allen Reese wrote:
 Try again now that I'm subscribed.

If libzip.so is provided by the JVM installer (and I think it is) then
it is still a JVM problem. See [1] for an example where Oracle did
accept that an crash in libzip.so was a bug.

All changing the Tomcat code indicates is that some paths trigger the
bug and some don't. That does not absolve the JVM of responsibility.

Mark

[1] http://bugs.sun.com/view_bug.do?bug_id=6866479

 
 
 -Original Message-
 From: Allen Reese
 Sent: Thursday, February 09, 2012 12:03 PM
 To: 'users@tomcat.apache.org'
 Cc: Lars Anderson
 Subject: Cores with FlushableGzipOutputStream

 We've just upgraded from tomcat 6.0.33 to 6.0.35 and started having the
 JVM core on our production boxes.

 I'm trying to determine what the next course of action should be here.
 I have an Oracle Support contract, but they don't seem to see this as a
 JVM issue, and blame it on a native lib.


 Thanks.

 Allen Reese
 Core Platforms
 Yahoo!, Inc.

 Running on linux x86-64, jdk 6u27, 6u29, 6u30, 7u2

 We run several tests and the output is:

 Jdk  | Version   | flags
  |
 6u30 | 6.0.33| compression enabled
  | works
 6u30 | 6.0.35| compression enabled
  | cores
 6u30 | 6.0.35| compression disabled
  | works
 6u30 | 6.0.35| Remove changes to FlushableGzipOutputStream [1]
  | works
 6u30 | 6.0.35| -Dsun.zip.disableMemoryMapping=true
  | cores

 7u2  | 6.0.35| compression enabled
  | cores
 7u2  | 6.0.35| compression disabled
  |
 7u2  | 6.0.35| Remove changes to FlushableGzipOutputStream [1]
  |
 7u2  | 6.0.35| -Dsun.zip.disableMemoryMapping=true
  | cores

 https://issues.apache.org/bugzilla/show_bug.cgi?id=52121

 I filed an SR with Oracle, as this looks like a JVM bug and got the
 following response:

 Generic Note
 
 Hi Allen,

 Thank you for sending the hotspot error logs (hs_err_pidpid). Each
 one of them has verbiage that indicates the problem is not with Java,
 but with native code:

 # Problematic frame:
 # C [libzip.so+0x77e3] char+0xa3
 #
 # If you would like to submit a bug report, please visit:
 # http://java.sun.com/webapps/bugreport/crash.jsp
 # The crash happened outside the Java Virtual Machine in native code.
 # See problematic frame for where to report the bug.
 #

 The case description also noted:

 Rolling back this patch to tomcat increases stability:
 http://svn.apache.org/viewvc?view=revisionrevision=1197382

 Again, this points to software other than Java. The Java defect
 mentioned, 4813885, was fixed in June of 2009.
 =

 Allen Reese
 Core Platforms
 Yahoo!, Inc.

 [1]: Patch to remove changes to FlushableGZIPOutputStream from 6.0.35.

 --- apache-tomcat-6.0.35-
 src/java/org/apache/coyote/http11/filters/FlushableGZIPOutputStream.jav
 a2011-11-28 02:22:45.0 -0800
 +++ apache-tomcat-6.0.33-
 src/java/org/apache/coyote/http11/filters/FlushableGZIPOutputStream.jav
 a2011-08-16 05:26:14.0 -0700
 @@ -35,93 +35,58 @@
  super(os);
  }

 +private static final byte[] EMPTYBYTEARRAY = new byte[0];
 +private boolean hasData = false;
 +
  /**
 - * It is used to reserve one byte of real data so that it can be
 used when
 - * flushing the stream.
 + * Here we make sure we have received data, so that the header has
 been for
 + * sure written to the output stream already.
   */
 -private byte[] lastByte = new byte[1];
 -private boolean hasLastByte = false;
 -
 -@Override
 -public void write(byte[] bytes) throws IOException {
 -write(bytes, 0, bytes.length);
 -}
 -
  @Override
 -public synchronized void write(byte[] bytes, int offset, int
 length)
 +public synchronized void write(byte[] bytes, int i, int i1)
  throws IOException {
 -if (length  0) {
 -flushLastByte();
 -if (length  1) {
 -super.write(bytes, offset, length - 1);
 -}
 -rememberLastByte(bytes[offset + length - 1]);
 -}
 +super.write(bytes, i, i1);
 +hasData = true;
  }

  @Override
  public synchronized void write(int i) throws IOException {
 -flushLastByte();
 -rememberLastByte((byte) i);
 +super.write(i);
 +hasData = true;
  }

  @Override
 -public synchronized void finish() throws IOException {
 -try {
 -flushLastByte();
 -} catch (IOException ignore) {
 -// If our write failed, then trailer write in finish()
 will fail
 -// with IOException as well, but it will leave Deflater in
 more
 -// consistent state.
 -}
 -super.finish();
 +public synchronized void write(byte[] bytes) throws IOException {
 +super.write(bytes);
 +hasData = true;
  }

  @Override
 -public

Re: Cores with FlushableGzipOutputStream

2012-02-09 Thread Konstantin Kolinko 
2012/2/10 Allen Reese are...@yahoo-inc.com:
 Try again now that I'm subscribed.


 -Original Message-
 From: Allen Reese
 Sent: Thursday, February 09, 2012 12:03 PM
 To: 'users@tomcat.apache.org'
 Cc: Lars Anderson
 Subject: Cores with FlushableGzipOutputStream

 We've just upgraded from tomcat 6.0.33 to 6.0.35 and started having the
 JVM core on our production boxes.

 I'm trying to determine what the next course of action should be here.
 I have an Oracle Support contract, but they don't seem to see this as a
 JVM issue, and blame it on a native lib.


 Thanks.

 Allen Reese
 Core Platforms
 Yahoo!, Inc.

 Running on linux x86-64, jdk 6u27, 6u29, 6u30, 7u2

 We run several tests and the output is:

 Jdk   | Version       | flags
       |
 6u30  | 6.0.33        | compression enabled
       | works
 6u30  | 6.0.35        | compression enabled
       | cores
 6u30  | 6.0.35        | compression disabled
       | works
 6u30  | 6.0.35        | Remove changes to FlushableGzipOutputStream [1]
       | works
 6u30  | 6.0.35    | -Dsun.zip.disableMemoryMapping=true
       | cores

 7u2   | 6.0.35        | compression enabled
       | cores
 7u2   | 6.0.35        | compression disabled
       |
 7u2   | 6.0.35        | Remove changes to FlushableGzipOutputStream [1]
       |
 7u2   | 6.0.35    | -Dsun.zip.disableMemoryMapping=true
       | cores

 https://issues.apache.org/bugzilla/show_bug.cgi?id=52121

 I filed an SR with Oracle, as this looks like a JVM bug and got the
 following response:

 Generic Note
 
 Hi Allen,

 Thank you for sending the hotspot error logs (hs_err_pidpid). Each
 one of them has verbiage that indicates the problem is not with Java,
 but with native code:

 # Problematic frame:
 # C [libzip.so+0x77e3] char+0xa3
 #
 # If you would like to submit a bug report, please visit:
 # http://java.sun.com/webapps/bugreport/crash.jsp
 # The crash happened outside the Java Virtual Machine in native code.
 # See problematic frame for where to report the bug.
 #

 The case description also noted:

 Rolling back this patch to tomcat increases stability:
 http://svn.apache.org/viewvc?view=revisionrevision=1197382

 Again, this points to software other than Java. The Java defect
 mentioned, 4813885, was fixed in June of 2009.
 =

 Allen Reese
 Core Platforms
 Yahoo!, Inc.

 [1]: Patch to remove changes to FlushableGZIPOutputStream from 6.0.35.

(...)


Hi!

If you revert r1197382 you will be facing BZ 52121 [1].


Overall, BZ 52121 itself was about a bug in Deflater implementation in
JRE. The difference is that I tried to make the new implementation as
safe as possible,  making sure that Deflater.setLevel() calls do not
occur in a row, but there is some data between them.

Either way Deflater API does not have such limitation on the use of
setLevel(), so the root cause of BZ 52121 is a JRE bug.


I think that the current code is safe and you hitting a different
issue. Your better stability is likely a luck caused by a different
state of Deflater thanks to different implementation at Tomcat side.

Maybe you have some web resource that consistently produces a response
that triggers this issue.

From experience of BZ 52121 we were lucky that someone was able to
provide sample data that reproduced the issue.  It looks that
originally it was some product listing generated by JSP page. Maybe
you can find something similar.

That data is now part of Tomcat testsuite (see
TestFlushableGZIPOutputStream class).  Note that the reproducer is
rather simple and does not use Tomcat, but just the streams I/O.


Anyway, a JRE should not fail fatally with a core dump regardless of
what public API the program uses.

[1] https://issues.apache.org/bugzilla/show_bug.cgi?id=52121

Best regards,
Konstantin Kolinko

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Cores with FlushableGzipOutputStream

2012-02-09 Thread Allen Reese 


 -Original Message-
 From: Konstantin Kolinko [mailto:knst.koli...@gmail.com]
 Sent: Thursday, February 09, 2012 1:00 PM
 To: Tomcat Users List
 Subject: Re: Cores with FlushableGzipOutputStream
 
 2012/2/10 Allen Reese are...@yahoo-inc.com:
  Try again now that I'm subscribed.
 
 
  -Original Message-
  From: Allen Reese
  Sent: Thursday, February 09, 2012 12:03 PM
  To: 'users@tomcat.apache.org'
  Cc: Lars Anderson
  Subject: Cores with FlushableGzipOutputStream
 
  We've just upgraded from tomcat 6.0.33 to 6.0.35 and started having
  the JVM core on our production boxes.
 
  I'm trying to determine what the next course of action should be
 here.
  I have an Oracle Support contract, but they don't seem to see this
 as
  a JVM issue, and blame it on a native lib.
 
 
  Thanks.
 
  Allen Reese
  Core Platforms
  Yahoo!, Inc.
 
  Running on linux x86-64, jdk 6u27, 6u29, 6u30, 7u2
 
  We run several tests and the output is:
 
  Jdk   | Version       | flags
        |
  6u30  | 6.0.33        | compression enabled
        | works
  6u30  | 6.0.35        | compression enabled
        | cores
  6u30  | 6.0.35        | compression disabled
        | works
  6u30  | 6.0.35        | Remove changes to FlushableGzipOutputStream
  [1]
        | works
  6u30  | 6.0.35    | -Dsun.zip.disableMemoryMapping=true
        | cores
 
  7u2   | 6.0.35        | compression enabled
        | cores
  7u2   | 6.0.35        | compression disabled
        |
  7u2   | 6.0.35        | Remove changes to FlushableGzipOutputStream
  [1]
        |
  7u2   | 6.0.35    | -Dsun.zip.disableMemoryMapping=true
        | cores
 
  https://issues.apache.org/bugzilla/show_bug.cgi?id=52121
 
  I filed an SR with Oracle, as this looks like a JVM bug and got the
  following response:
 
  Generic Note
  
  Hi Allen,
 
  Thank you for sending the hotspot error logs (hs_err_pidpid). Each
  one of them has verbiage that indicates the problem is not with
 Java,
  but with native code:
 
  # Problematic frame:
  # C [libzip.so+0x77e3] char+0xa3
  #
  # If you would like to submit a bug report, please visit:
  # http://java.sun.com/webapps/bugreport/crash.jsp
  # The crash happened outside the Java Virtual Machine in native
 code.
  # See problematic frame for where to report the bug.
  #
 
  The case description also noted:
 
  Rolling back this patch to tomcat increases stability:
  http://svn.apache.org/viewvc?view=revisionrevision=1197382
 
  Again, this points to software other than Java. The Java defect
  mentioned, 4813885, was fixed in June of 2009.
  =
 
  Allen Reese
  Core Platforms
  Yahoo!, Inc.
 
  [1]: Patch to remove changes to FlushableGZIPOutputStream from
 6.0.35.
 
 (...)
 
 
 Hi!
 
 If you revert r1197382 you will be facing BZ 52121 [1].
 
 
 Overall, BZ 52121 itself was about a bug in Deflater implementation in
 JRE. The difference is that I tried to make the new implementation as
 safe as possible,  making sure that Deflater.setLevel() calls do not
 occur in a row, but there is some data between them.
 
 Either way Deflater API does not have such limitation on the use of
 setLevel(), so the root cause of BZ 52121 is a JRE bug.
 
 
 I think that the current code is safe and you hitting a different
 issue. Your better stability is likely a luck caused by a different
 state of Deflater thanks to different implementation at Tomcat side.
 
 Maybe you have some web resource that consistently produces a response
 that triggers this issue.
 
 From experience of BZ 52121 we were lucky that someone was able to
 provide sample data that reproduced the issue.  It looks that
 originally it was some product listing generated by JSP page. Maybe you
 can find something similar.
 
 That data is now part of Tomcat testsuite (see
 TestFlushableGZIPOutputStream class).  Note that the reproducer is
 rather simple and does not use Tomcat, but just the streams I/O.
 
 
 Anyway, a JRE should not fail fatally with a core dump regardless of
 what public API the program uses.
Thanks.  This is what I was after, I'm going to put this thread in my SR with 
Oracle along with Mark's reply.

I'll see if we can get the data that triggers it, it's a high volume system and 
the cores come after about 3 to 4 hours.


 [1] https://issues.apache.org/bugzilla/show_bug.cgi?id=52121
 
 Best regards,
 Konstantin Kolinko
 
 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org