Re: JsonParserFactoryImpl holds 20mb heap memory when used as jaxrs client in tomee

[Romain Manni-Bucau]

2016-04-03 19:13 GMT+02:00 ravi sankar :
> Hi Romain,
> I have done some tests with JacksonProvider
>
> final Client client = clientBuilder.build().register(new 
> JacksonJsonProvider());
> final ExecutorService es = Executors.newFixedThreadPool(100);
> Json Message - 
> https://raw.githubusercontent.com/Ravisankar-Challa/tomee_embedded/master/src/main/resources/miscellaneousinfo.js
> Heap Memory 64 MB - No Issue
>
>
> final ExecutorService es = Executors.newFixedThreadPool(34);
> This time I had a String in json message around 3MB
> Heap Memory 768 MB - No issues
>

jackson doesn't protect anything there and let to the user/infra the
security concerns.

> So Jackson does handles Json strings of different sizes with out any settings.
> If auto adjustment causes problems in Johnzon then we can leave it as is.
>

that's more in term of design than implementation.

FYI:

import com.example.rest.JsonContentTypeResponseFilter;
import com.example.rest.model.misc.MiscellaneousInfo;
import com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider;

import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;

public class Run {
public static void main(final String[] args) throws InterruptedException {
final ExecutorService es = Executors.newFixedThreadPool(128);
final ClientBuilder clientBuilder = ClientBuilder.newBuilder();
clientBuilder.property("http.connection.timeout",
String.valueOf(3));
clientBuilder.property("http.receive.timeout", String.valueOf(3));
final Client client = clientBuilder.build().register(new
JacksonJsonProvider());
final AtomicInteger errors = new AtomicInteger();
for (int i = 0; i < 1000; i++) {
es.submit(() -> {
try {
if
(client.target("https://raw.githubusercontent.com/Ravisankar-Challa/tomee_embedded/master/src/main/resources/miscellaneousinfo.js;)
.register(JsonContentTypeResponseFilter.class)
.request()
.get(MiscellaneousInfo.class).getTitles() == null) {
throw new IllegalArgumentException();
}
} catch (final Exception e) {
e.printStackTrace();
errors.incrementAndGet();
}
});
}
es.shutdown();
es.awaitTermination(1, TimeUnit.HOURS);
System.out.println("Errors: " + errors.get());
}
}

ends with Caused by: java.lang.OutOfMemoryError: GC overhead limit
exceeded too. Didn't check if lower numbers avoids it but 128 threads
and 1000 iterations is a few for a webapp ;)


> Thanks,Ravisankar Challa
>
> On Monday, 4 April 2016 2:30 AM, Romain Manni-Bucau 
>  wrote:
>
>
>  http is paticular cause you have the content length so we could maybe
> do a bit better but even jackson has this dynamic mem issue.
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Github | LinkedIn | Tomitriber
>
>
> 2016-04-03 18:26 GMT+02:00 ravi sankar :
>> Hi Romain,
>> Yes you are right it doesn't solve the actual problem. I wonder how Jackson 
>> is able to handle payloads of different sizes out of the box.
>>
>> Thanks,Ravi
>>
>>On Monday, 4 April 2016 12:02 AM, Romain Manni-Bucau 
>>  wrote:
>>
>>
>>  Hi Ravi,
>>
>> 2016-04-03 14:44 GMT+02:00 ravi sankar :
>>> Hi Romain,
>>> Scenario: Retrieve the json file at 
>>> https://raw.githubusercontent.com/Ravisankar-Challa/tomee_embedded/master/src/main/resources/miscellaneousinfo.js
>>> using JaxRs client.
>>>
>>> git checkout master
>>>
>>> mvn -f tomee-embedded-shade.xml clean package -Dmaven.test.skip
>>>
>>> java -Xmx768M -jar target/tomeedemo.jar -c
>>>
>>> Using Apache Jmeter I tried firing 50 concurrent requests. 
>>> (Url:http://localhost:8080/api/client/test)
>>> Result: SEVERE: java.lang.OutOfMemoryError: Java heap space
>>>javax.ws.rs.ProcessingException: java.lang.OutOfMemoryError: Java heap 
>>> space
>>>at 
>>> org.apache.cxf.jaxrs.client.WebClient.handleResponse(WebClient.java:1187)
>>>
>>
>> reproduced with the same JVM settings and:
>>
>> import com.example.rest.JsonContentTypeResponseFilter;
>> import com.example.rest.model.misc.MiscellaneousInfo;
>> import org.apache.johnzon.jaxrs.JohnzonProvider;
>>
>> import javax.ws.rs.client.Client;
>> import javax.ws.rs.client.ClientBuilder;
>> import java.util.concurrent.ExecutorService;
>> import java.util.concurrent.Executors;
>> import java.util.concurrent.TimeUnit;
>> import java.util.concurrent.atomic.AtomicInteger;
>>
>> public class Run {
>>public static void main(final String[] args) throws 

Re: JsonParserFactoryImpl holds 20mb heap memory when used as jaxrs client in tomee

[ravi sankar]

Hi Romain,
I have done some tests with JacksonProvider

final Client client = clientBuilder.build().register(new 
JacksonJsonProvider());  
final ExecutorService es = Executors.newFixedThreadPool(100); 
Json Message - 
https://raw.githubusercontent.com/Ravisankar-Challa/tomee_embedded/master/src/main/resources/miscellaneousinfo.js
Heap Memory 64 MB - No Issue


final ExecutorService es = Executors.newFixedThreadPool(34);
This time I had a String in json message around 3MB
Heap Memory 768 MB - No issues

So Jackson does handles Json strings of different sizes with out any settings.
If auto adjustment causes problems in Johnzon then we can leave it as is.

Thanks,Ravisankar Challa 

On Monday, 4 April 2016 2:30 AM, Romain Manni-Bucau  
wrote:
 

 http is paticular cause you have the content length so we could maybe
do a bit better but even jackson has this dynamic mem issue.

Romain Manni-Bucau
@rmannibucau |  Blog | Github | LinkedIn | Tomitriber


2016-04-03 18:26 GMT+02:00 ravi sankar :
> Hi Romain,
> Yes you are right it doesn't solve the actual problem. I wonder how Jackson 
> is able to handle payloads of different sizes out of the box.
>
> Thanks,Ravi
>
>    On Monday, 4 April 2016 12:02 AM, Romain Manni-Bucau 
> wrote:
>
>
>  Hi Ravi,
>
> 2016-04-03 14:44 GMT+02:00 ravi sankar :
>> Hi Romain,
>> Scenario: Retrieve the json file at 
>> https://raw.githubusercontent.com/Ravisankar-Challa/tomee_embedded/master/src/main/resources/miscellaneousinfo.js
>> using JaxRs client.
>>
>> git checkout master
>>
>> mvn -f tomee-embedded-shade.xml clean package -Dmaven.test.skip
>>
>> java -Xmx768M -jar target/tomeedemo.jar -c
>>
>> Using Apache Jmeter I tried firing 50 concurrent requests. 
>> (Url:http://localhost:8080/api/client/test)
>> Result: SEVERE: java.lang.OutOfMemoryError: Java heap space
>>    javax.ws.rs.ProcessingException: java.lang.OutOfMemoryError: Java heap 
>>space
>>    at 
>>org.apache.cxf.jaxrs.client.WebClient.handleResponse(WebClient.java:1187)
>>
>
> reproduced with the same JVM settings and:
>
> import com.example.rest.JsonContentTypeResponseFilter;
> import com.example.rest.model.misc.MiscellaneousInfo;
> import org.apache.johnzon.jaxrs.JohnzonProvider;
>
> import javax.ws.rs.client.Client;
> import javax.ws.rs.client.ClientBuilder;
> import java.util.concurrent.ExecutorService;
> import java.util.concurrent.Executors;
> import java.util.concurrent.TimeUnit;
> import java.util.concurrent.atomic.AtomicInteger;
>
> public class Run {
>    public static void main(final String[] args) throws InterruptedException {
>        final ExecutorService es = Executors.newFixedThreadPool(64);
>        final ClientBuilder clientBuilder = ClientBuilder.newBuilder();
>        clientBuilder.property("http.connection.timeout",
> String.valueOf(3));
>        clientBuilder.property("http.receive.timeout", String.valueOf(3));
>        final Client client = clientBuilder.build().register(new
> JohnzonProvider<>());
>        final AtomicInteger errors = new AtomicInteger();
>        for (int i = 0; i < 100; i++) {
>            es.submit(() -> {
>                try {
>                    if
> (client.target("https://raw.githubusercontent.com/Ravisankar-Challa/tomee_embedded/master/src/main/resources/miscellaneousinfo.js;)
>                            .register(JsonContentTypeResponseFilter.class)
>                            .request()
>                            .get(MiscellaneousInfo.class).getTitles() == null) 
>{
>                        throw new IllegalArgumentException();
>                    }
>                } catch (final Exception e) {
>                    e.printStackTrace();
>                    errors.incrementAndGet();
>                }
>            });
>        }
>        es.shutdown();
>        es.awaitTermination(1, TimeUnit.HOURS);
>        System.out.println("Errors: " + errors.get());
>    }
> }
>
>
>
> That said also setting -Dorg.apache.johnzon.max-string-length=4096
> solved it - can be done outside the application, typically in
> conf/system.properties.
>
>> I have analyzed the heap dump using Ecilpse mat.
>> One instance of "org.apache.johnzon.core.JsonReaderFactoryImpl" loaded by 
>> "sun.misc.Launcher$AppClassLoader @ 0xc019df10" occupies 717,491,152 
>> (91.98%) bytes.
>> The memory is accumulated in one instance of 
>> "java.util.concurrent.ConcurrentLinkedQueue$Node" loaded by "> loader>".
>>
>> valueBufferProvider of JsonParserFactoryImpl.java is holding all this memory.
>>
>> Auto adjust buffer Fix:
>> JsonParserFactoryImpl.java line number 64this.valueBufferProvider = 
>> getBufferProvider().newCharProvider(1);  //Set initial size to 1
>>
>> JsonStreamParserImpl.java line number 70
>> private char[] fallBackCopyBuffer;
>>
>> JsonStreamParserImpl.java line number 562
>> else if((endOfValueInBuffer - startOfValueInBuffer) > 
>> 

Re: JsonParserFactoryImpl holds 20mb heap memory when used as jaxrs client in tomee

[Romain Manni-Bucau]

http is paticular cause you have the content length so we could maybe
do a bit better but even jackson has this dynamic mem issue.

Romain Manni-Bucau
@rmannibucau |  Blog | Github | LinkedIn | Tomitriber


2016-04-03 18:26 GMT+02:00 ravi sankar :
> Hi Romain,
> Yes you are right it doesn't solve the actual problem. I wonder how Jackson 
> is able to handle payloads of different sizes out of the box.
>
> Thanks,Ravi
>
> On Monday, 4 April 2016 12:02 AM, Romain Manni-Bucau 
>  wrote:
>
>
>  Hi Ravi,
>
> 2016-04-03 14:44 GMT+02:00 ravi sankar :
>> Hi Romain,
>> Scenario: Retrieve the json file at 
>> https://raw.githubusercontent.com/Ravisankar-Challa/tomee_embedded/master/src/main/resources/miscellaneousinfo.js
>> using JaxRs client.
>>
>> git checkout master
>>
>> mvn -f tomee-embedded-shade.xml clean package -Dmaven.test.skip
>>
>> java -Xmx768M -jar target/tomeedemo.jar -c
>>
>> Using Apache Jmeter I tried firing 50 concurrent requests. 
>> (Url:http://localhost:8080/api/client/test)
>> Result: SEVERE: java.lang.OutOfMemoryError: Java heap space
>>javax.ws.rs.ProcessingException: java.lang.OutOfMemoryError: Java heap 
>> space
>>at 
>> org.apache.cxf.jaxrs.client.WebClient.handleResponse(WebClient.java:1187)
>>
>
> reproduced with the same JVM settings and:
>
> import com.example.rest.JsonContentTypeResponseFilter;
> import com.example.rest.model.misc.MiscellaneousInfo;
> import org.apache.johnzon.jaxrs.JohnzonProvider;
>
> import javax.ws.rs.client.Client;
> import javax.ws.rs.client.ClientBuilder;
> import java.util.concurrent.ExecutorService;
> import java.util.concurrent.Executors;
> import java.util.concurrent.TimeUnit;
> import java.util.concurrent.atomic.AtomicInteger;
>
> public class Run {
> public static void main(final String[] args) throws InterruptedException {
> final ExecutorService es = Executors.newFixedThreadPool(64);
> final ClientBuilder clientBuilder = ClientBuilder.newBuilder();
> clientBuilder.property("http.connection.timeout",
> String.valueOf(3));
> clientBuilder.property("http.receive.timeout", String.valueOf(3));
> final Client client = clientBuilder.build().register(new
> JohnzonProvider<>());
> final AtomicInteger errors = new AtomicInteger();
> for (int i = 0; i < 100; i++) {
> es.submit(() -> {
> try {
> if
> (client.target("https://raw.githubusercontent.com/Ravisankar-Challa/tomee_embedded/master/src/main/resources/miscellaneousinfo.js;)
> .register(JsonContentTypeResponseFilter.class)
> .request()
> .get(MiscellaneousInfo.class).getTitles() == 
> null) {
> throw new IllegalArgumentException();
> }
> } catch (final Exception e) {
> e.printStackTrace();
> errors.incrementAndGet();
> }
> });
> }
> es.shutdown();
> es.awaitTermination(1, TimeUnit.HOURS);
> System.out.println("Errors: " + errors.get());
> }
> }
>
>
>
> That said also setting -Dorg.apache.johnzon.max-string-length=4096
> solved it - can be done outside the application, typically in
> conf/system.properties.
>
>> I have analyzed the heap dump using Ecilpse mat.
>> One instance of "org.apache.johnzon.core.JsonReaderFactoryImpl" loaded by 
>> "sun.misc.Launcher$AppClassLoader @ 0xc019df10" occupies 717,491,152 
>> (91.98%) bytes.
>> The memory is accumulated in one instance of 
>> "java.util.concurrent.ConcurrentLinkedQueue$Node" loaded by "> loader>".
>>
>> valueBufferProvider of JsonParserFactoryImpl.java is holding all this memory.
>>
>> Auto adjust buffer Fix:
>> JsonParserFactoryImpl.java line number 64this.valueBufferProvider = 
>> getBufferProvider().newCharProvider(1);  //Set initial size to 1
>>
>> JsonStreamParserImpl.java line number 70
>> private char[] fallBackCopyBuffer;
>>
>> JsonStreamParserImpl.java line number 562
>> else if((endOfValueInBuffer - startOfValueInBuffer) > 
>> fallBackCopyBuffer.length) {
>>System.out.println("Adjusting Buffer from "+fallBackCopyBuffer.length+" 
>> to "+(endOfValueInBuffer-startOfValueInBuffer));
>>fallBackCopyBuffer = new char[endOfValueInBuffer - startOfValueInBuffer]; 
>>  //Auto adjust to required size
>> }
>>
>
> This doesn't solve the whole issue. of course then you have a small
> buffer but now let's enrich your .js to have a string of ~10M (can
> happen when you don't control what you call). Then you:
>
> 1. still have your issue
> 2. have no way to control the memory you can allocate (today you can
> computing threads using the mapper x memory)
>
>
> I'm happy to resize the default (just start a discussion on
> dev@johnzon list) to something more reasonable (we have to check
> twitter, 

Re: JsonParserFactoryImpl holds 20mb heap memory when used as jaxrs client in tomee

[ravi sankar]

Hi Romain,
Yes you are right it doesn't solve the actual problem. I wonder how Jackson is 
able to handle payloads of different sizes out of the box.

Thanks,Ravi 

On Monday, 4 April 2016 12:02 AM, Romain Manni-Bucau 
 wrote:
 

 Hi Ravi,

2016-04-03 14:44 GMT+02:00 ravi sankar :
> Hi Romain,
> Scenario: Retrieve the json file at 
> https://raw.githubusercontent.com/Ravisankar-Challa/tomee_embedded/master/src/main/resources/miscellaneousinfo.js
> using JaxRs client.
>
> git checkout master
>
> mvn -f tomee-embedded-shade.xml clean package -Dmaven.test.skip
>
> java -Xmx768M -jar target/tomeedemo.jar -c
>
> Using Apache Jmeter I tried firing 50 concurrent requests. 
> (Url:http://localhost:8080/api/client/test)
> Result: SEVERE: java.lang.OutOfMemoryError: Java heap space
>    javax.ws.rs.ProcessingException: java.lang.OutOfMemoryError: Java heap 
>space
>    at 
>org.apache.cxf.jaxrs.client.WebClient.handleResponse(WebClient.java:1187)
>

reproduced with the same JVM settings and:

import com.example.rest.JsonContentTypeResponseFilter;
import com.example.rest.model.misc.MiscellaneousInfo;
import org.apache.johnzon.jaxrs.JohnzonProvider;

import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;

public class Run {
    public static void main(final String[] args) throws InterruptedException {
        final ExecutorService es = Executors.newFixedThreadPool(64);
        final ClientBuilder clientBuilder = ClientBuilder.newBuilder();
        clientBuilder.property("http.connection.timeout",
String.valueOf(3));
        clientBuilder.property("http.receive.timeout", String.valueOf(3));
        final Client client = clientBuilder.build().register(new
JohnzonProvider<>());
        final AtomicInteger errors = new AtomicInteger();
        for (int i = 0; i < 100; i++) {
            es.submit(() -> {
                try {
                    if
(client.target("https://raw.githubusercontent.com/Ravisankar-Challa/tomee_embedded/master/src/main/resources/miscellaneousinfo.js;)
                            .register(JsonContentTypeResponseFilter.class)
                            .request()
                            .get(MiscellaneousInfo.class).getTitles() == null) {
                        throw new IllegalArgumentException();
                    }
                } catch (final Exception e) {
                    e.printStackTrace();
                    errors.incrementAndGet();
                }
            });
        }
        es.shutdown();
        es.awaitTermination(1, TimeUnit.HOURS);
        System.out.println("Errors: " + errors.get());
    }
}



That said also setting -Dorg.apache.johnzon.max-string-length=4096
solved it - can be done outside the application, typically in
conf/system.properties.

> I have analyzed the heap dump using Ecilpse mat.
> One instance of "org.apache.johnzon.core.JsonReaderFactoryImpl" loaded by 
> "sun.misc.Launcher$AppClassLoader @ 0xc019df10" occupies 717,491,152 (91.98%) 
> bytes.
> The memory is accumulated in one instance of 
> "java.util.concurrent.ConcurrentLinkedQueue$Node" loaded by " loader>".
>
> valueBufferProvider of JsonParserFactoryImpl.java is holding all this memory.
>
> Auto adjust buffer Fix:
> JsonParserFactoryImpl.java line number 64this.valueBufferProvider = 
> getBufferProvider().newCharProvider(1);  //Set initial size to 1
>
> JsonStreamParserImpl.java line number 70
> private char[] fallBackCopyBuffer;
>
> JsonStreamParserImpl.java line number 562
> else if((endOfValueInBuffer - startOfValueInBuffer) > 
> fallBackCopyBuffer.length) {
>    System.out.println("Adjusting Buffer from "+fallBackCopyBuffer.length+" to 
>"+(endOfValueInBuffer-startOfValueInBuffer));
>    fallBackCopyBuffer = new char[endOfValueInBuffer - startOfValueInBuffer];  
>//Auto adjust to required size
> }
>

This doesn't solve the whole issue. of course then you have a small
buffer but now let's enrich your .js to have a string of ~10M (can
happen when you don't control what you call). Then you:

1. still have your issue
2. have no way to control the memory you can allocate (today you can
computing threads using the mapper x memory)


I'm happy to resize the default (just start a discussion on
dev@johnzon list) to something more reasonable (we have to check
twitter, github, jira at least) but I think this auto adjusting
solution is a seducing solution which can lead us to crossing fingers
in production which is probably the worse I can see when deploying an
application.

Wdyt?

> git checkout johnzon_buffer_fix
> mvn -f tomee-embedded-shade.xml clean package -Dmaven.test.skip
> java -Xmx768M -jar target/tomeedemo.jar -c
>
> Using Apache Jmeter I tried firing 50 concurrent requests. 
> (Url:http://localhost:8080/api/client/test)
> 

Re: JsonParserFactoryImpl holds 20mb heap memory when used as jaxrs client in tomee

[Romain Manni-Bucau]

Hi Ravi,

2016-04-03 14:44 GMT+02:00 ravi sankar :
> Hi Romain,
> Scenario: Retrieve the json file at 
> https://raw.githubusercontent.com/Ravisankar-Challa/tomee_embedded/master/src/main/resources/miscellaneousinfo.js
> using JaxRs client.
>
> git checkout master
>
> mvn -f tomee-embedded-shade.xml clean package -Dmaven.test.skip
>
> java -Xmx768M -jar target/tomeedemo.jar -c
>
> Using Apache Jmeter I tried firing 50 concurrent requests. 
> (Url:http://localhost:8080/api/client/test)
> Result: SEVERE: java.lang.OutOfMemoryError: Java heap space
> javax.ws.rs.ProcessingException: java.lang.OutOfMemoryError: Java heap 
> space
> at 
> org.apache.cxf.jaxrs.client.WebClient.handleResponse(WebClient.java:1187)
>

reproduced with the same JVM settings and:

import com.example.rest.JsonContentTypeResponseFilter;
import com.example.rest.model.misc.MiscellaneousInfo;
import org.apache.johnzon.jaxrs.JohnzonProvider;

import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;

public class Run {
public static void main(final String[] args) throws InterruptedException {
final ExecutorService es = Executors.newFixedThreadPool(64);
final ClientBuilder clientBuilder = ClientBuilder.newBuilder();
clientBuilder.property("http.connection.timeout",
String.valueOf(3));
clientBuilder.property("http.receive.timeout", String.valueOf(3));
final Client client = clientBuilder.build().register(new
JohnzonProvider<>());
final AtomicInteger errors = new AtomicInteger();
for (int i = 0; i < 100; i++) {
es.submit(() -> {
try {
if
(client.target("https://raw.githubusercontent.com/Ravisankar-Challa/tomee_embedded/master/src/main/resources/miscellaneousinfo.js;)
.register(JsonContentTypeResponseFilter.class)
.request()
.get(MiscellaneousInfo.class).getTitles() == null) {
throw new IllegalArgumentException();
}
} catch (final Exception e) {
e.printStackTrace();
errors.incrementAndGet();
}
});
}
es.shutdown();
es.awaitTermination(1, TimeUnit.HOURS);
System.out.println("Errors: " + errors.get());
}
}



That said also setting -Dorg.apache.johnzon.max-string-length=4096
solved it - can be done outside the application, typically in
conf/system.properties.

> I have analyzed the heap dump using Ecilpse mat.
> One instance of "org.apache.johnzon.core.JsonReaderFactoryImpl" loaded by 
> "sun.misc.Launcher$AppClassLoader @ 0xc019df10" occupies 717,491,152 (91.98%) 
> bytes.
> The memory is accumulated in one instance of 
> "java.util.concurrent.ConcurrentLinkedQueue$Node" loaded by " loader>".
>
> valueBufferProvider of JsonParserFactoryImpl.java is holding all this memory.
>
> Auto adjust buffer Fix:
> JsonParserFactoryImpl.java line number 64this.valueBufferProvider = 
> getBufferProvider().newCharProvider(1);  //Set initial size to 1
>
> JsonStreamParserImpl.java line number 70
> private char[] fallBackCopyBuffer;
>
> JsonStreamParserImpl.java line number 562
> else if((endOfValueInBuffer - startOfValueInBuffer) > 
> fallBackCopyBuffer.length) {
> System.out.println("Adjusting Buffer from "+fallBackCopyBuffer.length+" 
> to "+(endOfValueInBuffer-startOfValueInBuffer));
> fallBackCopyBuffer = new char[endOfValueInBuffer - startOfValueInBuffer]; 
>  //Auto adjust to required size
> }
>

This doesn't solve the whole issue. of course then you have a small
buffer but now let's enrich your .js to have a string of ~10M (can
happen when you don't control what you call). Then you:

1. still have your issue
2. have no way to control the memory you can allocate (today you can
computing threads using the mapper x memory)


I'm happy to resize the default (just start a discussion on
dev@johnzon list) to something more reasonable (we have to check
twitter, github, jira at least) but I think this auto adjusting
solution is a seducing solution which can lead us to crossing fingers
in production which is probably the worse I can see when deploying an
application.

Wdyt?

> git checkout johnzon_buffer_fix
> mvn -f tomee-embedded-shade.xml clean package -Dmaven.test.skip
> java -Xmx768M -jar target/tomeedemo.jar -c
>
> Using Apache Jmeter I tried firing 50 concurrent requests. 
> (Url:http://localhost:8080/api/client/test)
> Using Ecilipse mat I analyzed the Heap Dump and didn't found any issues.
>
> Thanks,Ravisankar Challa
>
> On Friday, 1 April 2016 1:23 AM, Romain Manni-Bucau 
>  wrote:
>
>
>  2016-03-31 15:58 GMT+02:00 ravi sankar 

Re: JsonParserFactoryImpl holds 20mb heap memory when used as jaxrs client in tomee

[ravi sankar]

Hi Romain,
Scenario: Retrieve the json file at 
https://raw.githubusercontent.com/Ravisankar-Challa/tomee_embedded/master/src/main/resources/miscellaneousinfo.js
 
using JaxRs client.

git checkout master

mvn -f tomee-embedded-shade.xml clean package -Dmaven.test.skip

java -Xmx768M -jar target/tomeedemo.jar -c

Using Apache Jmeter I tried firing 50 concurrent requests. 
(Url:http://localhost:8080/api/client/test)
Result: SEVERE: java.lang.OutOfMemoryError: Java heap space
    javax.ws.rs.ProcessingException: java.lang.OutOfMemoryError: Java heap space
    at org.apache.cxf.jaxrs.client.WebClient.handleResponse(WebClient.java:1187)

I have analyzed the heap dump using Ecilpse mat. 
One instance of "org.apache.johnzon.core.JsonReaderFactoryImpl" loaded by 
"sun.misc.Launcher$AppClassLoader @ 0xc019df10" occupies 717,491,152 (91.98%) 
bytes. 
The memory is accumulated in one instance of 
"java.util.concurrent.ConcurrentLinkedQueue$Node" loaded by "".

valueBufferProvider of JsonParserFactoryImpl.java is holding all this memory.

Auto adjust buffer Fix:
JsonParserFactoryImpl.java line number 64this.valueBufferProvider = 
getBufferProvider().newCharProvider(1);  //Set initial size to 1

JsonStreamParserImpl.java line number 70
private char[] fallBackCopyBuffer;

JsonStreamParserImpl.java line number 562
else if((endOfValueInBuffer - startOfValueInBuffer) > 
fallBackCopyBuffer.length) {
    System.out.println("Adjusting Buffer from "+fallBackCopyBuffer.length+" to 
"+(endOfValueInBuffer-startOfValueInBuffer));
    fallBackCopyBuffer = new char[endOfValueInBuffer - startOfValueInBuffer];  
//Auto adjust to required size
}

git checkout johnzon_buffer_fix
mvn -f tomee-embedded-shade.xml clean package -Dmaven.test.skip
java -Xmx768M -jar target/tomeedemo.jar -c

Using Apache Jmeter I tried firing 50 concurrent requests. 
(Url:http://localhost:8080/api/client/test)
Using Ecilipse mat I analyzed the Heap Dump and didn't found any issues.

Thanks,Ravisankar Challa 

On Friday, 1 April 2016 1:23 AM, Romain Manni-Bucau  
wrote:
 

 2016-03-31 15:58 GMT+02:00 ravi sankar :
> right now valueBufferProvider  is eating up 20mb by default, even for small 
> json strings/numbers.
> org.apache.johnzon.max-string-length is there to control the buffer size but 
> how many people will be aware of this option. Every one expects the framework 
> to do best with default settings :-)
>

I get it but this setting is important - this thread is not about that
so let's not detail there - and if we use a lower value then at
runtime, after 1 week of run you start to just fail and need to
redeploy if you didnt identify it: this is way worse. If we implement
the automatic adjustment: same punishment: it works, it works, OOME,
restart, it works, OOME, restart, ...

Side note: a switch of the impl from char[] to StringBuilder or
equivalent is not an option cause it is insanely slow compared to
current version, only possible auto adjustment would really be "oops
it fails, can I increase?". If you want to give it a try for a patch
(even without adding a min option but using min(8k, max) as min) I
would be more than happy to review it but please ensure this behavior
is logged explicitly at startup and at runtime in case of adjustment
in INFO level - stating the buffer was not adapted and should be
configured - each time the buffer is adjusted and probably use an
ArrayList like algorithm (adjustment is not +1 char but size*2 in the
limit of the max value). If your patch doesn't imply any slow down or
is done thanks to a BufferProvider (= configurable) then I'm happy to
push it upstream.



>
>    On Friday, 1 April 2016 12:32 AM, Romain Manni-Bucau 
> wrote:
>
>
>  2016-03-31 15:26 GMT+02:00 ravi sankar :
>> Hi Romain,
>>>setting a min-max means you'll use max only pretty quickly with 
>>>auto-adjustment.Lets assume JsonParser has the following default buffer 
>>>sizes of min value 1 and max 10 * 1024 * 1024.and needs to process a message 
>>>like this {"hello":"hai"}
>> Parser adjusts the buffer size to 5 to process the json field "hello". It 
>> never gonna hit maximum value.
>
> What I mean if you will parse different json and will keep the bigger
> size you visited. So you will likely be at max so min is useless. You
> can adjust your max but any adjustment doesn't bring much in practise
> excepted some memory instability.
>
>> Thanks,Ravi
>>
>>    On Thursday, 31 March 2016 11:45 PM, Romain Manni-Bucau 
>> wrote:
>>
>>
>>  2016-03-31 14:34 GMT+02:00 ravi sankar :
>>> Hi Romain,
>>
>> Hello Ravi,
>>
>>>
>>> I noticed the below issue when using JaxRsClient code.
>>>
>>> Looking in to the code of JsonParserFactoryImpl.java
>>>
>>>  public static final String MAX_STRING_LENGTH = 
>>>"org.apache.johnzon.max-string-length";
>>>  public static final int 

Re: Problem fetching JSF resources over SSL when ActiveMQ broker is running

[Ihsan Ecemis]

Is SSL config different for JSF resources than static files under Tomcat?  Note 
that I can fetch static files without any problem,  I have the problem only 
with JSF resources!  (I didn’t add static files to the sample project to show 
this but my application have them)

If you’d like I can setup an EC2 server for you to try this.Would you have 
a few minutes to login and see it for yourself?  
 

> On Apr 3, 2016, at 7:07 AM, Romain Manni-Bucau  wrote:
> 
> tried on windows so wonder if it can be linked to gnu tools.
> 
> If you get such an issue the only thing I can think about is AMQ
> changing the SSL config of the JVM which would then make the SSL
> config of tomcat changing if relying on the JVM settings for any of
> them.
> 
> Can also be a local firewall etc even if weird for a UNIx box - saw
> several company windows boxes to fail even on localhost cause of
> McAfee or equivalent.
> 
> 
> Romain Manni-Bucau
> @rmannibucau |  Blog | Github | LinkedIn | Tomitriber
> 
> 
> 2016-04-03 13:04 GMT+02:00 Ihsan Ecemis :
>> 
>> That makes this even more interesting.  I can consistently reproduce this on:
>> 
>> Mac OS X 10.11.4,  Java 1.8.0_73-b02
>> Linux 4.1.10-17.31.amzn1.x86_64,  Java 1.8.0_71-b15
>> 
>> Thank you for trying to reproduce.  I would appreciate if you could give me 
>> any ideas/tips about how to proceed in terms of debugging/experimenting.
>> 
>> 
>>> On Apr 3, 2016, at 4:26 AM, Romain Manni-Bucau  
>>> wrote:
>>> 
>>> Hi
>>> 
>>> Didn't reproduce it with your sample. Ran on java8, not sure it is linked
>>> 
>>> Romain Manni-Bucau
>>> @rmannibucau |  Blog | Github | LinkedIn | Tomitriber
>>> 
>>> 
>>> 2016-04-02 13:27 GMT+02:00 Ihsan Ecemis :
 
 Hello,
 
 I was having some weird problems with my JSF pages lately, and after some 
 brute-force experimentation, I realized that TomEE has problems with 
 serving JSF resources over SSL if I start an embedded ActiveMQ broker.  I 
 found this awkward as I wouldn’t expect these 2 things to interfere with 
 each other.
 
 I created a barebones project to show the problem:  
 https://github.com/ecemis/activemq-jsf-troubleshoot 
 
 
 The project has 2 scripts at top level:
 
 run_server.sh:  That’s how I start the server
 test_command.sh:  That’s the test script that fetches a JSF resource via 
 curl. I ran this on a separate terminal
 
 And there are 3 tags you can checkout to test 3 cases:
 
 case-1-working:  This case doesn’t have ActiveMQ and test_command.sh can 
 successfully fetch a JSF resource
 case-2-not-working:  This case adds resources.xml file that starts an 
 embedded ActiveMQ broker. As a result, test_command.sh fails to fetch the 
 JSF resource
 case-3-working:  This one changes the service from https to http, and  
 test_command.sh works again (while ActiveMQ broker is running)
 
 (You can checkout and try each case separately, e.g. git checkout 
 case-2-not-working. You can also compare the changes between the cases via 
 git diff, e.g. git diff case-1-working case-2-not-working)
 
 Note that fetching static files works in all cases. The problem I could 
 observe so far is just with fetching JSF resources.
 
 All help will be greatly appreciated.
 
 Thanks,
 
 Ihsan.
 
 
 PS: For the impatient who wants to know the errors before cloning the 
 sample project above, here is what curl gives me:
 
 * SSLRead() return error -9845
 * Closing connection 0
 curl: (56) SSLRead() return error -9845
 
 
 And here is what I get on the server:
 
 WARNING - JSF1064: Unable to find or serve resource, primefaces.js, from 
 library, primefaces.
 WARNING -
 org.apache.catalina.connector.ClientAbortException: java.io.IOException: 
 Broken pipe
   at 
 org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393)
   at 
 org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426)
   at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:339)
   at 
 org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:418)
   at 
 org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:406)
   at 
 org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:97)
   at 
 java.nio.channels.Channels$WritableByteChannelImpl.write(Channels.java:458)
   at 
 com.sun.faces.application.resource.ResourceHandlerImpl.handleResourceRequest(ResourceHandlerImpl.java:358)
   at 
 javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153)
   at 
 

Re: Problem fetching JSF resources over SSL when ActiveMQ broker is running

[Romain Manni-Bucau]

tried on windows so wonder if it can be linked to gnu tools.

If you get such an issue the only thing I can think about is AMQ
changing the SSL config of the JVM which would then make the SSL
config of tomcat changing if relying on the JVM settings for any of
them.

Can also be a local firewall etc even if weird for a UNIx box - saw
several company windows boxes to fail even on localhost cause of
McAfee or equivalent.


Romain Manni-Bucau
@rmannibucau |  Blog | Github | LinkedIn | Tomitriber


2016-04-03 13:04 GMT+02:00 Ihsan Ecemis :
>
> That makes this even more interesting.  I can consistently reproduce this on:
>
> Mac OS X 10.11.4,  Java 1.8.0_73-b02
> Linux 4.1.10-17.31.amzn1.x86_64,  Java 1.8.0_71-b15
>
> Thank you for trying to reproduce.  I would appreciate if you could give me 
> any ideas/tips about how to proceed in terms of debugging/experimenting.
>
>
>> On Apr 3, 2016, at 4:26 AM, Romain Manni-Bucau  wrote:
>>
>> Hi
>>
>> Didn't reproduce it with your sample. Ran on java8, not sure it is linked
>>
>> Romain Manni-Bucau
>> @rmannibucau |  Blog | Github | LinkedIn | Tomitriber
>>
>>
>> 2016-04-02 13:27 GMT+02:00 Ihsan Ecemis :
>>>
>>> Hello,
>>>
>>> I was having some weird problems with my JSF pages lately, and after some 
>>> brute-force experimentation, I realized that TomEE has problems with 
>>> serving JSF resources over SSL if I start an embedded ActiveMQ broker.  I 
>>> found this awkward as I wouldn’t expect these 2 things to interfere with 
>>> each other.
>>>
>>> I created a barebones project to show the problem:  
>>> https://github.com/ecemis/activemq-jsf-troubleshoot 
>>> 
>>>
>>> The project has 2 scripts at top level:
>>>
>>> run_server.sh:  That’s how I start the server
>>> test_command.sh:  That’s the test script that fetches a JSF resource via 
>>> curl. I ran this on a separate terminal
>>>
>>> And there are 3 tags you can checkout to test 3 cases:
>>>
>>> case-1-working:  This case doesn’t have ActiveMQ and test_command.sh can 
>>> successfully fetch a JSF resource
>>> case-2-not-working:  This case adds resources.xml file that starts an 
>>> embedded ActiveMQ broker. As a result, test_command.sh fails to fetch the 
>>> JSF resource
>>> case-3-working:  This one changes the service from https to http, and  
>>> test_command.sh works again (while ActiveMQ broker is running)
>>>
>>> (You can checkout and try each case separately, e.g. git checkout 
>>> case-2-not-working. You can also compare the changes between the cases via 
>>> git diff, e.g. git diff case-1-working case-2-not-working)
>>>
>>> Note that fetching static files works in all cases. The problem I could 
>>> observe so far is just with fetching JSF resources.
>>>
>>> All help will be greatly appreciated.
>>>
>>> Thanks,
>>>
>>> Ihsan.
>>>
>>>
>>> PS: For the impatient who wants to know the errors before cloning the 
>>> sample project above, here is what curl gives me:
>>>
>>> * SSLRead() return error -9845
>>> * Closing connection 0
>>> curl: (56) SSLRead() return error -9845
>>>
>>>
>>> And here is what I get on the server:
>>>
>>> WARNING - JSF1064: Unable to find or serve resource, primefaces.js, from 
>>> library, primefaces.
>>> WARNING -
>>> org.apache.catalina.connector.ClientAbortException: java.io.IOException: 
>>> Broken pipe
>>>at 
>>> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393)
>>>at 
>>> org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426)
>>>at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:339)
>>>at 
>>> org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:418)
>>>at 
>>> org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:406)
>>>at 
>>> org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:97)
>>>at 
>>> java.nio.channels.Channels$WritableByteChannelImpl.write(Channels.java:458)
>>>at 
>>> com.sun.faces.application.resource.ResourceHandlerImpl.handleResourceRequest(ResourceHandlerImpl.java:358)
>>>at 
>>> javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153)
>>>at 
>>> org.primefaces.application.resource.PrimeResourceHandler.handleResourceRequest(PrimeResourceHandler.java:87)
>>>at 
>>> javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153)
>>>at javax.faces.webapp.FacesServlet.service(FacesServlet.java:655)
>>>at 
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
>>>at 
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
>>>at 
>>> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
>>>at 
>>> 

Re: Problem fetching JSF resources over SSL when ActiveMQ broker is running

[Romain Manni-Bucau]

Hi

Didn't reproduce it with your sample. Ran on java8, not sure it is linked

Romain Manni-Bucau
@rmannibucau |  Blog | Github | LinkedIn | Tomitriber


2016-04-02 13:27 GMT+02:00 Ihsan Ecemis :
>
> Hello,
>
> I was having some weird problems with my JSF pages lately, and after some 
> brute-force experimentation, I realized that TomEE has problems with serving 
> JSF resources over SSL if I start an embedded ActiveMQ broker.  I found this 
> awkward as I wouldn’t expect these 2 things to interfere with each other.
>
> I created a barebones project to show the problem:  
> https://github.com/ecemis/activemq-jsf-troubleshoot 
> 
>
> The project has 2 scripts at top level:
>
> run_server.sh:  That’s how I start the server
> test_command.sh:  That’s the test script that fetches a JSF resource via 
> curl. I ran this on a separate terminal
>
> And there are 3 tags you can checkout to test 3 cases:
>
> case-1-working:  This case doesn’t have ActiveMQ and test_command.sh can 
> successfully fetch a JSF resource
> case-2-not-working:  This case adds resources.xml file that starts an 
> embedded ActiveMQ broker. As a result, test_command.sh fails to fetch the JSF 
> resource
> case-3-working:  This one changes the service from https to http, and  
> test_command.sh works again (while ActiveMQ broker is running)
>
> (You can checkout and try each case separately, e.g. git checkout 
> case-2-not-working. You can also compare the changes between the cases via 
> git diff, e.g. git diff case-1-working case-2-not-working)
>
> Note that fetching static files works in all cases. The problem I could 
> observe so far is just with fetching JSF resources.
>
> All help will be greatly appreciated.
>
> Thanks,
>
> Ihsan.
>
>
> PS: For the impatient who wants to know the errors before cloning the sample 
> project above, here is what curl gives me:
>
> * SSLRead() return error -9845
> * Closing connection 0
> curl: (56) SSLRead() return error -9845
>
>
> And here is what I get on the server:
>
> WARNING - JSF1064: Unable to find or serve resource, primefaces.js, from 
> library, primefaces.
> WARNING -
> org.apache.catalina.connector.ClientAbortException: java.io.IOException: 
> Broken pipe
> at 
> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393)
> at 
> org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426)
> at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:339)
> at 
> org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:418)
> at 
> org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:406)
> at 
> org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:97)
> at 
> java.nio.channels.Channels$WritableByteChannelImpl.write(Channels.java:458)
> at 
> com.sun.faces.application.resource.ResourceHandlerImpl.handleResourceRequest(ResourceHandlerImpl.java:358)
> at 
> javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153)
> at 
> org.primefaces.application.resource.PrimeResourceHandler.handleResourceRequest(PrimeResourceHandler.java:87)
> at 
> javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153)
> at javax.faces.webapp.FacesServlet.service(FacesServlet.java:655)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
> at 
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
> at 
> org.apache.openejb.server.httpd.WebBeansFilter.doFilter(WebBeansFilter.java:52)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
> at org.apache.tomee.catalina.OpenEJBValve.invoke(OpenEJBValve.java:44)
> at 
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:614)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at 
>