Re: [Dev] Issue on WSRegistry Service Client on content reading concurrently on a single connection

2014-06-13 Thread Senaka Fernando 
Guys,

We have to do a thorough perf test with the WS-API. There can be design
issues that affect performance to a great extent.

Thanks,
Senaka.


On Fri, Jun 13, 2014 at 6:42 AM, Subash Chaturanga sub...@wso2.com wrote:

 Hi,
 Here I am attaching the patch which does the check of the content size.
 And if it is 1MB then we load it to memory and pass a ByteArray data
 source for the data handler. And if 1MB we proceed as previously.

 @Danushka,
 Please use the following patch and verify your scenario.



 On Tue, Jun 10, 2014 at 8:34 PM, Ajith Vitharana aji...@wso2.com wrote:

 Please find the blog post [1].

 [1]
 http://ajithvblogs.blogspot.de/2013/11/jmeter-java-request-to-test.html

 Thanks.
 Ajith


 On Tue, Jun 10, 2014 at 8:28 PM, Subash Chaturanga sub...@wso2.com
 wrote:

 Hi ,
 This is regarding  issue on WS Registry Service Client in high
 concurrency which seems blocks the AF release.

 When reading the content with several threads in AF, i.e two resources

 path = /foo/r1 content = 12345
 path = /foo/r2 content =  67

 For AF guys, when they retrieve r1 and r2 resources, i.e for
 r1 - returns  12345
 but r2 returns 67*45*

 ( r2 returns with some values from old resource content. )

 When dig deep with Janaka, found that, in server side
  org.wso2.carbon.registry.ws.api.WSRegistry maintains single tempFile per
 instance. And it get used to maintain the content to pass to client side
 inside the DataHolder.

 Hence obviously we can expect this issue in a but high concurrency. So
 as a solution when  creating the data holder we create a copy of the
 aforementioned file as follows. And then for multiple requests on the same
 ws connection it deals with separate files and issue went away.

 But this leads to grow the tmp files in the tmp directory but upon
 server downtime those will get vanished.

 With the following fix, since files are growing, one option is to do an
 additional ws call inside ws-client to server side to clean up the
 particular file. But it seems not a best solution.


 *So @Senaka @Ajith, Any better solution you can see ? *


  Index:
 src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
 ===
 --- src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
(revision 204294)
 +++ src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
(working copy)
 @@ -193,6 +193,9 @@
 }

 public static DataHandler makeDataHandler(Resource resource, File
 tempFile) throws IOException, RegistryException{
 +tempFile = File.createTempFile(wsresource, .tmp);
 +tempFile.deleteOnExit();
 +
 if (resource.getContent() == null) {
 return null;
 }


 --
 Thanks
 /subash

 *Subash Chaturanga*
 Senior Software Engineer :Platform TG; WSO2 Inc. http://wso2.com

 email: sub...@wso2.com
 blog:  http://subashsdm.blogspot.com/
 twitter: @subash89
 phone: +9477 2225922
 Lean . Enterprise . Middleware




 --
 Ajith Vitharana.
 WSO2 Inc. - http://wso2.org
 Email  :  aji...@wso2.com
 Mobile : +94772217350




 --
 Thanks
 /subash

 *Subash Chaturanga*
 Senior Software Engineer  Lead WSO2 Governance Registry
  Platform TG; WSO2 Inc. http://wso2.com
 Contact:
 email: sub...@wso2.com
 blog:  http://subashsdm.blogspot.com/
 twitter: @subash89
 phone: +9477 2225922
 Lean . Enterprise . Middleware




-- 


*[image: http://wso2.com] http://wso2.com Senaka Fernando*
Software Architect; WSO2 Inc.; http://wso2.com



* Member; Apache Software Foundation; http://apache.org
http://apache.orgE-mail: senaka AT wso2.com http://wso2.com**P: +1 408
754 7388; ext: 51736*;


*M: +44 782 741 1966 Linked-In: http://linkedin.com/in/senakafernando
http://linkedin.com/in/senakafernando*Lean . Enterprise . Middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] Issue on WSRegistry Service Client on content reading concurrently on a single connection

2014-06-13 Thread Danushka Fernando 
Hi Subash
I tested this with our scenario and it seems to work fine.

Thanks  Regards
Danushka Fernando
Software Engineer
WSO2 inc. http://wso2.com/
Mobile : +94716332729


On Fri, Jun 13, 2014 at 12:26 PM, Senaka Fernando sen...@wso2.com wrote:

 Guys,

 We have to do a thorough perf test with the WS-API. There can be design
 issues that affect performance to a great extent.

 Thanks,
 Senaka.


 On Fri, Jun 13, 2014 at 6:42 AM, Subash Chaturanga sub...@wso2.com
 wrote:

 Hi,
 Here I am attaching the patch which does the check of the content size.
 And if it is 1MB then we load it to memory and pass a ByteArray data
 source for the data handler. And if 1MB we proceed as previously.

 @Danushka,
 Please use the following patch and verify your scenario.



 On Tue, Jun 10, 2014 at 8:34 PM, Ajith Vitharana aji...@wso2.com wrote:

 Please find the blog post [1].

 [1]
 http://ajithvblogs.blogspot.de/2013/11/jmeter-java-request-to-test.html

 Thanks.
 Ajith


 On Tue, Jun 10, 2014 at 8:28 PM, Subash Chaturanga sub...@wso2.com
 wrote:

 Hi ,
 This is regarding  issue on WS Registry Service Client in high
 concurrency which seems blocks the AF release.

 When reading the content with several threads in AF, i.e two resources

 path = /foo/r1 content = 12345
 path = /foo/r2 content =  67

 For AF guys, when they retrieve r1 and r2 resources, i.e for
 r1 - returns  12345
 but r2 returns 67*45*

 ( r2 returns with some values from old resource content. )

 When dig deep with Janaka, found that, in server side
  org.wso2.carbon.registry.ws.api.WSRegistry maintains single tempFile per
 instance. And it get used to maintain the content to pass to client side
 inside the DataHolder.

 Hence obviously we can expect this issue in a but high concurrency. So
 as a solution when  creating the data holder we create a copy of the
 aforementioned file as follows. And then for multiple requests on the same
 ws connection it deals with separate files and issue went away.

 But this leads to grow the tmp files in the tmp directory but upon
 server downtime those will get vanished.

 With the following fix, since files are growing, one option is to do an
 additional ws call inside ws-client to server side to clean up the
 particular file. But it seems not a best solution.


 *So @Senaka @Ajith, Any better solution you can see ? *


  Index:
 src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
 ===
 --- src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
(revision 204294)
 +++ src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
(working copy)
 @@ -193,6 +193,9 @@
 }

 public static DataHandler makeDataHandler(Resource resource, File
 tempFile) throws IOException, RegistryException{
 +tempFile = File.createTempFile(wsresource, .tmp);
 +tempFile.deleteOnExit();
 +
 if (resource.getContent() == null) {
 return null;
 }


 --
 Thanks
 /subash

 *Subash Chaturanga*
 Senior Software Engineer :Platform TG; WSO2 Inc. http://wso2.com

 email: sub...@wso2.com
 blog:  http://subashsdm.blogspot.com/
 twitter: @subash89
 phone: +9477 2225922
 Lean . Enterprise . Middleware




 --
 Ajith Vitharana.
 WSO2 Inc. - http://wso2.org
 Email  :  aji...@wso2.com
 Mobile : +94772217350




 --
 Thanks
 /subash

 *Subash Chaturanga*
 Senior Software Engineer  Lead WSO2 Governance Registry
  Platform TG; WSO2 Inc. http://wso2.com
 Contact:
 email: sub...@wso2.com
 blog:  http://subashsdm.blogspot.com/
 twitter: @subash89
 phone: +9477 2225922
 Lean . Enterprise . Middleware




 --


 *[image: http://wso2.com] http://wso2.com Senaka Fernando*
 Software Architect; WSO2 Inc.; http://wso2.com



 * Member; Apache Software Foundation; http://apache.org
 http://apache.orgE-mail: senaka AT wso2.com http://wso2.com**P: +1
 408 754 7388 %2B1%20408%20754%207388; ext: 51736*;


 *M: +44 782 741 1966 %2B44%20782%20741%201966 Linked-In:
 http://linkedin.com/in/senakafernando
 http://linkedin.com/in/senakafernando*
 Lean . Enterprise . Middleware

___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] Issue on WSRegistry Service Client on content reading concurrently on a single connection

2014-06-12 Thread Subash Chaturanga 
Hi,
Here I am attaching the patch which does the check of the content size. And
if it is 1MB then we load it to memory and pass a ByteArray data source
for the data handler. And if 1MB we proceed as previously.

@Danushka,
Please use the following patch and verify your scenario.



On Tue, Jun 10, 2014 at 8:34 PM, Ajith Vitharana aji...@wso2.com wrote:

 Please find the blog post [1].

 [1]http://ajithvblogs.blogspot.de/2013/11/jmeter-java-request-to-test.html

 Thanks.
 Ajith


 On Tue, Jun 10, 2014 at 8:28 PM, Subash Chaturanga sub...@wso2.com
 wrote:

 Hi ,
 This is regarding  issue on WS Registry Service Client in high
 concurrency which seems blocks the AF release.

 When reading the content with several threads in AF, i.e two resources

 path = /foo/r1 content = 12345
 path = /foo/r2 content =  67

 For AF guys, when they retrieve r1 and r2 resources, i.e for
 r1 - returns  12345
 but r2 returns 67*45*

 ( r2 returns with some values from old resource content. )

 When dig deep with Janaka, found that, in server side
  org.wso2.carbon.registry.ws.api.WSRegistry maintains single tempFile per
 instance. And it get used to maintain the content to pass to client side
 inside the DataHolder.

 Hence obviously we can expect this issue in a but high concurrency. So as
 a solution when  creating the data holder we create a copy of the
 aforementioned file as follows. And then for multiple requests on the same
 ws connection it deals with separate files and issue went away.

 But this leads to grow the tmp files in the tmp directory but upon server
 downtime those will get vanished.

 With the following fix, since files are growing, one option is to do an
 additional ws call inside ws-client to server side to clean up the
 particular file. But it seems not a best solution.


 *So @Senaka @Ajith, Any better solution you can see ? *


  Index:
 src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
 ===
 --- src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
  (revision 204294)
 +++ src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
  (working copy)
 @@ -193,6 +193,9 @@
 }

 public static DataHandler makeDataHandler(Resource resource, File
 tempFile) throws IOException, RegistryException{
 +tempFile = File.createTempFile(wsresource, .tmp);
 +tempFile.deleteOnExit();
 +
 if (resource.getContent() == null) {
 return null;
 }


 --
 Thanks
 /subash

 *Subash Chaturanga*
 Senior Software Engineer :Platform TG; WSO2 Inc. http://wso2.com

 email: sub...@wso2.com
 blog:  http://subashsdm.blogspot.com/
 twitter: @subash89
 phone: +9477 2225922
 Lean . Enterprise . Middleware




 --
 Ajith Vitharana.
 WSO2 Inc. - http://wso2.org
 Email  :  aji...@wso2.com
 Mobile : +94772217350




-- 
Thanks
/subash

*Subash Chaturanga*
Senior Software Engineer  Lead WSO2 Governance Registry
Platform TG; WSO2 Inc. http://wso2.com
Contact:
email: sub...@wso2.com
blog:  http://subashsdm.blogspot.com/
twitter: @subash89
phone: +9477 2225922
Lean . Enterprise . Middleware
Index: src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
===
--- src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java	(revision 191396)
+++ src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java	(working copy)
@@ -46,9 +46,11 @@
 
 import javax.activation.DataHandler;
 import javax.activation.FileDataSource;
+import javax.mail.util.ByteArrayDataSource;
 import javax.servlet.http.HttpServletRequest;
 
 import org.apache.axis2.context.MessageContext;
+import org.apache.commons.io.IOUtils;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 import org.wso2.carbon.registry.core.*;
@@ -192,32 +194,39 @@
 return output.toByteArray();
 }
 
-public static DataHandler makeDataHandler(Resource resource, File tempFile) throws IOException, RegistryException{
+public static DataHandler makeDataHandler(Resource resource, File tempFile) throws IOException, RegistryException {
 if (resource.getContent() == null) {
 return null;
 }
 InputStream is = null;
 OutputStream os = null;
+
+
 try {
-os = new FileOutputStream(tempFile);
 if (resource.getContent() instanceof String[]) {
 String[] strArray = (String[]) resource.getContent();
 
 ObjectOutputStream oos = new ObjectOutputStream(os);
 oos.writeObject(strArray);
 } else {
-try {
-is = resource.getContentStream();
-//os = new FileOutputStream(tempFile);
+is = resource.getContentStream();
+if (isContentLarge(resource.getContentStream())) {
+try {
+

[Dev] Issue on WSRegistry Service Client on content reading concurrently on a single connection

2014-06-10 Thread Subash Chaturanga 
Hi ,
This is regarding  issue on WS Registry Service Client in high concurrency
which seems blocks the AF release.

When reading the content with several threads in AF, i.e two resources

path = /foo/r1 content = 12345
path = /foo/r2 content =  67

For AF guys, when they retrieve r1 and r2 resources, i.e for
r1 - returns  12345
but r2 returns 67*45*

( r2 returns with some values from old resource content. )

When dig deep with Janaka, found that, in server side
 org.wso2.carbon.registry.ws.api.WSRegistry maintains single tempFile per
instance. And it get used to maintain the content to pass to client side
inside the DataHolder.

Hence obviously we can expect this issue in a but high concurrency. So as a
solution when  creating the data holder we create a copy of the
aforementioned file as follows. And then for multiple requests on the same
ws connection it deals with separate files and issue went away.

But this leads to grow the tmp files in the tmp directory but upon server
downtime those will get vanished.

With the following fix, since files are growing, one option is to do an
additional ws call inside ws-client to server side to clean up the
particular file. But it seems not a best solution.


*So @Senaka @Ajith, Any better solution you can see ? *


 Index: src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
===
--- src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
   (revision 204294)
+++ src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
   (working copy)
@@ -193,6 +193,9 @@
}

public static DataHandler makeDataHandler(Resource resource, File
tempFile) throws IOException, RegistryException{
+tempFile = File.createTempFile(wsresource, .tmp);
+tempFile.deleteOnExit();
+
if (resource.getContent() == null) {
return null;
}


-- 
Thanks
/subash

*Subash Chaturanga*
Senior Software Engineer :Platform TG; WSO2 Inc. http://wso2.com

email: sub...@wso2.com
blog:  http://subashsdm.blogspot.com/
twitter: @subash89
phone: +9477 2225922
Lean . Enterprise . Middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] Issue on WSRegistry Service Client on content reading concurrently on a single connection

2014-06-10 Thread Ajith Vitharana 
Please find the blog post [1].

[1]http://ajithvblogs.blogspot.de/2013/11/jmeter-java-request-to-test.html

Thanks.
Ajith


On Tue, Jun 10, 2014 at 8:28 PM, Subash Chaturanga sub...@wso2.com wrote:

 Hi ,
 This is regarding  issue on WS Registry Service Client in high concurrency
 which seems blocks the AF release.

 When reading the content with several threads in AF, i.e two resources

 path = /foo/r1 content = 12345
 path = /foo/r2 content =  67

 For AF guys, when they retrieve r1 and r2 resources, i.e for
 r1 - returns  12345
 but r2 returns 67*45*

 ( r2 returns with some values from old resource content. )

 When dig deep with Janaka, found that, in server side
  org.wso2.carbon.registry.ws.api.WSRegistry maintains single tempFile per
 instance. And it get used to maintain the content to pass to client side
 inside the DataHolder.

 Hence obviously we can expect this issue in a but high concurrency. So as
 a solution when  creating the data holder we create a copy of the
 aforementioned file as follows. And then for multiple requests on the same
 ws connection it deals with separate files and issue went away.

 But this leads to grow the tmp files in the tmp directory but upon server
 downtime those will get vanished.

 With the following fix, since files are growing, one option is to do an
 additional ws call inside ws-client to server side to clean up the
 particular file. But it seems not a best solution.


 *So @Senaka @Ajith, Any better solution you can see ? *


  Index:
 src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
 ===
 --- src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
  (revision 204294)
 +++ src/main/java/org/wso2/carbon/registry/ws/api/utils/CommonUtil.java
  (working copy)
 @@ -193,6 +193,9 @@
 }

 public static DataHandler makeDataHandler(Resource resource, File
 tempFile) throws IOException, RegistryException{
 +tempFile = File.createTempFile(wsresource, .tmp);
 +tempFile.deleteOnExit();
 +
 if (resource.getContent() == null) {
 return null;
 }


 --
 Thanks
 /subash

 *Subash Chaturanga*
 Senior Software Engineer :Platform TG; WSO2 Inc. http://wso2.com

 email: sub...@wso2.com
 blog:  http://subashsdm.blogspot.com/
 twitter: @subash89
 phone: +9477 2225922
 Lean . Enterprise . Middleware




-- 
Ajith Vitharana.
WSO2 Inc. - http://wso2.org
Email  :  aji...@wso2.com
Mobile : +94772217350
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev