Hi Sagara, Thanks for your feedback, Yes i have implemented 2nd approach as same as you have suggested. It only process XMLSchema at the first message. Here i have used intermediate representation to keep the xmlSchema structure therefore i do not need to process XMLSchema for every request but only very first request. I store this intermediate representation in configuration context with method Qname as a property ,yes sound is not good there may be a memory problem with large XMLSchema, and will use it to process next requests.
Thanks, Shameera. On Sat, Aug 11, 2012 at 7:38 PM, Sagara Gunathunga < [email protected]> wrote: > It seems your 1st approach provides much better performance than 2nd > approach, can you incorporate improvements that I mentioned here[1] ? > I believe applying those modification will solve performance issue > with 2nd approach. > > [1] - http://markmail.org/message/xyfxzn2clc3r5zfg > > Thanks ! > > On Sat, Aug 11, 2012 at 4:07 AM, Shameera Rathnayaka > <[email protected]> wrote: > > Hi devs, > > > > I did performance test with 1. SOAP support , 2 JSON Badgerfish , 3 JSON > > first approach(pure json) and 4. JSON second > > approach(XMLStreamReader/XMLStreamWriter implementation). Here i have > > summarize the results i got. > > I used same POJO service and send the exact same request in relevent > > convention. > > > > Environment: > > OS : Ubuntu 11.04 > > JDK 1.6_30 > > memory 4 GB > > > > SOAP JSON badgerfish > > JSON 1st approach JSON 2nd approach > > > > Concurrency Level: 50 50 > > 50 50 > > Time taken for tests: 243.13 sec 287.18 sec > > 118.99 200.0 > > Complete requests: 500000 500000 > > 500000 500000 > > Total transferred: 2.2 GB 34.42 GB > > 0.61 GB 0.63 GB > > Requests per second: 2056.53 #/sec 1741.04 #/sec > 4201.69 > > #/sec 2499.91 #/sec > > Time per request: 24.3 ms 28.7 ms > > 11.90 ms 20.00 ms > > Time per request:(mean) 0.486 ms 0.544 ms > 0.238 > > ms 0.4 ms > > > > Thanks, > > Shameera. > > > > > > On Wed, Aug 8, 2012 at 12:17 PM, Shameera Rathnayaka > > <[email protected]> wrote: > >> > >> HI devs, > >> > >> I finished the implementation and did a performance test with existing > >> SOAP and implemented JSON. Here is the summary of the test results. > >> For this i used same service and send the same request in both SOAP and > >> JSON ways, size of JSON message is lower than SOAP message but requests > are > >> exactly same. > >> > >> Environment : > >> OS - Ubuntu 11.10 > >> Axis2 1.7(current trunk) > >> Tool - Java benchmark > >> > >> num of requests : 50000 > >> concurrency level: 10 (Total num of requests are 50000*10 = 500000 ) > >> Time taken for test : with SOAP 178 sec and with JSON 140 sec > >> Total transferred : with SOAP 2371000000 bytes and with JSON 674000000 > >> bytes > >> Requests per second: with SOAP 2,801.76 [#/sec] (mean) and with JSON > >> 3,568.89 [#/sec] (mean) > >> > >> will do a performance test with concurrency level 50 and num of requests > >> 500000 and give the summary of results. > >> > >> Thanks, > >> Shameera. > >> > >> > >> > >> > >> On Tue, Jul 31, 2012 at 10:50 AM, Shameera Rathnayaka > >> <[email protected]> wrote: > >>> > >>> HI devs, > >>> > >>> I have implemented XMLStreamWriter as well as XMLStreamReader to > convert > >>> XML <----> JSON with XMLSchema, > >>> Basically it works fine, but still need few test to be done after that > I > >>> am willing to write performance test with implemented JSON support with > >>> existing SOAP support. > >>> > >>> Thanks, > >>> Shameera. > >>> > >>> > >>> > >>> On Mon, Jul 16, 2012 at 10:43 AM, Amila Suriarachchi > >>> <[email protected]> wrote: > >>>> > >>>> > >>>> > >>>> On Sun, Jul 15, 2012 at 10:53 PM, Shameera Rathnayaka > >>>> <[email protected]> wrote: > >>>>> > >>>>> Hi devs, > >>>>> > >>>>> I implemented this feature completely to convert XML to JSON with the > >>>>> help of XmlSchema of that XML document. I tested this with different > >>>>> Schemas, it is working fine. I implemented this as a separate > project, so i > >>>>> will integrate this with axis2 trunk and try with actual scenario. > >>>> > >>>> > >>>> Can you use this technique at the reader level as well to support > >>>> namespaces? > >>>> > >>>> Then do a performance comparison using a POJO service with the exiting > >>>> json support and based the new gson based XMLstream API. According to > the > >>>> result we need to optimise the later to have better performance. > >>>> > >>>> thanks, > >>>> Amila. > >>>> > >>>>> > >>>>> > >>>>> Yes, This may be a standard way to convert XML to JSON conversion as > >>>>> other implementations don't convert JSON message correctly when it > has one > >>>>> value array, unless implementation supports special XML structure > where > >>>>> Json-lib does. > >>>>> > >>>>> Definitely we can improve the performance of this implementation. May > >>>>> be in the way that Sagara had suggested in his reply or in another > way. > >>>>> > >>>>> Thanks, > >>>>> Shameera. > >>>>> > >>>>> On Sun, Jul 15, 2012 at 6:56 PM, Amila Suriarachchi > >>>>> <[email protected]> wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Sun, Jul 15, 2012 at 2:19 PM, Sagara Gunathunga > >>>>>> <[email protected]> wrote: > >>>>>>> > >>>>>>> On Sun, Jul 15, 2012 at 1:27 PM, Amila Suriarachchi > >>>>>>> <[email protected]> wrote: > >>>>>>> > > >>>>>>> > > >>>>>>> > On Sat, Jul 14, 2012 at 1:14 PM, Sagara Gunathunga > >>>>>>> > <[email protected]> wrote: > >>>>>>> >> > >>>>>>> >> On Sat, Jul 14, 2012 at 12:42 PM, Shameera Rathnayaka > >>>>>>> >> <[email protected]> wrote: > >>>>>>> >> > Hi devs, > >>>>>>> >> > > >>>>>>> >> > As Amila(Project mentor) suggested in previous thread[0] , I > >>>>>>> >> > have > >>>>>>> >> > implemented > >>>>>>> >> > XMLStreamWriter API to use schema of the processing xml and > gson > >>>>>>> >> > together > >>>>>>> >> > to convert XML ---> JSON. I have made my progress up to > support > >>>>>>> >> > xml > >>>>>>> >> > which > >>>>>>> >> > don't have complex type elements with maxOccur >1, but may > have > >>>>>>> >> > simple > >>>>>>> >> > type elements with maxOccur >=1 . That means In the JSON point > >>>>>>> >> > of view, > >>>>>>> >> > my > >>>>>>> >> > present implementation supports XML --> JSON where there is > not > >>>>>>> >> > any JSON > >>>>>>> >> > Arrays which has JSON objects in the converted JSON String. > >>>>>>> >> > > >>>>>>> >> > Here is a summary of the implementation. > >>>>>>> >> > > >>>>>>> >> > GsonXmlStreamWriter which implements XMLStreamWriter has a > >>>>>>> >> > constructor > >>>>>>> >> > to get JsonWriter and XmlSchema object. Then it process > >>>>>>> >> > XmlSchema and > >>>>>>> >> > >>>>>>> >> Are you plan to process XmlSchema object with every response ? > If > >>>>>>> >> so > >>>>>>> >> processing XmlSchema object will add some overhead and result > >>>>>>> >> into > >>>>>>> >> performance. Axis2 already has JSON support main objective of > >>>>>>> >> this > >>>>>>> >> project is to provide a high performance JSON implementation, > >>>>>>> >> processing XmlSchema object with every response does not help > to > >>>>>>> >> meet > >>>>>>> >> your expectations. Instead I suggest to process XmlSchema > object > >>>>>>> >> only > >>>>>>> >> one-time during the deployment after the AxisService creation > and > >>>>>>> >> store result of XmlSchema processing with AxisService object > as > >>>>>>> >> a > >>>>>>> >> light-weight object ( Let's say a map) and use this light > weight > >>>>>>> >> map > >>>>>>> >> with response. In this approach overhead if relatively low. > >>>>>>> > > >>>>>>> > > >>>>>>> > This is a POC level approach to convert xml stream to json using > >>>>>>> > xml schema. > >>>>>>> > For an example there are some techniques like > >>>>>>> > bagger fish and others to address this problem. But those things > >>>>>>> > complicated > >>>>>>> > the produced json stream. > >>>>>>> > > >>>>>>> > Once it proves at the POC level we can think of improving > >>>>>>> > performance. > >>>>>>> > > >>>>>>> > If you look at the current axis2 json support does that properly > >>>>>>> > address the > >>>>>>> > problem shameera mentioned earlier? i.e serialising the > >>>>>>> > array elements when there is on element. And also can that be > used > >>>>>>> > with some > >>>>>>> > thing like ADB or WSO2 Data Services,rules etc.. > >>>>>>> > Idea here to develop a technique to address those problems > properly > >>>>>>> > while > >>>>>>> > converting an xml stream to json stream. > >>>>>>> > >>>>>>> I don't have any issue with the technique proposed here (i.e. > >>>>>>> processing XMLSchema to identify structure ). My only concern is > why > >>>>>>> we should process XMLSchema with every response because in each and > >>>>>>> every response we end up getting same result, why shouldn't we > >>>>>>> process > >>>>>>> XMLSchema only one time and reuse results ? May be as last step of > >>>>>>> deployment or with very first response processing ? > >>>>>>> > >>>>>>> Of course we can keep it as future improvement for the moment but > >>>>>>> Axis2 having good history of forgetting those improvement after > >>>>>>> initial development every time when I dig into code I found couple > of > >>>>>>> such TODOs. > >>>>>> > >>>>>> > >>>>>> :) I am not telling to do it after GSoc project. But first we need > to > >>>>>> have a proper POC. > >>>>>> > >>>>>> thanks, > >>>>>> Amila. > >>>>>>> > >>>>>>> > >>>>>>> Thanks ! > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > > >>>>>>> > thanks, > >>>>>>> > Amila. > >>>>>>> > > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> Thanks ! > >>>>>>> >> > >>>>>>> >> > use a queue to keep the schema structure. At the moment i have > >>>>>>> >> > only > >>>>>>> >> > implemented > >>>>>>> >> > writeStartElement(String localName) , writeCharaters(String > >>>>>>> >> > text) , > >>>>>>> >> > writeEndElement() > >>>>>>> >> > and writeEndDocument() functions. I am processing the queue > >>>>>>> >> > which has > >>>>>>> >> > the schema structure and put it to another stack to identify > the > >>>>>>> >> > scope > >>>>>>> >> > of > >>>>>>> >> > end elements. > >>>>>>> >> > > >>>>>>> >> > I am now working on, improving this to support xml which has > >>>>>>> >> > complex > >>>>>>> >> > type > >>>>>>> >> > elements > >>>>>>> >> > with maxOccour > 1 . I'll keep updating my progress mean > while. > >>>>>>> >> > > >>>>>>> >> > Thanks, > >>>>>>> >> > Shameera. > >>>>>>> >> > > >>>>>>> >> > -- > >>>>>>> >> > Shameera Rathnayaka > >>>>>>> >> > Undergraduate > >>>>>>> >> > Department of Computer Science and Engineering > >>>>>>> >> > University of Moratuwa. > >>>>>>> >> > Sri Lanka. > >>>>>>> >> > > >>>>>>> >> > Blog : http://shameerarathnayaka.blogspot.com/ > >>>>>>> >> > > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> -- > >>>>>>> >> Sagara Gunathunga > >>>>>>> >> > >>>>>>> >> Blog - http://ssagara.blogspot.com > >>>>>>> >> Web - http://people.apache.org/~sagara/ > >>>>>>> >> LinkedIn - http://www.linkedin.com/in/ssagara > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > --------------------------------------------------------------------- > >>>>>>> >> To unsubscribe, e-mail: [email protected] > >>>>>>> >> For additional commands, e-mail: [email protected] > >>>>>>> >> > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > -- > >>>>>>> > Amila Suriarachchi > >>>>>>> > WSO2 Inc. > >>>>>>> > blog: http://amilachinthaka.blogspot.com/ > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Sagara Gunathunga > >>>>>>> > >>>>>>> Blog - http://ssagara.blogspot.com > >>>>>>> Web - http://people.apache.org/~sagara/ > >>>>>>> LinkedIn - http://www.linkedin.com/in/ssagara > >>>>>>> > >>>>>>> > --------------------------------------------------------------------- > >>>>>>> To unsubscribe, e-mail: [email protected] > >>>>>>> For additional commands, e-mail: [email protected] > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Amila Suriarachchi > >>>>>> WSO2 Inc. > >>>>>> blog: http://amilachinthaka.blogspot.com/ > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Shameera Rathnayaka > >>>>> Undergraduate > >>>>> Department of Computer Science and Engineering > >>>>> University of Moratuwa. > >>>>> Sri Lanka. > >>>>> > >>>>> Blog : http://shameerarathnayaka.blogspot.com/ > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Amila Suriarachchi > >>>> WSO2 Inc. > >>>> blog: http://amilachinthaka.blogspot.com/ > >>> > >>> > >>> > >>> > >>> -- > >>> Shameera Rathnayaka > >>> Undergraduate > >>> Department of Computer Science and Engineering > >>> University of Moratuwa. > >>> Sri Lanka. > >>> > >>> Blog : http://shameerarathnayaka.blogspot.com/ > >>> > >> > >> > >> > >> -- > >> Shameera Rathnayaka > >> Undergraduate > >> Department of Computer Science and Engineering > >> University of Moratuwa. > >> Sri Lanka. > >> > >> Blog : http://shameerarathnayaka.blogspot.com/ > >> > > > > > > > > -- > > Shameera Rathnayaka > > Undergraduate > > Department of Computer Science and Engineering > > University of Moratuwa. > > Sri Lanka. > > > > Blog : http://shameerarathnayaka.blogspot.com/ > > > > > > -- > Sagara Gunathunga > > Blog - http://ssagara.blogspot.com > Web - http://people.apache.org/~sagara/ > LinkedIn - http://www.linkedin.com/in/ssagara > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- Shameera Rathnayaka Undergraduate Department of Computer Science and Engineering University of Moratuwa. Sri Lanka. Blog : http://shameerarathnayaka.blogspot.com/
