Hi Preston and Michael, Thanks for enlightening me. Idea of retrieving results from hyracks only when the user requests did not come to my mind. I have changed my implementation accordingly [1]. I will submit a PR today as well.
[1] https://github.com/erandiganepola/vxquery/blob/master/vxquery-rest/src/main/java/org/apache/vxquery/rest/core/VXQuery.java Thanks and regards, *Erandi Ganepola* Undergraduate BSc. Management and Information Technology (IT Sp.) University of Kelaniya, Sri Lanka <https://www.linkedin.com/in/erandiganepola/> <https://github.com/erandiganepola> <https://twitter.com/erandiganepola> On Tue, Jul 11, 2017 at 11:26 PM, Michael Carey <[email protected]> wrote: > Erandi, > > AsterixDB actually spools its results to local result partitions and then > delivers them out the top - it does this so that resources like memory > can't be held hostage by the system's clients. @Till, can you add a few > sentences about this? > > Cheers, > > MIke > > > On 7/10/17 9:48 PM, Erandi Ganepola wrote: > >> Hi all, >> >> As per the REST API definition [1], it operates in two steps. >> >> 1. Users send the statement to be queried and REST API responds with a >> restult Id. >> 2. Later users ask for the results for the result id (received at earlier >> step) and server returns the results. >> >> Because of this two step process, we need to keep the results stored until >> the user come and ask for it. I'm currently storing these results in >> memory >> [2] which consumes a lot of memory when results for multiple requests gets >> accumulated. AsterixDB do not have this complexity (as per my knowledge) >> because they either return the results immediately or asynchronously where >> they do not have to store results for previous requests. >> >> I was thinking of using files to store results until users come asking for >> the results. This will add some complexity to code, but will be more >> convenient and efficient memory wise during runtime. Before moving >> forward, >> I would like to know your opinion on this? >> >> [1] >> https://cwiki.apache.org/confluence/display/VXQUERY/SwaggerI >> O+Configuration >> [2] >> https://github.com/erandiganepola/vxquery/blob/master/ >> vxquery-rest/src/main/java/org/apache/vxquery/rest/core/VXQuery.java#L97 >> >> Thanks and Regards, >> *Erandi Ganepola* >> Undergraduate >> BSc. Management and Information Technology (IT Sp.) >> University of Kelaniya, Sri Lanka >> >> <https://www.linkedin.com/in/erandiganepola/> >> <https://github.com/erandiganepola> <https://twitter.com/erandiganepola >> > >> >> On Thu, Jun 22, 2017 at 12:55 PM, Erandi Ganepola < >> [email protected]> >> wrote: >> >> Hi Ian, >>> >>> Thanks for the clarification. I saw AsterixDB is passing the >>> HyracksClientConnection through the ctx. Right now I have implemented >>> VXQuery to create a new HyracksClientConnection for each query request >>> since I thought there will be concurrency problems. I will alter that to >>> make use of only one connection along with proper synchronization. >>> >>> Thanks and Regards, >>> >>> *Erandi Ganepola* >>> Undergraduate >>> BSc. Management and Information Technology (IT Sp.) >>> University of Kelaniya, Sri Lanka >>> >>> <https://www.linkedin.com/in/erandiganepola/> >>> <https://github.com/erandiganepola> <https://twitter.com/erandiga >>> nepola> >>> >>> On Thu, Jun 22, 2017 at 4:05 AM, Ian Maxon <[email protected]> wrote: >>> >>> You should only need one HyracksClientConnection to run queries. If >>>> you look in RestApiServlet in AsterixDB you can see that it's stored >>>> in a context that's kept between requests, and passed as a parameter >>>> to the query compiler which then uses it to submit the compiled >>>> Hyracks job for execution. >>>> >>>> On Wed, Jun 21, 2017 at 1:56 AM, Erandi Ganepola >>>> <[email protected]> wrote: >>>> >>>>> Hi all, >>>>> >>>>> Small inquiry on HyracksConnection class. Do I need to create a new >>>>> HyracksConnection for each query that I'm trying to execute? Or can I >>>>> >>>> reuse >>>> >>>>> the same connection? >>>>> >>>>> Thank you! >>>>> >>>>> *Erandi Ganepola* >>>>> Undergraduate >>>>> BSc. Management and Information Technology (IT Sp.) >>>>> University of Kelaniya, Sri Lanka >>>>> >>>>> <https://www.linkedin.com/in/erandiganepola/> >>>>> <https://github.com/erandiganepola> <https://twitter.com/erandiga >>>>> >>>> nepola> >>>> >>>>> On Tue, Jun 13, 2017 at 6:23 PM, Erandi Ganepola < >>>>> >>>> [email protected]> >>>> >>>>> wrote: >>>>> >>>>> Hi all, >>>>>> >>>>>> One more question. I saw that VXQuery isn't using any logging >>>>>> framework >>>>>> (Other than java.util.logging). Is there any specific reason for that? >>>>>> >>>>>> Thank you! >>>>>> >>>>>> *Erandi Ganepola* >>>>>> Undergraduate >>>>>> BSc. Management and Information Technology (IT Sp.) >>>>>> University of Kelaniya, Sri Lanka >>>>>> >>>>>> <https://www.linkedin.com/in/erandiganepola/> >>>>>> <https://github.com/erandiganepola> <https://twitter.com/erandiga >>>>>> >>>>> nepola> >>>> >>>>> On Tue, Jun 13, 2017 at 6:12 AM, Till Westmann <[email protected]> >>>>>> >>>>> wrote: >>>> >>>>> I think that should be fine - assuming that we get the change merged >>>>>>> >>>>>> soon. >>>> >>>>> @Christina, @Preston: Do we have outstanding changes that need to go >>>>>>> >>>>>> in >>>> >>>>> before we release current master? >>>>>>> >>>>>>> Cheers, >>>>>>> Till >>>>>>> >>>>>>> On 10 Jun 2017, at 1:31, Erandi Ganepola wrote: >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>>> Thanks for the clarification. I have started working by adding only >>>>>>>> *hyracks-http* dependency separately (which is from AsterixDB's >>>>>>>> >>>>>>> inbuilt >>>> >>>>> hyracks version available in maven repository [1]) to the "pom". I >>>>>>>> >>>>>>> will >>>> >>>>> update after merging the change to VXQuery. I hope that will be okay? >>>>>>>> >>>>>>>> [1]https://mvnrepository.com/artifact/org.apache.hyracks/hyr >>>>>>>> >>>>>>> acks-http >>>> >>>>> Thanks and Regards, >>>>>>>> >>>>>>>> *Erandi Ganepola* >>>>>>>> Undergraduate >>>>>>>> BSc. Management and Information Technology (IT Sp.) >>>>>>>> University of Kelaniya, Sri Lanka >>>>>>>> >>>>>>>> <https://www.linkedin.com/in/erandiganepola/> >>>>>>>> <https://github.com/erandiganepola> <https://twitter.com/erandiga >>>>>>>> nepola> >>>>>>>> >>>>>>>> On Sat, Jun 10, 2017 at 3:26 AM, Preston Carman < >>>>>>>> [email protected] >>>>>>>> wrote: >>>>>>>> >>>>>>>> Yes it does require an updated. Till has been working on a PR for >>>>>>>> >>>>>>> this >>>> >>>>> change [1]. You should be able to build off this branch for >>>>>>>>> development. We >>>>>>>>> are just waiting until after the next release of VXQuery to merge >>>>>>>>> >>>>>>>> this >>>> >>>>> change. Christina is working on creating a new release with all the >>>>>>>>> >>>>>>>> work >>>> >>>>> done to support the JSONiq extension to XQuery. Hopefully this will >>>>>>>>> >>>>>>>> be >>>> >>>>> done >>>>>>>>> soon. >>>>>>>>> >>>>>>>>> [1] https://github.com/apache/vxquery/pull/168 >>>>>>>>> >>>>>>>>> On Fri, Jun 9, 2017 at 12:14 PM, Ian Maxon <[email protected]> wrote: >>>>>>>>> >>>>>>>>> I believe VXQuery needs to be updated to work with the latest >>>>>>>>> >>>>>>>> Hyracks to >>>> >>>>> use hyracks-http, though I think Till and Preston would know more >>>>>>>>>> >>>>>>>>> on >>>> >>>>> this >>>>>>>>>> topic than I >>>>>>>>>> >>>>>>>>>> On Wed, Jun 7, 2017 at 10:18 PM, Erandi Ganepola < >>>>>>>>>> >>>>>>>>>> [email protected] >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>>> As I noticed, VXQuery is using the main >>>>>>>>>>> >>>>>>>>>> *incubator-asterixdb-hyracks* >>>> >>>>> [1] >>>>>>>>>> project (0.2.17-incubating) while AsterixDB has an *inbuilt >>>>>>>>>> hyracks >>>>>>>>>> >>>>>>>>>>> version* [2] (stable release 0.3.1). Out of these two, >>>>>>>>>>> >>>>>>>>>> *hyracks-http* >>>> >>>>> is >>>>>>>>>>> from AsterixDB's inbuilt hyracks version [3] (Main hyracks >>>>>>>>>>> project >>>>>>>>>>> >>>>>>>>>>> doesn't >>>>>>>>>> have hyracks-http module) and is available in maven repository >>>>>>>>>> >>>>>>>>> [4]. I >>>> >>>>> assumed this isn't an issue and moved on. However, I would like to >>>>>>>>>>> know >>>>>>>>>>> what do you think on this? >>>>>>>>>>> >>>>>>>>>>> [1] https://github.com/apache/incubator-asterixdb-hyracks >>>>>>>>>>> [2] https://github.com/apache/asterixdb/tree/master/hyracks-full >>>>>>>>>>> >>>>>>>>>> stack >>>> >>>>> [3] https://github.com/apache/asterixdb/tree/master/hyracks- >>>>>>>>>>> fullstack/hyracks/hyracks-http >>>>>>>>>>> [4] https://mvnrepository.com/artifact/org.apache.hyracks/hyrack >>>>>>>>>>> s-http >>>>>>>>>>> >>>>>>>>>>> Thanks and Regards, >>>>>>>>>>> >>>>>>>>>>> *Erandi Ganepola* >>>>>>>>>>> Undergraduate >>>>>>>>>>> BSc. Management and Information Technology (IT Sp.) >>>>>>>>>>> University of Kelaniya, Sri Lanka >>>>>>>>>>> >>>>>>>>>>> <https://www.linkedin.com/in/erandiganepola/> >>>>>>>>>>> <https://github.com/erandiganepola> >>>>>>>>>>> <https://twitter.com/erandiganepola> >>>>>>>>>>> >>>>>>>>>>> On Sat, Jun 3, 2017 at 12:17 PM, Erandi Ganepola < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>>> I went through Hyracks-http framework. I do agree with Till >>>>>>>>>>>> >>>>>>>>>>> Westmann >>>> >>>>> since Jersey is having both CDDL and GPL with classpath >>>>>>>>>>>> >>>>>>>>>>> exception. I >>>> >>>>> understand why its not suiting to Apache license policies. So I'm >>>>>>>>>>>> starting to implement REST API with Hyracks-http framework. >>>>>>>>>>>> Thanks for all your information. More suggestions are highly >>>>>>>>>>>> >>>>>>>>>>>> appreciated. >>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> *Erandi Ganepola* >>>>>>>>>>>> Undergraduate >>>>>>>>>>>> BSc. Management and Information Technology (IT Sp.) >>>>>>>>>>>> University of Kelaniya, Sri Lanka >>>>>>>>>>>> >>>>>>>>>>>> <https://www.linkedin.com/in/erandiganepola/> >>>>>>>>>>>> <https://github.com/erandiganepola> >>>>>>>>>>>> <https://twitter.com/erandiganepola> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 30, 2017 at 11:23 PM, Erandi Ganepola < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Ian and Till, >>>>>>>>>>>> >>>>>>>>>>>>> Thank you for your opinions. I will further look into them and >>>>>>>>>>>>> >>>>>>>>>>>> get >>>> >>>>> back >>>>>>>>>>>> >>>>>>>>>>> to you with more details. >>>>>>>>>> >>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> *Erandi Ganepola* >>>>>>>>>>>>> Undergraduate >>>>>>>>>>>>> BSc. Management and Information Technology (IT Sp.) >>>>>>>>>>>>> University of Kelaniya, Sri Lanka >>>>>>>>>>>>> >>>>>>>>>>>>> <https://www.linkedin.com/in/erandiganepola/> >>>>>>>>>>>>> <https://github.com/erandiganepola> >>>>>>>>>>>>> <https://twitter.com/erandiganepola> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 30, 2017 at 7:05 AM, Ian Maxon <[email protected]> >>>>>>>>>>>>> >>>>>>>>>>>> wrote: >>>> >>>>> Hey Erandi, >>>>>>>>>>>>> >>>>>>>>>>>>>> Sorry for the late reply, this got buried in my mailing list >>>>>>>>>>>>>> >>>>>>>>>>>>> tags. >>>> >>>>> I >>>>>>>>>>>>>> also think reusing hyracks-http is probably the way to go. I >>>>>>>>>>>>>> >>>>>>>>>>>>> would >>>> >>>>> reckon there are probably more ways a library that would >>>>>>>>>>>>>> >>>>>>>>>>>>> attempt to >>>> >>>>> constrain to RESTful-ness might be harder to use than >>>>>>>>>>>>>> >>>>>>>>>>>>> vice-versa, >>>> >>>>> but >>>>>>>>>>>>>> this is just a feeling. Maybe if you could give an example of >>>>>>>>>>>>>> >>>>>>>>>>>>> where >>>> >>>>> you think the advantage might be, I could understand the >>>>>>>>>>>>>> >>>>>>>>>>>>> tradeoff >>>> >>>>> better. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Ian >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, May 29, 2017 at 10:12 AM, Till Westmann < >>>>>>>>>>>>>> >>>>>>>>>>>>> [email protected]> >>>> >>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Erandi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> as you might have seen Hyracks (the runtime system underlying >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> VXQuery) has >>>>>>>>>>>>>> >>>>>>>>>>>>>> it’s own servlet-like framework [1] that is based on the >>>>>>>>>>>>>>> >>>>>>>>>>>>>> Netty NIO >>>> >>>>> framework >>>>>>>>>>>>>> >>>>>>>>>>>>>> [2]. >>>>>>>>>>>>>>> One of the reasons why this framework was adopted is that the >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> licenses for >>>>>>>>>>>>>> >>>>>>>>>>>>>> the servlet API and corresponding frameworks like Jersey >>>>>>>>>>>>>>> >>>>>>>>>>>>>> (CDDL and >>>> >>>>> GPL with >>>>>>>>>>>>>> >>>>>>>>>>>>>> classpath exception) are not ideal for downstream reuse of the >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> project. The >>>>>>>>>>>>>> >>>>>>>>>>>>>> Apache Software foundation categorizes the CDDL as "category >>>>>>>>>>>>>>> >>>>>>>>>>>>>> B" >>>> >>>>> [3] >>>>>>>>>>>>>>> (acceptable with appropriate labeling) and the GPL with >>>>>>>>>>>>>>> >>>>>>>>>>>>>> classpath >>>> >>>>> exception >>>>>>>>>>>>>> >>>>>>>>>>>>>> as "category X" [4] (not acceptable). >>>>>>>>>>>>>>> As hyracks-http framework is available and as it makes >>>>>>>>>>>>>>> >>>>>>>>>>>>>> downstream >>>> >>>>> consumption easy, I think that your implementation of the >>>>>>>>>>>>>>> >>>>>>>>>>>>>> HTTP API >>>> >>>>> for >>>>>>>>>>>>>> >>>>>>>>>>>>>> VXQuery should be based on it. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>> Till >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>> https://github.com/apache/asterixdb/tree/master/hyracks-full >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> stack/hyracks/hyracks-http >>>>>>>>>>>>>> >>>>>>>>>>>>>> [2] https://netty.io/ >>>>>>>>>>>>>>> [3] https://www.apache.org/legal/resolved.html#category-b >>>>>>>>>>>>>>> [4] https://www.apache.org/legal/resolved.html#category-x >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 25 May 2017, at 1:06, Erandi Ganepola wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Ian and Preston, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I would be really grateful if you can look into my previous >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> email >>>> >>>>> in which >>>>>>>>>>>>>>> I suggested few possible alternatives for the REST API >>>>>>>>>>>>>>> implementation and >>>>>>>>>>>>>>> kindly give your opinions on that. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *Erandi Ganepola* >>>>>>>>>>>>>>>> Undergraduate >>>>>>>>>>>>>>>> BSc. Management and Information Technology (IT Sp.) >>>>>>>>>>>>>>>> University of Kelaniya, Sri Lanka >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> <https://www.linkedin.com/in/erandiganepola/> >>>>>>>>>>>>>>>> <https://github.com/erandiganepola> < >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://twitter.com/erandiga >>>>>>>>>>>>>>> >>>>>>>>>>>>>> nepola> >>>>>>>>>> >>>>>>>>>>> On Sat, May 13, 2017 at 10:22 PM, Erandi Ganepola >>>>>>>>>>>>>>>> <[email protected]> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Ian and Preston, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> As we discussed, I went through the AsterixDB's REST API >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> implementation. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> They have written the REST API from scratch using servlet >>>>>>>>>>>>>>> >>>>>>>>>>>>>> basics >>>> >>>>> [1]. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> What >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I feel is that it won't be efficient to put such effort to >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> write >>>> >>>>> servlets >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> from scratch since we have more easy to use, yet stable >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> implementations >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> like jersey [2]. Therefore, my opinion is that using >>>>>>>>>>>>>>> jersey[2] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> will make >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> the VXQuery REST API implementation much easier and easily >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> understandable. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> However, I'm fine to continue on the REST server >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> implementation >>>> >>>>> with any >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> of those two methods (pure servlet based or Jersey & JAX-RS >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> based). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Please >>>>>>>>>> >>>>>>>>>>> let me know your opinion on these two methods and which >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> method >>>> >>>>> you >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> think >>>>>>>>>> >>>>>>>>>>> will be better. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [1] https://github.com/apache/aste >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> rixdb/tree/master/asterixd >>>> >>>>> b/asterix-app/src/main/java/or >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> g/apache/asterix/api/http/server >>>> >>>>> <https://www.google.com/url?q= >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https%3A%2F%2Fgithub.com%2Fapa >>>> >>>>> che%2Fasterixdb%2Ftree%2Fmaster%2Fasterixdb%2Fasterix-app%2F >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> src%2Fmain%2Fjava%2Forg%2Fapache%2Fasterix%2Fapi%2Fhttp%2Fse >>>>>>>>>>>>>> rver&sa=D&sntz=1&usg=AFQjCNGUlYltQj2h-sheau2X44kyNc0O7g> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [2] https://jersey.java.net >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> <https://www.google.com/url?q= >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https%3A%2F%2Fjersey.java.net& >>>> >>>>> sa=D&sntz=1&usg=AFQjCNGnWT2V5YQbw8EuNfRlHPkxgNQ3uQ> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> *Erandi Ganepola* >>>>>>>>>>>>>>>>> Undergraduate >>>>>>>>>>>>>>>>> BSc. Management and Information Technology (IT Sp.) >>>>>>>>>>>>>>>>> University of Kelaniya, Sri Lanka >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> <https://www.linkedin.com/in/erandiganepola/> >>>>>>>>>>>>>>>>> <https://github.com/erandiganepola> >>>>>>>>>>>>>>>>> <https://twitter.com/erandiganepola> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>> >
