Erandi,
AsterixDB actually spools its results to local result partitions and
then delivers them out the top - it does this so that resources like
memory can't be held hostage by the system's clients. @Till, can you
add a few sentences about this?
Cheers,
MIke
On 7/10/17 9:48 PM, Erandi Ganepola wrote:
Hi all,
As per the REST API definition [1], it operates in two steps.
1. Users send the statement to be queried and REST API responds with a
restult Id.
2. Later users ask for the results for the result id (received at earlier
step) and server returns the results.
Because of this two step process, we need to keep the results stored until
the user come and ask for it. I'm currently storing these results in memory
[2] which consumes a lot of memory when results for multiple requests gets
accumulated. AsterixDB do not have this complexity (as per my knowledge)
because they either return the results immediately or asynchronously where
they do not have to store results for previous requests.
I was thinking of using files to store results until users come asking for
the results. This will add some complexity to code, but will be more
convenient and efficient memory wise during runtime. Before moving forward,
I would like to know your opinion on this?
[1]
https://cwiki.apache.org/confluence/display/VXQUERY/SwaggerIO+Configuration
[2]
https://github.com/erandiganepola/vxquery/blob/master/vxquery-rest/src/main/java/org/apache/vxquery/rest/core/VXQuery.java#L97
Thanks and Regards,
*Erandi Ganepola*
Undergraduate
BSc. Management and Information Technology (IT Sp.)
University of Kelaniya, Sri Lanka
<https://www.linkedin.com/in/erandiganepola/>
<https://github.com/erandiganepola> <https://twitter.com/erandiganepola>
On Thu, Jun 22, 2017 at 12:55 PM, Erandi Ganepola <[email protected]>
wrote:
Hi Ian,
Thanks for the clarification. I saw AsterixDB is passing the
HyracksClientConnection through the ctx. Right now I have implemented
VXQuery to create a new HyracksClientConnection for each query request
since I thought there will be concurrency problems. I will alter that to
make use of only one connection along with proper synchronization.
Thanks and Regards,
*Erandi Ganepola*
Undergraduate
BSc. Management and Information Technology (IT Sp.)
University of Kelaniya, Sri Lanka
<https://www.linkedin.com/in/erandiganepola/>
<https://github.com/erandiganepola> <https://twitter.com/erandiganepola>
On Thu, Jun 22, 2017 at 4:05 AM, Ian Maxon <[email protected]> wrote:
You should only need one HyracksClientConnection to run queries. If
you look in RestApiServlet in AsterixDB you can see that it's stored
in a context that's kept between requests, and passed as a parameter
to the query compiler which then uses it to submit the compiled
Hyracks job for execution.
On Wed, Jun 21, 2017 at 1:56 AM, Erandi Ganepola
<[email protected]> wrote:
Hi all,
Small inquiry on HyracksConnection class. Do I need to create a new
HyracksConnection for each query that I'm trying to execute? Or can I
reuse
the same connection?
Thank you!
*Erandi Ganepola*
Undergraduate
BSc. Management and Information Technology (IT Sp.)
University of Kelaniya, Sri Lanka
<https://www.linkedin.com/in/erandiganepola/>
<https://github.com/erandiganepola> <https://twitter.com/erandiga
nepola>
On Tue, Jun 13, 2017 at 6:23 PM, Erandi Ganepola <
[email protected]>
wrote:
Hi all,
One more question. I saw that VXQuery isn't using any logging framework
(Other than java.util.logging). Is there any specific reason for that?
Thank you!
*Erandi Ganepola*
Undergraduate
BSc. Management and Information Technology (IT Sp.)
University of Kelaniya, Sri Lanka
<https://www.linkedin.com/in/erandiganepola/>
<https://github.com/erandiganepola> <https://twitter.com/erandiga
nepola>
On Tue, Jun 13, 2017 at 6:12 AM, Till Westmann <[email protected]>
wrote:
I think that should be fine - assuming that we get the change merged
soon.
@Christina, @Preston: Do we have outstanding changes that need to go
in
before we release current master?
Cheers,
Till
On 10 Jun 2017, at 1:31, Erandi Ganepola wrote:
Hi All,
Thanks for the clarification. I have started working by adding only
*hyracks-http* dependency separately (which is from AsterixDB's
inbuilt
hyracks version available in maven repository [1]) to the "pom". I
will
update after merging the change to VXQuery. I hope that will be okay?
[1]https://mvnrepository.com/artifact/org.apache.hyracks/hyr
acks-http
Thanks and Regards,
*Erandi Ganepola*
Undergraduate
BSc. Management and Information Technology (IT Sp.)
University of Kelaniya, Sri Lanka
<https://www.linkedin.com/in/erandiganepola/>
<https://github.com/erandiganepola> <https://twitter.com/erandiga
nepola>
On Sat, Jun 10, 2017 at 3:26 AM, Preston Carman <[email protected]
wrote:
Yes it does require an updated. Till has been working on a PR for
this
change [1]. You should be able to build off this branch for
development. We
are just waiting until after the next release of VXQuery to merge
this
change. Christina is working on creating a new release with all the
work
done to support the JSONiq extension to XQuery. Hopefully this will
be
done
soon.
[1] https://github.com/apache/vxquery/pull/168
On Fri, Jun 9, 2017 at 12:14 PM, Ian Maxon <[email protected]> wrote:
I believe VXQuery needs to be updated to work with the latest
Hyracks to
use hyracks-http, though I think Till and Preston would know more
on
this
topic than I
On Wed, Jun 7, 2017 at 10:18 PM, Erandi Ganepola <
[email protected]
wrote:
Hi all,
As I noticed, VXQuery is using the main
*incubator-asterixdb-hyracks*
[1]
project (0.2.17-incubating) while AsterixDB has an *inbuilt hyracks
version* [2] (stable release 0.3.1). Out of these two,
*hyracks-http*
is
from AsterixDB's inbuilt hyracks version [3] (Main hyracks project
doesn't
have hyracks-http module) and is available in maven repository
[4]. I
assumed this isn't an issue and moved on. However, I would like to
know
what do you think on this?
[1] https://github.com/apache/incubator-asterixdb-hyracks
[2] https://github.com/apache/asterixdb/tree/master/hyracks-full
stack
[3] https://github.com/apache/asterixdb/tree/master/hyracks-
fullstack/hyracks/hyracks-http
[4] https://mvnrepository.com/artifact/org.apache.hyracks/hyrack
s-http
Thanks and Regards,
*Erandi Ganepola*
Undergraduate
BSc. Management and Information Technology (IT Sp.)
University of Kelaniya, Sri Lanka
<https://www.linkedin.com/in/erandiganepola/>
<https://github.com/erandiganepola>
<https://twitter.com/erandiganepola>
On Sat, Jun 3, 2017 at 12:17 PM, Erandi Ganepola <
[email protected]> wrote:
Hi All,
I went through Hyracks-http framework. I do agree with Till
Westmann
since Jersey is having both CDDL and GPL with classpath
exception. I
understand why its not suiting to Apache license policies. So I'm
starting to implement REST API with Hyracks-http framework.
Thanks for all your information. More suggestions are highly
appreciated.
Regards,
*Erandi Ganepola*
Undergraduate
BSc. Management and Information Technology (IT Sp.)
University of Kelaniya, Sri Lanka
<https://www.linkedin.com/in/erandiganepola/>
<https://github.com/erandiganepola>
<https://twitter.com/erandiganepola>
On Tue, May 30, 2017 at 11:23 PM, Erandi Ganepola <
[email protected]> wrote:
Hi Ian and Till,
Thank you for your opinions. I will further look into them and
get
back
to you with more details.
Thanks and Regards,
*Erandi Ganepola*
Undergraduate
BSc. Management and Information Technology (IT Sp.)
University of Kelaniya, Sri Lanka
<https://www.linkedin.com/in/erandiganepola/>
<https://github.com/erandiganepola>
<https://twitter.com/erandiganepola>
On Tue, May 30, 2017 at 7:05 AM, Ian Maxon <[email protected]>
wrote:
Hey Erandi,
Sorry for the late reply, this got buried in my mailing list
tags.
I
also think reusing hyracks-http is probably the way to go. I
would
reckon there are probably more ways a library that would
attempt to
constrain to RESTful-ness might be harder to use than
vice-versa,
but
this is just a feeling. Maybe if you could give an example of
where
you think the advantage might be, I could understand the
tradeoff
better.
-Ian
On Mon, May 29, 2017 at 10:12 AM, Till Westmann <
[email protected]>
wrote:
Hi Erandi,
as you might have seen Hyracks (the runtime system underlying
VXQuery) has
it’s own servlet-like framework [1] that is based on the
Netty NIO
framework
[2].
One of the reasons why this framework was adopted is that the
licenses for
the servlet API and corresponding frameworks like Jersey
(CDDL and
GPL with
classpath exception) are not ideal for downstream reuse of the
project. The
Apache Software foundation categorizes the CDDL as "category
B"
[3]
(acceptable with appropriate labeling) and the GPL with
classpath
exception
as "category X" [4] (not acceptable).
As hyracks-http framework is available and as it makes
downstream
consumption easy, I think that your implementation of the
HTTP API
for
VXQuery should be based on it.
Cheers,
Till
[1]
https://github.com/apache/asterixdb/tree/master/hyracks-full
stack/hyracks/hyracks-http
[2] https://netty.io/
[3] https://www.apache.org/legal/resolved.html#category-b
[4] https://www.apache.org/legal/resolved.html#category-x
On 25 May 2017, at 1:06, Erandi Ganepola wrote:
Hi Ian and Preston,
I would be really grateful if you can look into my previous
email
in which
I suggested few possible alternatives for the REST API
implementation and
kindly give your opinions on that.
Thanks and Regards,
*Erandi Ganepola*
Undergraduate
BSc. Management and Information Technology (IT Sp.)
University of Kelaniya, Sri Lanka
<https://www.linkedin.com/in/erandiganepola/>
<https://github.com/erandiganepola> <
https://twitter.com/erandiga
nepola>
On Sat, May 13, 2017 at 10:22 PM, Erandi Ganepola
<[email protected]>
wrote:
Hi Ian and Preston,
As we discussed, I went through the AsterixDB's REST API
implementation.
They have written the REST API from scratch using servlet
basics
[1].
What
I feel is that it won't be efficient to put such effort to
write
servlets
from scratch since we have more easy to use, yet stable
implementations
like jersey [2]. Therefore, my opinion is that using jersey[2]
will make
the VXQuery REST API implementation much easier and easily
understandable.
However, I'm fine to continue on the REST server
implementation
with any
of those two methods (pure servlet based or Jersey & JAX-RS
based).
Please
let me know your opinion on these two methods and which
method
you
think
will be better.
[1] https://github.com/apache/aste
rixdb/tree/master/asterixd
b/asterix-app/src/main/java/or
g/apache/asterix/api/http/server
<https://www.google.com/url?q=
https%3A%2F%2Fgithub.com%2Fapa
che%2Fasterixdb%2Ftree%2Fmaster%2Fasterixdb%2Fasterix-app%2F
src%2Fmain%2Fjava%2Forg%2Fapache%2Fasterix%2Fapi%2Fhttp%2Fse
rver&sa=D&sntz=1&usg=AFQjCNGUlYltQj2h-sheau2X44kyNc0O7g>
[2] https://jersey.java.net
<https://www.google.com/url?q=
https%3A%2F%2Fjersey.java.net&
sa=D&sntz=1&usg=AFQjCNGnWT2V5YQbw8EuNfRlHPkxgNQ3uQ>
Thanks and Regards,
*Erandi Ganepola*
Undergraduate
BSc. Management and Information Technology (IT Sp.)
University of Kelaniya, Sri Lanka
<https://www.linkedin.com/in/erandiganepola/>
<https://github.com/erandiganepola>
<https://twitter.com/erandiganepola>
