Yes with lombok plugin, these errors are gone. Thanks Ajay
On Fri, Jul 24, 2015 at 2:40 PM, Ajay <[email protected]> wrote: > For example 1 such error in QueryExecutionServiceImp.java Lens Server > project > > > private LensResultSet getResultset(QueryHandle queryHandle) throws > LensException { > QueryContext ctx = allQueries.get(queryHandle); > if (ctx == null) { > return getResultsetFromDAO(queryHandle); > } else { > synchronized (ctx) { > if (ctx.isFinishedQueryPersisted()) { > return getResultsetFromDAO(queryHandle); > } > LensResultSet resultSet = resultSets.get(queryHandle); > if (resultSet == null) { > if (ctx.isPersistent() && ctx.getQueryOutputFormatter() != null) { > resultSets > .put(queryHandle, > new LensPersistentResult( > ctx.getQueryOutputFormatter().getMetadata(), > ctx.getQueryOutputFormatter().getFinalOutputPath(), > ctx.getQueryOutputFormatter().getNumRows())); > } else if (allQueries.get(queryHandle).isResultAvailableInDriver()) { > resultSet = > allQueries.get(queryHandle).getSelectedDriver().fetchResultSet(allQueries.get(queryHandle)); > resultSets.put(queryHandle, resultSet); > } else { > throw new NotFoundException("Result set not available for query:" + > queryHandle); > } > } > } > if (resultSets.get(queryHandle) instanceof InMemoryResultSet) { > ((InMemoryResultSet) resultSets.get(queryHandle)).seekToStart(); > } > return resultSets.get(queryHandle); > } > } > > There are no such methods in QueryContext() but annotated with @Getter and > @Setter. Do we need any plugin to solve it? > > Thanks > > Ajay > > > > On Fri, Jul 24, 2015 at 2:33 PM, Rajat Khandelwal <[email protected]> > wrote: > >> Yes, can you paste the exact error? >> >> On Fri, Jul 24, 2015 at 2:11 PM Yash Sharma <[email protected]> wrote: >> >> > Its certainly something wrt Env. Can you just copy paste couple of >> errors ? >> > >> > On Fri, Jul 24, 2015 at 2:00 PM, Ajay <[email protected]> wrote: >> > >> > > I did the same from command line. >> > > >> > > mvn clean install -DskipTests -DskipCheck >> > > mvn idea:idea >> > > >> > > >> > > Java version is 1.8 >> > > >> > > I tried opening the project directly and selected >> > > /home/ajay/openSource/incubator-lens. Also I tried importing as maven >> > > project. Both the cases I see compilation errors. >> > > >> > > - >> > > Ajay >> > > >> > > >> > > On Fri, Jul 24, 2015 at 1:42 PM, Rajat Khandelwal < >> > [email protected]> >> > > wrote: >> > > >> > > > I'm using idea too. There are no compilation issues. Can you do >> > > > >> > > > mvn clean install -DskipTests -DskipCheck >> > > > mvn idea:idea >> > > > >> > > > from command line. >> > > > >> > > > >> > > > Also, check your java version. Lens is configured for java >=7 >> > > > >> > > > On Fri, Jul 24, 2015 at 1:36 PM Ajay <[email protected]> wrote: >> > > > >> > > > > From console, the build goes thru fine. >> > > > > >> > > > > From Idea IDE, it is giving compilation errors (say one such file >> > > > > QueryExecutionServiceImp.java >> > > > > Lens Server project). >> > > > > >> > > > > Are you guys using Eclipse? >> > > > > >> > > > > >> > > > > On Fri, Jul 24, 2015 at 1:14 PM, Yash Sharma <[email protected]> >> > > wrote: >> > > > > >> > > > > > Could you share the error. Might be env issues. >> > > > > > Also can you build from the console directly ? >> > > > > > >> > > > > > On Fri, Jul 24, 2015 at 12:54 PM, Ajay <[email protected]> >> > wrote: >> > > > > > >> > > > > > > Thanks. >> > > > > > > >> > > > > > > I am using intelliJ Idea IDE. I checked out the source from >> GIT. >> > > But >> > > > I >> > > > > am >> > > > > > > seeing some compilation issues. Is there any documentation or >> > notes >> > > > on >> > > > > > how >> > > > > > > to set up the development envirnment? >> > > > > > > >> > > > > > > - >> > > > > > > Ajay >> > > > > > > >> > > > > > > On Fri, Jul 24, 2015 at 11:44 AM, amareshwarisr . < >> > > > > [email protected] >> > > > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Hello Ajay, >> > > > > > > > >> > > > > > > > Nice to see your interest in exploring Lens. >> > > > > > > > >> > > > > > > > Looking at the requirement posted by you, seems you are only >> > > > > interested >> > > > > > > in >> > > > > > > > REST api on top of cassandra. See the lens REST api for >> query >> > > > service >> > > > > > > here >> > > > > > > > - >> > > > > >> http://lens.incubator.apache.org/resource_QueryServiceResource.html >> >> > > > > > > > >> > > > > > > > Lens provides REST api for query actions (submit, fetch >> > results, >> > > > > > cancel, >> > > > > > > > getstatus), query history and OLAP data model. >> > > > > > > > >> > > > > > > > If you are not interested in OLAP data model, you can simple >> > run >> > > > > native >> > > > > > > > queries on lens which would give REST api and history for >> you >> > as >> > > > > well. >> > > > > > > For >> > > > > > > > more details on OLAP data model, you can read the doc on >> > website. >> > > > > > > > >> > > > > > > > For new driver to be rewritten, it need not know anything >> about >> > > > OLAP >> > > > > > data >> > > > > > > > model. But all drivers are given HiveQL query to execute, if >> > > driver >> > > > > > > accepts >> > > > > > > > Hive QL directly you dont have to do much on the query >> itself. >> > If >> > > > > not, >> > > > > > > the >> > > > > > > > driver has to translate the query into something driver can >> > > > > understand. >> > > > > > > > >> > > > > > > > For example : JDBCDriver converts HiveQL to SQL. - You can >> have >> > > > look >> > > > > at >> > > > > > > > JDBCDriver for converting into SQL. Hive QL is almost SQL, >> > except >> > > > it >> > > > > > > might >> > > > > > > > have to map some UDFs to SQL udfs. >> > > > > > > > >> > > > > > > > ok. Coming back to your questions, answers are inline. >> > > > > > > > >> > > > > > > > >> > > > > > > > On Fri, Jul 24, 2015 at 9:08 AM, Ajay <[email protected] >> > >> > > > wrote: >> > > > > > > > >> > > > > > > > > Thanks. >> > > > > > > > > >> > > > > > > > > Couple of questions (that comes to my mind right now) >> > > > > > > > > >> > > > > > > > > 1) To create a Cluster instance to connect to a Cassandra >> > > cluster >> > > > > > > > > (Cassandra Java driver), we need to know the following: >> > > > > > > > > a) ip address and port no of few instances of the cluster >> > > > > > > > > b) authentication (if enabled) >> > > > > > > > > c) few more configuration (loading balancing policy, ret >> > policy >> > > > and >> > > > > > > > > compression etc..) >> > > > > > > > > >> > > > > > > > > Lens drivers are following xml based configuration for >> > passing >> > > > > > > > information. See >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/resources/hivedriver-default.xml >> > > > > > > > and >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/incubator-lens/blob/master/lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml >> > > > > > > > , >> > > > > > > > how driver specific configuration is put. Also read >> > > > > > > > http://lens.incubator.apache.org/admin/config-server.html >> to >> > > > > > understand >> > > > > > > > configuration in lens. >> > > > > > > > >> > > > > > > > >> > > > > > > > > Do we have any configuration support(yaml file or any >> other) >> > in >> > > > > Lens. >> > > > > > > If >> > > > > > > > > yes, any pointers (java file names or WIKI) >> > > > > > > > > >> > > > > > > > > 2) Cassandra cluser/session and prepared statements are >> multi >> > > > > thread >> > > > > > > safe >> > > > > > > > > and recommended to be cached and reused. >> > > > > > > > > >> > > > > > > > > How does Lens support Caching (in memory or distributed)?. >> > Any >> > > > > > pointers >> > > > > > > > > (java file names or WIKI) >> > > > > > > > > >> > > > > > > > > I did not understand the caching part here. But if you are >> > > > talking >> > > > > > > about >> > > > > > > > cassandra sessions to be used across user queries, you can >> look >> > > at >> > > > > how >> > > > > > > hive >> > > > > > > > sessions and connections are used in HiveDriver - >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java >> > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > On Thu, Jul 23, 2015 at 11:16 PM, Yash Sharma < >> > > [email protected] >> > > > > >> > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > That is great. We can probably pick the most recent Java >> > > driver >> > > > > and >> > > > > > > > > > Cassandra version then. Since it addresses couple of old >> > > > issues. >> > > > > > > > > > >> > > > > > > > > > Regarding Spark SQL for querying Cassandra I would let >> > other >> > > > > > > > contributors >> > > > > > > > > > suggest. >> > > > > > > > > > >> > > > > > > > > > On Thu, Jul 23, 2015 at 11:04 PM, Ajay < >> > [email protected] >> >> > > > >> > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > Thanks Yash. >> > > > > > > > > > > >> > > > > > > > > > > In Between, ALLOW FILTERING is supported in Cassandra >> > Java >> > > > > driver >> > > > > > > > [1]. >> > > > > > > > > > What >> > > > > > > > > > > is the Cassandra and Java driver version we plan to >> > > > support?. I >> > > > > > > have >> > > > > > > > > > worked >> > > > > > > > > > > on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and >> > > ALLOW >> > > > > > > > FILTERING >> > > > > > > > > > > worked. >> > > > > > > > > > > >> > > > > > > > > > > Secondly, as you mentioned I am aware of these >> limitation >> > > in >> > > > > CQL. >> > > > > > > But >> > > > > > > > > > more >> > > > > > > > > > > features recently added in Cassandra 2.2.x. Also, >> other >> > > > option >> > > > > to >> > > > > > > > work >> > > > > > > > > > > around this to use Cassandra Spark connector and use >> > Spark >> > > > SQL. >> > > > > > > > > > > >> > > > > > > > > > > 1) >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma < >> > > > > [email protected] >> >> > > > > > > >> > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > Just to add on... >> > > > > > > > > > > > >> > > > > > > > > > > > There will be few challenges to support entire SQL >> on >> > > > > Cassandra >> > > > > > > > > > however. >> > > > > > > > > > > > Last time I tried translating SQL to CQL I faced >> couple >> > > of >> > > > > > > > scenarios >> > > > > > > > > - >> > > > > > > > > > > > >> > > > > > > > > > > > 1. Filtering non indexed column in Cassandra >> > > > > > > > > > > > 2. Filtering by subset of primary key >> > > > > > > > > > > > 3. OR condition in where clause >> > > > > > > > > > > > >> > > > > > > > > > > > You can probably start without these being a >> blocker - >> > In >> > > > > > > parallel >> > > > > > > > we >> > > > > > > > > > can >> > > > > > > > > > > > discuss how this can be implemented on our stack. >> > Others >> > > > can >> > > > > > > > provide >> > > > > > > > > > > their >> > > > > > > > > > > > opinions here. >> > > > > > > > > > > > >> > > > > > > > > > > > Hope its helpful. >> > > > > > > > > > > > >> > > > > > > > > > > > ----- >> > > > > > > > > > > > Examples: >> > > > > > > > > > > > Here 'trending_now' is a dummy table with (id, rank, >> > > > pog_id) >> > > > > > > where >> > > > > > > > > > > > (id,rank) is primary key pair. >> > > > > > > > > > > > >> > > > > > > > > > > > 1. >> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where >> > > > pog_id=10004 ; >> > > > > > > > > > > > Bad Request: No indexed columns present in >> by-columns >> > > > clause >> > > > > > with >> > > > > > > > > Equal >> > > > > > > > > > > > operator >> > > > > > > > > > > > >> > > > > > > > > > > > 2. >> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where >> rank=4; >> > > > > > > > > > > > Bad Request: Cannot execute this query as it might >> > > involve >> > > > > data >> > > > > > > > > > filtering >> > > > > > > > > > > > and thus may have unpredictable performance. If you >> > want >> > > to >> > > > > > > execute >> > > > > > > > > > this >> > > > > > > > > > > > query despite the performance unpredictability, use >> > ALLOW >> > > > > > > FILTERING >> > > > > > > > > > > > P.S. ALLOW FILTERING is not permitted in Cassandra >> java >> > > > > driver >> > > > > > as >> > > > > > > > of >> > > > > > > > > > now. >> > > > > > > > > > > > >> > > > > > > > > > > > 3. >> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where >> rank=4 >> > or >> > > > > > > > id='id0004'; >> > > > > > > > > > > > Bad Request: line 1:40 missing EOF at 'or' >> > > > > > > > > > > > >> > > > > > > > > > > > 4. Valid Query: >> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where >> > > id='id0004' >> > > > > and >> > > > > > > > > rank=4; >> > > > > > > > > > > > >> > > > > > > > > > > > id | rank | pog_id >> > > > > > > > > > > > --------+------+-------- >> > > > > > > > > > > > id0004 | 4 | 10002 >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma < >> > > > > > [email protected]> >> > > > > > > > > > wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > > The basic idea is to translate the Lens Query plan >> > > (Which >> > > > > is >> > > > > > a >> > > > > > > > Hive >> > > > > > > > > > > AST) >> > > > > > > > > > > > > to the DataStore specific Plan/Query. >> > > > > > > > > > > > > >> > > > > > > > > > > > > The best example I can think of right now is the >> > > Elastic >> > > > > > Search >> > > > > > > > > patch >> > > > > > > > > > > by >> > > > > > > > > > > > > Amruth. You can go through the JIRA [1] for the >> > > detailed >> > > > > > > > discussion >> > > > > > > > > > and >> > > > > > > > > > > > the >> > > > > > > > > > > > > Review Board [2] for code reference. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Best regards >> > > > > > > > > > > > > >> > > > > > > > > > > > > 1. https://issues.apache.org/jira/browse/LENS-252 >> > > > > > > > > > > > > 2. https://reviews.apache.org/r/36434 >> > > > > > > > > > > > > >> > > > > > > > > > > > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay < >> > > > > [email protected]> >> > > > > > > > > wrote: >> > > > > > > > > > > > > >> > > > > > > > > > > > >> Thanks Yash. >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> Is there any documentation or WIKI on the Lens >> > > Driver. I >> > > > > am >> > > > > > > > going >> > > > > > > > > > > > through >> > > > > > > > > > > > >> the code as well. >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> Thanks >> > > > > > > > > > > > >> Ajay >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma < >> > > > > > > > [email protected]> >> > > > > > > > > > > > wrote: >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> > Hi Ajay, >> > > > > > > > > > > > >> > Welcome to the Lens Dev. >> > > > > > > > > > > > >> > We do have plans for a Cassandra driver [1] for >> > Lens >> > > > but >> > > > > > no >> > > > > > > > one >> > > > > > > > > > has >> > > > > > > > > > > > >> picked >> > > > > > > > > > > > >> > it up yet. It would be great if you can pick up >> > the >> > > > task >> > > > > > and >> > > > > > > > > > submit >> > > > > > > > > > > a >> > > > > > > > > > > > >> patch >> > > > > > > > > > > > >> > for review. >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > Also drop a note on the list in case you >> stumble >> > > upon >> > > > > any >> > > > > > > > issue. >> > > > > > > > > > > > Someone >> > > > > > > > > > > > >> > will always be around to help you out. >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > 1. >> https://issues.apache.org/jira/browse/LENS-654 >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay < >> > > > > > > [email protected]> >> >> > > > > > > > > > > wrote: >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > Hi, >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > I recently noticed about Apache Lens project. >> > > > > Currently >> > > > > > we >> > > > > > > > are >> > > > > > > > > > > > >> building >> > > > > > > > > > > > >> > > REST APIs for Apache Cassandra (proprietary) >> as >> > > > there >> > > > > no >> > > > > > > > such >> > > > > > > > > > > Apache >> > > > > > > > > > > > >> open >> > > > > > > > > > > > >> > > source project exists. Now as Lens supports >> REST >> > > > APIs >> > > > > > for >> > > > > > > > > Hadoop >> > > > > > > > > > > and >> > > > > > > > > > > > >> > JDBC, >> > > > > > > > > > > > >> > > I want to know is there any plan to support >> for >> > > > Apache >> > > > > > > > > Cassandra >> > > > > > > > > > > as >> > > > > > > > > > > > >> its >> > > > > > > > > > > > >> > > support CQL and Spark SQL (thru Spark >> connector) >> > > > which >> > > > > > are >> > > > > > > > > more >> > > > > > > > > > > SQL >> > > > > > > > > > > > >> like. >> > > > > > > > > > > > >> > > If yes, I wish to know the details and >> > contribute >> > > as >> > > > > > well. >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > Thanks >> > > > > > > > > > > > >> > > Ajay >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >
