Hi Now i can use carbondata 1.0.0 with spark-shell(spark 2.1) as:
./bin/spark-shell --jars <carbondata assembly jar path> but it's inconvenient to get the query time , so i try to use ./bin/spark-sql --jars <carbondata assembly jar path>,but i found some errors when create table : spark-sql> create table if not exists test_table(id string, name string, city string, age int) stored by 'carbondata'; Error in query: Operation not allowed:STORED BY(line 1, pos 87) it seems that the carbondata jar is not load successfully. How can i use ./bin/spark-sql? Regards Libis 2017-02-07 13:16 GMT+08:00 Liang Chen <[email protected]>: > Hi > > I used the below method in spark shell for DEMO, for your reference: > > import org.apache.spark.sql.catalyst.util._ > > benchmark { carbondf.filter($"name" === "Allen" and $"gender" === "Male" > and $"province" === "NB" and $"singler" === "false").count } > > > Regards > > Liang > > 2017-02-06 22:07 GMT-05:00 Yinwei Li <[email protected]>: > > > Hi all, > > > > > > When we are using sparkshell + carbondata to send a query, how can we > > get the excution duration? Some topics are thrown as follows: > > > > > > 1. One query can produce one or more jobs, and some of the jobs may > have > > DAG dependence, thus we can't get the excution duration by sum up all the > > jobs' duration or get the max duration of the jobs roughly. > > > > > > 2. In the spark shell console or spark application web ui, we can get > > each job's duration, but we can't get the carbondata-query directly, if > > some improvement would take by carbondata in the near future. > > > > > > 3. Maybe we can use the following command to get a approximate result: > > > > > > scala > val begin = new Date();cc.sql("$SQL_COMMAND").show;val end = > > new Date(); > > > > > > Any other opinions? > > > > > -- > Regards > Liang >
