Hi Chamila, You can't ignore the ':key' column of Cassandra CF when you map it to a hive table. Drop all the meta tables and try with mapping :key column to a column in hive table.
On Mon, Dec 23, 2013 at 11:26 AM, Gihan Anuruddha <[email protected]> wrote: > Hi Chamila, > > First to create a column family you have to create a stream definition. To > do that you have to make some small change in the sample. > > Below code snippet will create stream definition in the sample. > > streamId = dataPublisher.defineStream("{" + > " 'name':'" +* HTTPD_LOG_STREAM* + "'," + > " 'version':'" + VERSION + "'," + > " 'nickName': 'Httpd_Log_Stream'," + > " 'description': 'Sample of Httpd logs'," + > " 'metaData':[" + > " {'name':'clientType','type':'STRING'}" + > " ]," + > " 'payloadData':[" + > " {'name':'log','type':'STRING'}" + > " ]" + > "}"); > > > In your case change *HTTPD_LOG_STREAM *to chamila. So again when you > execute the sample it will create new column family called chamila. Then > try again your script. If doesn't work please share your analytics script > with us. > > Regards, > Gihan > > > On Mon, Dec 23, 2013 at 11:00 AM, Chamila Wijayarathna > <[email protected]>wrote: > >> I executed both queries in that way, but it still gives the same error it >> gave earlier. >> >> >> On Mon, Dec 23, 2013 at 10:55 AM, Gihan Anuruddha <[email protected]> wrote: >> >>> Sorry, I forgot to tell you that above (Drop table Chamila) query you >>> have to execute through the hive query editor. >>> >>> To do that: start BAM and go to management console. Then go to Manage >>> -> Analytics -> Add and enter "Drop table Chamila" and execute. >>> >>> >>> >>> On Mon, Dec 23, 2013 at 10:49 AM, Chamila Wijayarathna <[email protected] >>> > wrote: >>> >>>> Hi, >>>> It still gives the same result. Also "SHOW TABLES", doesn't show that >>>> there is table "Chamila" existing. >>>> >>>> >>>> On Mon, Dec 23, 2013 at 10:29 AM, Gihan Anuruddha <[email protected]>wrote: >>>> >>>>> Hi Chamila, >>>>> >>>>> Execute this query before you execute above one "Drop table Chamila" >>>>> . Basically hive maintain meta information when it creates a new virtual >>>>> table. Once you create a table it won't change again unless you drop that >>>>> virtual table and create a new one with edited parameters. >>>>> >>>>> Regards, >>>>> Gihan >>>>> >>>>> >>>>> >>>>> On Mon, Dec 23, 2013 at 9:51 AM, Chamila Wijayarathna < >>>>> [email protected]> wrote: >>>>> >>>>>> Hii all, >>>>>> How can I create a column family, so that it will suite my need? I >>>>>> tried >>>>>> >>>>>> CREATE EXTERNAL TABLE IF NOT EXISTS Chamila ( >>>>>> remote_ip STRING, >>>>>> request_date STRING,request STRING, >>>>>> httpCode STRING,length STRING, >>>>>> version STRING) >>>>>> STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' >>>>>> WITH SERDEPROPERTIES ( >>>>>> "wso2.carbon.datasource.name" = "WSO2BAM_CASSANDRA_DATASOURCE" , >>>>>> "cassandra.cf.name" = "chamila" , >>>>>> "cassandra.columns.mapping" = >>>>>> *"*remote_ip, request_date, request,httpCode,length, version" ); >>>>>> >>>>>> So now I have all columns in column family has mapped to my hive >>>>>> table, but still I am getting same error[1]. >>>>>> >>>>>> Do I have to configure column family somewhere else before running >>>>>> this code? If yes, how can I do that? >>>>>> >>>>>> 1. Backend Console Log - >>>>>> https://gist.github.com/cdwijayarathna/8091591 >>>>>> >>>>>> Thank You! >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Sun, Dec 22, 2013 at 1:00 PM, Maninda Edirisooriya < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Chamila, >>>>>>> >>>>>>> And also note that the error displayed on the UI when a Hive script >>>>>>> is executed is not enough to figure out the problem. You need to look at >>>>>>> the backend console log as well. >>>>>>> Anyway for the given problem Inosh's correction will work. >>>>>>> >>>>>>> >>>>>>> *Maninda Edirisooriya* >>>>>>> Software Engineer >>>>>>> >>>>>>> *WSO2, Inc. *lean.enterprise.middleware. >>>>>>> >>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>> *Phone* : +94 777603226 >>>>>>> >>>>>>> >>>>>>> On Sat, Dec 21, 2013 at 11:27 PM, Inosh Goonewardena <[email protected] >>>>>>> > wrote: >>>>>>> >>>>>>>> On Sat, Dec 21, 2013 at 10:25 PM, Chamila Wijayarathna < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hii Maninda, >>>>>>>>> Thanks for replying. >>>>>>>>> I tried creating a table which will suite my need, with different >>>>>>>>> set of columns from above example. >>>>>>>>> >>>>>>>>> CREATE EXTERNAL TABLE IF NOT EXISTS Chamila ( >>>>>>>>> remote_ip STRING, >>>>>>>>> request_date STRING,request STRING, >>>>>>>>> httpCode STRING,length STRING, >>>>>>>>> version STRING) >>>>>>>>> STORED BY >>>>>>>>> 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' >>>>>>>>> WITH SERDEPROPERTIES ( >>>>>>>>> "wso2.carbon.datasource.name" = "WSO2BAM_CASSANDRA_DATASOURCE" , >>>>>>>>> "cassandra.cf.name" = "processed_logs" , >>>>>>>>> "cassandra.columns.mapping" = >>>>>>>>> *":key*,remote_ip, request_date, request,httpCode,length, >>>>>>>>> version" ); >>>>>>>>> >>>>>>>>> But I get an error saying, "Error while executing Hive >>>>>>>>> script.Query returned non-zero code: 9, cause: FAILED: Execution >>>>>>>>> Error, >>>>>>>>> return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask". >>>>>>>>> >>>>>>>> >>>>>>>> Seems like ':key' column has not been mapped to any column of hive >>>>>>>> your table. For example, in original query it is mapped to logId. >>>>>>>> >>>>>>>> How can I create a table to suite what I want. As I understood, it >>>>>>>>> has no meaning to create a virtual table without linking it with >>>>>>>>> Column >>>>>>>>> Family. >>>>>>>>> >>>>>>>>> Thank you! >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Dec 20, 2013 at 5:58 PM, Maninda Edirisooriya < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> On Fri, Dec 20, 2013 at 4:54 PM, Chamila Wijayarathna < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hello All, >>>>>>>>>>> I am running BAM sample 1. >>>>>>>>>>> In the analytic script which get run when I sent log line event >>>>>>>>>>> it has query >>>>>>>>>>> >>>>>>>>>>> CREATE EXTERNAL TABLE IF NOT EXISTS ProcessedLogs ( >>>>>>>>>>> logId STRING,remote_ip STRING, >>>>>>>>>>> request_date STRING,request STRING, >>>>>>>>>>> httpCode STRING,length STRING, >>>>>>>>>>> version STRING) >>>>>>>>>>> STORED BY >>>>>>>>>>> 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' >>>>>>>>>>> WITH SERDEPROPERTIES ( >>>>>>>>>>> "wso2.carbon.datasource.name" = "WSO2BAM_CASSANDRA_DATASOURCE" >>>>>>>>>>> , >>>>>>>>>>> "cassandra.cf.name" = "processed_logs" , >>>>>>>>>>> "cassandra.columns.mapping" = >>>>>>>>>>> ":key,remote_ip, request_date, request,httpCode,length, version" >>>>>>>>>>> ); >>>>>>>>>>> >>>>>>>>>>> What I understand about this query is that it creates a table >>>>>>>>>>> and load data from "processed_logs" column family which is >>>>>>>>>>> available at >>>>>>>>>>> Cassandra Keyspaces / EVENT_KS. >>>>>>>>>>> >>>>>>>>>> In this query it created the "processed_logs" column family and >>>>>>>>>> creates the Hive table (which is a virtual table), "ProcessedLogs" >>>>>>>>>> wrapping >>>>>>>>>> the column family, "processed_logs". >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I need to analyze a log file which is in a format different from >>>>>>>>>>> access.log given in sample. How can I create a different column >>>>>>>>>>> family for >>>>>>>>>>> my requirement and access them in my script??? >>>>>>>>>>> >>>>>>>>>> The script above can be used for creating a column family. >>>>>>>>>> >>>>>>>>>> Read [1] for more information on writing Hive queries. >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> http://docs.wso2.org/display/BAM240/Creating+Hive+Queries+to+Analyze+Data >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thank You! >>>>>>>>>>> -- >>>>>>>>>>> *Chamila Wijayarathna* >>>>>>>>>>> Engineering Intern, >>>>>>>>>>> WSO2 Inc. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Dev mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Chamila Wijayarathna* >>>>>>>>> Engineering Intern, >>>>>>>>> WSO2 Inc. >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Dev mailing list >>>>>>>>> [email protected] >>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> >>>>>>>> Inosh Goonewardena >>>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>>> Mobile: +94779966317 >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Chamila Wijayarathna* >>>>>> Engineering Intern, >>>>>> WSO2 Inc. >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> [email protected] >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> W.G. Gihan Anuruddha >>>>> Senior Software Engineer | WSO2, Inc. >>>>> M: +94772272595 >>>>> >>>> >>>> >>>> >>>> -- >>>> *Chamila Wijayarathna* >>>> Engineering Intern, >>>> WSO2 Inc. >>>> >>>> >>> >>> >>> -- >>> W.G. Gihan Anuruddha >>> Senior Software Engineer | WSO2, Inc. >>> M: +94772272595 >>> >> >> >> >> -- >> *Chamila Wijayarathna* >> Engineering Intern, >> WSO2 Inc. >> >> > > > -- > W.G. Gihan Anuruddha > Senior Software Engineer | WSO2, Inc. > M: +94772272595 > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Regards, Inosh Goonewardena Associate Technical Lead- WSO2 Inc. Mobile: +94779966317
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
