Hi, I got that, I tested with dropping other attribute and it worked. Thank you!
On Mon, Dec 23, 2013 at 12:12 PM, Inosh Goonewardena <[email protected]> wrote: > Hi Chamila, > > You can't ignore the ':key' column of Cassandra CF when you map it to a > hive table. Drop all the meta tables and try with mapping :key column to a > column in hive table. > > > > On Mon, Dec 23, 2013 at 11:26 AM, Gihan Anuruddha <[email protected]> wrote: > >> Hi Chamila, >> >> First to create a column family you have to create a stream definition. >> To do that you have to make some small change in the sample. >> >> Below code snippet will create stream definition in the sample. >> >> streamId = dataPublisher.defineStream("{" + >> " 'name':'" +* HTTPD_LOG_STREAM* + "'," + >> " 'version':'" + VERSION + "'," + >> " 'nickName': 'Httpd_Log_Stream'," + >> " 'description': 'Sample of Httpd logs'," + >> " 'metaData':[" + >> " {'name':'clientType','type':'STRING'}" + >> " ]," + >> " 'payloadData':[" + >> " {'name':'log','type':'STRING'}" + >> " ]" + >> "}"); >> >> >> In your case change *HTTPD_LOG_STREAM *to chamila. So again when you >> execute the sample it will create new column family called chamila. Then >> try again your script. If doesn't work please share your analytics script >> with us. >> >> Regards, >> Gihan >> >> >> On Mon, Dec 23, 2013 at 11:00 AM, Chamila Wijayarathna >> <[email protected]>wrote: >> >>> I executed both queries in that way, but it still gives the same error >>> it gave earlier. >>> >>> >>> On Mon, Dec 23, 2013 at 10:55 AM, Gihan Anuruddha <[email protected]>wrote: >>> >>>> Sorry, I forgot to tell you that above (Drop table Chamila) query you >>>> have to execute through the hive query editor. >>>> >>>> To do that: start BAM and go to management console. Then go to Manage >>>> -> Analytics -> Add and enter "Drop table Chamila" and execute. >>>> >>>> >>>> >>>> On Mon, Dec 23, 2013 at 10:49 AM, Chamila Wijayarathna < >>>> [email protected]> wrote: >>>> >>>>> Hi, >>>>> It still gives the same result. Also "SHOW TABLES", doesn't show that >>>>> there is table "Chamila" existing. >>>>> >>>>> >>>>> On Mon, Dec 23, 2013 at 10:29 AM, Gihan Anuruddha <[email protected]>wrote: >>>>> >>>>>> Hi Chamila, >>>>>> >>>>>> Execute this query before you execute above one "Drop table Chamila" >>>>>> . Basically hive maintain meta information when it creates a new virtual >>>>>> table. Once you create a table it won't change again unless you drop that >>>>>> virtual table and create a new one with edited parameters. >>>>>> >>>>>> Regards, >>>>>> Gihan >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Dec 23, 2013 at 9:51 AM, Chamila Wijayarathna < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hii all, >>>>>>> How can I create a column family, so that it will suite my need? I >>>>>>> tried >>>>>>> >>>>>>> CREATE EXTERNAL TABLE IF NOT EXISTS Chamila ( >>>>>>> remote_ip STRING, >>>>>>> request_date STRING,request STRING, >>>>>>> httpCode STRING,length STRING, >>>>>>> version STRING) >>>>>>> STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' >>>>>>> WITH SERDEPROPERTIES ( >>>>>>> "wso2.carbon.datasource.name" = "WSO2BAM_CASSANDRA_DATASOURCE" , >>>>>>> "cassandra.cf.name" = "chamila" , >>>>>>> "cassandra.columns.mapping" = >>>>>>> *"*remote_ip, request_date, request,httpCode,length, version" ); >>>>>>> >>>>>>> So now I have all columns in column family has mapped to my hive >>>>>>> table, but still I am getting same error[1]. >>>>>>> >>>>>>> Do I have to configure column family somewhere else before running >>>>>>> this code? If yes, how can I do that? >>>>>>> >>>>>>> 1. Backend Console Log - >>>>>>> https://gist.github.com/cdwijayarathna/8091591 >>>>>>> >>>>>>> Thank You! >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Sun, Dec 22, 2013 at 1:00 PM, Maninda Edirisooriya < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Chamila, >>>>>>>> >>>>>>>> And also note that the error displayed on the UI when a Hive script >>>>>>>> is executed is not enough to figure out the problem. You need to look >>>>>>>> at >>>>>>>> the backend console log as well. >>>>>>>> Anyway for the given problem Inosh's correction will work. >>>>>>>> >>>>>>>> >>>>>>>> *Maninda Edirisooriya* >>>>>>>> Software Engineer >>>>>>>> >>>>>>>> *WSO2, Inc. *lean.enterprise.middleware. >>>>>>>> >>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>> *Phone* : +94 777603226 >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Dec 21, 2013 at 11:27 PM, Inosh Goonewardena < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> On Sat, Dec 21, 2013 at 10:25 PM, Chamila Wijayarathna < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hii Maninda, >>>>>>>>>> Thanks for replying. >>>>>>>>>> I tried creating a table which will suite my need, with different >>>>>>>>>> set of columns from above example. >>>>>>>>>> >>>>>>>>>> CREATE EXTERNAL TABLE IF NOT EXISTS Chamila ( >>>>>>>>>> remote_ip STRING, >>>>>>>>>> request_date STRING,request STRING, >>>>>>>>>> httpCode STRING,length STRING, >>>>>>>>>> version STRING) >>>>>>>>>> STORED BY >>>>>>>>>> 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' >>>>>>>>>> WITH SERDEPROPERTIES ( >>>>>>>>>> "wso2.carbon.datasource.name" = "WSO2BAM_CASSANDRA_DATASOURCE" , >>>>>>>>>> "cassandra.cf.name" = "processed_logs" , >>>>>>>>>> "cassandra.columns.mapping" = >>>>>>>>>> *":key*,remote_ip, request_date, request,httpCode,length, >>>>>>>>>> version" ); >>>>>>>>>> >>>>>>>>>> But I get an error saying, "Error while executing Hive >>>>>>>>>> script.Query returned non-zero code: 9, cause: FAILED: Execution >>>>>>>>>> Error, >>>>>>>>>> return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask". >>>>>>>>>> >>>>>>>>> >>>>>>>>> Seems like ':key' column has not been mapped to any column of hive >>>>>>>>> your table. For example, in original query it is mapped to logId. >>>>>>>>> >>>>>>>>> How can I create a table to suite what I want. As I understood, it >>>>>>>>>> has no meaning to create a virtual table without linking it with >>>>>>>>>> Column >>>>>>>>>> Family. >>>>>>>>>> >>>>>>>>>> Thank you! >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Dec 20, 2013 at 5:58 PM, Maninda Edirisooriya < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> On Fri, Dec 20, 2013 at 4:54 PM, Chamila Wijayarathna < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hello All, >>>>>>>>>>>> I am running BAM sample 1. >>>>>>>>>>>> In the analytic script which get run when I sent log line event >>>>>>>>>>>> it has query >>>>>>>>>>>> >>>>>>>>>>>> CREATE EXTERNAL TABLE IF NOT EXISTS ProcessedLogs ( >>>>>>>>>>>> logId STRING,remote_ip STRING, >>>>>>>>>>>> request_date STRING,request STRING, >>>>>>>>>>>> httpCode STRING,length STRING, >>>>>>>>>>>> version STRING) >>>>>>>>>>>> STORED BY >>>>>>>>>>>> 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' >>>>>>>>>>>> WITH SERDEPROPERTIES ( >>>>>>>>>>>> "wso2.carbon.datasource.name" = >>>>>>>>>>>> "WSO2BAM_CASSANDRA_DATASOURCE" , >>>>>>>>>>>> "cassandra.cf.name" = "processed_logs" , >>>>>>>>>>>> "cassandra.columns.mapping" = >>>>>>>>>>>> ":key,remote_ip, request_date, request,httpCode,length, >>>>>>>>>>>> version" ); >>>>>>>>>>>> >>>>>>>>>>>> What I understand about this query is that it creates a table >>>>>>>>>>>> and load data from "processed_logs" column family which is >>>>>>>>>>>> available at >>>>>>>>>>>> Cassandra Keyspaces / EVENT_KS. >>>>>>>>>>>> >>>>>>>>>>> In this query it created the "processed_logs" column family and >>>>>>>>>>> creates the Hive table (which is a virtual table), "ProcessedLogs" >>>>>>>>>>> wrapping >>>>>>>>>>> the column family, "processed_logs". >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I need to analyze a log file which is in a format different >>>>>>>>>>>> from access.log given in sample. How can I create a different >>>>>>>>>>>> column family >>>>>>>>>>>> for my requirement and access them in my script??? >>>>>>>>>>>> >>>>>>>>>>> The script above can be used for creating a column family. >>>>>>>>>>> >>>>>>>>>>> Read [1] for more information on writing Hive queries. >>>>>>>>>>> >>>>>>>>>>> [1] >>>>>>>>>>> http://docs.wso2.org/display/BAM240/Creating+Hive+Queries+to+Analyze+Data >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thank You! >>>>>>>>>>>> -- >>>>>>>>>>>> *Chamila Wijayarathna* >>>>>>>>>>>> Engineering Intern, >>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Dev mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Chamila Wijayarathna* >>>>>>>>>> Engineering Intern, >>>>>>>>>> WSO2 Inc. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Dev mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Inosh Goonewardena >>>>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>>>> Mobile: +94779966317 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Chamila Wijayarathna* >>>>>>> Engineering Intern, >>>>>>> WSO2 Inc. >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> [email protected] >>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> W.G. Gihan Anuruddha >>>>>> Senior Software Engineer | WSO2, Inc. >>>>>> M: +94772272595 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Chamila Wijayarathna* >>>>> Engineering Intern, >>>>> WSO2 Inc. >>>>> >>>>> >>>> >>>> >>>> -- >>>> W.G. Gihan Anuruddha >>>> Senior Software Engineer | WSO2, Inc. >>>> M: +94772272595 >>>> >>> >>> >>> >>> -- >>> *Chamila Wijayarathna* >>> Engineering Intern, >>> WSO2 Inc. >>> >>> >> >> >> -- >> W.G. Gihan Anuruddha >> Senior Software Engineer | WSO2, Inc. >> M: +94772272595 >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Regards, > > Inosh Goonewardena > Associate Technical Lead- WSO2 Inc. > Mobile: +94779966317 > -- *Chamila Wijayarathna* Engineering Intern, WSO2 Inc.
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
