Hi Chamila,
First to create a column family you have to create a stream definition. To
do that you have to make some small change in the sample.
Below code snippet will create stream definition in the sample.
streamId = dataPublisher.defineStream("{" +
" 'name':'" +* HTTPD_LOG_STREAM* + "'," +
" 'version':'" + VERSION + "'," +
" 'nickName': 'Httpd_Log_Stream'," +
" 'description': 'Sample of Httpd logs'," +
" 'metaData':[" +
" {'name':'clientType','type':'STRING'}" +
" ]," +
" 'payloadData':[" +
" {'name':'log','type':'STRING'}" +
" ]" +
"}");
In your case change *HTTPD_LOG_STREAM *to chamila. So again when you
execute the sample it will create new column family called chamila. Then
try again your script. If doesn't work please share your analytics script
with us.
Regards,
Gihan
On Mon, Dec 23, 2013 at 11:00 AM, Chamila Wijayarathna <[email protected]>wrote:
> I executed both queries in that way, but it still gives the same error it
> gave earlier.
>
>
> On Mon, Dec 23, 2013 at 10:55 AM, Gihan Anuruddha <[email protected]> wrote:
>
>> Sorry, I forgot to tell you that above (Drop table Chamila) query you
>> have to execute through the hive query editor.
>>
>> To do that: start BAM and go to management console. Then go to Manage ->
>> Analytics -> Add and enter "Drop table Chamila" and execute.
>>
>>
>>
>> On Mon, Dec 23, 2013 at 10:49 AM, Chamila Wijayarathna
>> <[email protected]>wrote:
>>
>>> Hi,
>>> It still gives the same result. Also "SHOW TABLES", doesn't show that
>>> there is table "Chamila" existing.
>>>
>>>
>>> On Mon, Dec 23, 2013 at 10:29 AM, Gihan Anuruddha <[email protected]>wrote:
>>>
>>>> Hi Chamila,
>>>>
>>>> Execute this query before you execute above one "Drop table Chamila" .
>>>> Basically hive maintain meta information when it creates a new virtual
>>>> table. Once you create a table it won't change again unless you drop that
>>>> virtual table and create a new one with edited parameters.
>>>>
>>>> Regards,
>>>> Gihan
>>>>
>>>>
>>>>
>>>> On Mon, Dec 23, 2013 at 9:51 AM, Chamila Wijayarathna <[email protected]
>>>> > wrote:
>>>>
>>>>> Hii all,
>>>>> How can I create a column family, so that it will suite my need? I
>>>>> tried
>>>>>
>>>>> CREATE EXTERNAL TABLE IF NOT EXISTS Chamila (
>>>>> remote_ip STRING,
>>>>> request_date STRING,request STRING,
>>>>> httpCode STRING,length STRING,
>>>>> version STRING)
>>>>> STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler'
>>>>> WITH SERDEPROPERTIES (
>>>>> "wso2.carbon.datasource.name" = "WSO2BAM_CASSANDRA_DATASOURCE" ,
>>>>> "cassandra.cf.name" = "chamila" ,
>>>>> "cassandra.columns.mapping" =
>>>>> *"*remote_ip, request_date, request,httpCode,length, version" );
>>>>>
>>>>> So now I have all columns in column family has mapped to my hive
>>>>> table, but still I am getting same error[1].
>>>>>
>>>>> Do I have to configure column family somewhere else before running
>>>>> this code? If yes, how can I do that?
>>>>>
>>>>> 1. Backend Console Log -
>>>>> https://gist.github.com/cdwijayarathna/8091591
>>>>>
>>>>> Thank You!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Dec 22, 2013 at 1:00 PM, Maninda Edirisooriya <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi Chamila,
>>>>>>
>>>>>> And also note that the error displayed on the UI when a Hive script
>>>>>> is executed is not enough to figure out the problem. You need to look at
>>>>>> the backend console log as well.
>>>>>> Anyway for the given problem Inosh's correction will work.
>>>>>>
>>>>>>
>>>>>> *Maninda Edirisooriya*
>>>>>> Software Engineer
>>>>>>
>>>>>> *WSO2, Inc. *lean.enterprise.middleware.
>>>>>>
>>>>>> *Blog* : http://maninda.blogspot.com/
>>>>>> *Phone* : +94 777603226
>>>>>>
>>>>>>
>>>>>> On Sat, Dec 21, 2013 at 11:27 PM, Inosh Goonewardena
>>>>>> <[email protected]>wrote:
>>>>>>
>>>>>>> On Sat, Dec 21, 2013 at 10:25 PM, Chamila Wijayarathna <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hii Maninda,
>>>>>>>> Thanks for replying.
>>>>>>>> I tried creating a table which will suite my need, with different
>>>>>>>> set of columns from above example.
>>>>>>>>
>>>>>>>> CREATE EXTERNAL TABLE IF NOT EXISTS Chamila (
>>>>>>>> remote_ip STRING,
>>>>>>>> request_date STRING,request STRING,
>>>>>>>> httpCode STRING,length STRING,
>>>>>>>> version STRING)
>>>>>>>> STORED BY
>>>>>>>> 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler'
>>>>>>>> WITH SERDEPROPERTIES (
>>>>>>>> "wso2.carbon.datasource.name" = "WSO2BAM_CASSANDRA_DATASOURCE" ,
>>>>>>>> "cassandra.cf.name" = "processed_logs" ,
>>>>>>>> "cassandra.columns.mapping" =
>>>>>>>> *":key*,remote_ip, request_date, request,httpCode,length, version"
>>>>>>>> );
>>>>>>>>
>>>>>>>> But I get an error saying, "Error while executing Hive
>>>>>>>> script.Query returned non-zero code: 9, cause: FAILED: Execution Error,
>>>>>>>> return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask".
>>>>>>>>
>>>>>>>
>>>>>>> Seems like ':key' column has not been mapped to any column of hive
>>>>>>> your table. For example, in original query it is mapped to logId.
>>>>>>>
>>>>>>> How can I create a table to suite what I want. As I understood, it
>>>>>>>> has no meaning to create a virtual table without linking it with Column
>>>>>>>> Family.
>>>>>>>>
>>>>>>>> Thank you!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Dec 20, 2013 at 5:58 PM, Maninda Edirisooriya <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> On Fri, Dec 20, 2013 at 4:54 PM, Chamila Wijayarathna <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> Hello All,
>>>>>>>>>> I am running BAM sample 1.
>>>>>>>>>> In the analytic script which get run when I sent log line event
>>>>>>>>>> it has query
>>>>>>>>>>
>>>>>>>>>> CREATE EXTERNAL TABLE IF NOT EXISTS ProcessedLogs (
>>>>>>>>>> logId STRING,remote_ip STRING,
>>>>>>>>>> request_date STRING,request STRING,
>>>>>>>>>> httpCode STRING,length STRING,
>>>>>>>>>> version STRING)
>>>>>>>>>> STORED BY
>>>>>>>>>> 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler'
>>>>>>>>>> WITH SERDEPROPERTIES (
>>>>>>>>>> "wso2.carbon.datasource.name" = "WSO2BAM_CASSANDRA_DATASOURCE" ,
>>>>>>>>>> "cassandra.cf.name" = "processed_logs" ,
>>>>>>>>>> "cassandra.columns.mapping" =
>>>>>>>>>> ":key,remote_ip, request_date, request,httpCode,length, version"
>>>>>>>>>> );
>>>>>>>>>>
>>>>>>>>>> What I understand about this query is that it creates a table and
>>>>>>>>>> load data from "processed_logs" column family which is available at
>>>>>>>>>> Cassandra Keyspaces / EVENT_KS.
>>>>>>>>>>
>>>>>>>>> In this query it created the "processed_logs" column family and
>>>>>>>>> creates the Hive table (which is a virtual table), "ProcessedLogs"
>>>>>>>>> wrapping
>>>>>>>>> the column family, "processed_logs".
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I need to analyze a log file which is in a format different from
>>>>>>>>>> access.log given in sample. How can I create a different column
>>>>>>>>>> family for
>>>>>>>>>> my requirement and access them in my script???
>>>>>>>>>>
>>>>>>>>> The script above can be used for creating a column family.
>>>>>>>>>
>>>>>>>>> Read [1] for more information on writing Hive queries.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> http://docs.wso2.org/display/BAM240/Creating+Hive+Queries+to+Analyze+Data
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thank You!
>>>>>>>>>> --
>>>>>>>>>> *Chamila Wijayarathna*
>>>>>>>>>> Engineering Intern,
>>>>>>>>>> WSO2 Inc.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Dev mailing list
>>>>>>>>>> [email protected]
>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Chamila Wijayarathna*
>>>>>>>> Engineering Intern,
>>>>>>>> WSO2 Inc.
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Dev mailing list
>>>>>>>> [email protected]
>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>>
>>>>>>> Inosh Goonewardena
>>>>>>> Associate Technical Lead- WSO2 Inc.
>>>>>>> Mobile: +94779966317
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Chamila Wijayarathna*
>>>>> Engineering Intern,
>>>>> WSO2 Inc.
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Dev mailing list
>>>>> [email protected]
>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> W.G. Gihan Anuruddha
>>>> Senior Software Engineer | WSO2, Inc.
>>>> M: +94772272595
>>>>
>>>
>>>
>>>
>>> --
>>> *Chamila Wijayarathna*
>>> Engineering Intern,
>>> WSO2 Inc.
>>>
>>>
>>
>>
>> --
>> W.G. Gihan Anuruddha
>> Senior Software Engineer | WSO2, Inc.
>> M: +94772272595
>>
>
>
>
> --
> *Chamila Wijayarathna*
> Engineering Intern,
> WSO2 Inc.
>
>
--
W.G. Gihan Anuruddha
Senior Software Engineer | WSO2, Inc.
M: +94772272595
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
