Re: ELB Log processing

2016-09-20 Thread Manish Rangari
Thanks Dudu, both the queries worked like a charm. I personally liked second query as it is quite easy to remember. --Manish On Tue, Sep 20, 2016 at 8:41 PM, Markovitz, Dudu wrote: > Or > > > > create view elb_raw_log_detailed > > as > > select request_date, elbname,

Re: need help with hive session getting killed

2016-09-20 Thread Anup
I am doing insert overwrite so results of query are getting written to an external table stored in parquet. Which the properties should I set and Can I set those in query ? Thanks > On Sep 20, 2016, at 8:27 AM, Jörn Franke wrote: > > Increase timeout or let the

Re: need help with hive session getting killed

2016-09-20 Thread Jörn Franke
Increase timeout or let the result of the query be written in a dedicated table. > On 20 Sep 2016, at 16:57, anup ahire wrote: > > > > > Hello, > > I am using hive-jdbc-1.2.1 to run a query. Query runs around an hour and > eventually completes. > But my hive session

RE: ELB Log processing

2016-09-20 Thread Markovitz, Dudu
Or create view elb_raw_log_detailed as select request_date, elbname, requestip, requestport, backendip, backendport, requestprocessingtime, backendprocessingtime, clientresponsetime, elbresponsecode, backendresponsecode, receivedbytes, sentbytes, requestverb, url, parse_url(url, 'QUERY','aid')

RE: ELB Log processing

2016-09-20 Thread Markovitz, Dudu
create view elb_raw_log_detailed as select request_date, elbname, requestip, requestport, backendip, backendport, requestprocessingtime, backendprocessingtime, clientresponsetime, elbresponsecode, backendresponsecode, receivedbytes, sentbytes, requestverb, url, u.aid, u.tid, u.eid,u.did,

Re: ELB Log processing

2016-09-20 Thread Manish Rangari
Guys, I am struggling to create this view. I am keep getting the error in bold. I found that I need to use lateral view but still I am not able to get the syntax right. hive> create view elb_raw_log_detailed as select request_date, elbname, requestip, requestport, backendip, backendport,

Re: ELB Log processing

2016-09-20 Thread Manish Rangari
Yes views looks like a way to go On Tue, Sep 20, 2016 at 3:49 PM, Damien Carol wrote: > The royal way to do that is a view IMHO. > > 2016-09-20 12:14 GMT+02:00 Manish Rangari > : > >> Thanks for the reply Damien. The suggestion you gave

Re: ELB Log processing

2016-09-20 Thread Damien Carol
The royal way to do that is a view IMHO. 2016-09-20 12:14 GMT+02:00 Manish Rangari : > Thanks for the reply Damien. The suggestion you gave is really useful. > Currently I am achieving my desired output by performing below steps. But I > want to achieve the

Re: ELB Log processing

2016-09-20 Thread Manish Rangari
Thanks for the reply Damien. The suggestion you gave is really useful. Currently I am achieving my desired output by performing below steps. But I want to achieve the desired result in one step instead of two. Do we have any way so that I can get the aid, did etc in create table statement? If not

Re: ELB Log processing

2016-09-20 Thread Damien Carol
see the udf *parse_url_tuple* SELECT b.* FROM src LATERAL VIEW parse_url_tuple(fullurl, 'HOST', 'PATH', 'QUERY', 'QUERY:id') b as host, path, query, query_id LIMIT 1; https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-parse_url_tuple 2016-09-20 11:22 GMT+02:00

ELB Log processing

2016-09-20 Thread Manish Rangari
Guys, I want to get the field of elb logs. A sample elb log is given below and I am using below create table definition. It is working fine. I am getting what I wanted but now I want the bold part as well. For example eid, tid, aid. Can anyone help me how can I match them as well. NOTE: The