date:20161123

Working Hive--> Spark --> HDFS

2016-11-23 Thread Joaquin Alzola

Hi Guys

Can somebody tell me a workin version of HoSoHDFS.

So far I have tested:
Hive1.2.1-->Spark-->1.6.3--> Hadoop 2.6

Hive 2.1 --> Spark2.0.2 --> Hadoop 2.7

And both of them give me varios exceptions.
I have to say the first one creates the job in HDFS and finish it successfully 
but give back an error on spark.

BR

Joaquin

This email is confidential and may be subject to privilege. If you are not the 
intended recipient, please do not copy or disclose its content but contact the 
sender immediately upon receipt.

Need advice on tracing errors with batch mode

2016-11-23 Thread neelima g

Hi,

I want to run batch mode task like and i am using *hiveconf
hive.cli.errors.ignore=true*


hive -f file.txt hiveconf hive.cli.errors.ignore=true



And my requirement is find what all queries failed after execution. Is that
possible ?

Neelima

Re: Need advice on tracing errors with batch mode

2016-11-23 Thread neelima g

Hi,

I want to run batch mode task like and i am using *hiveconf
hive.cli.errors.ignore=true*

hive -f file.txt hiveconf hive.cli.errors.ignore=true

And my requirement is find what all queries failed after execution. Is that
possible ? And i have create statements in the queries..it is DDL.

Neelima

On Wed, Nov 23, 2016 at 3:17 PM, neelima g  wrote:

> Hi,
>
> I want to run batch mode task like and i am using *hiveconf
> hive.cli.errors.ignore=true*
>
>
> hive -f file.txt hiveconf hive.cli.errors.ignore=true
>
>
>
> And my requirement is find what all queries failed after execution. Is
> that possible ?
>
> Neelima
>

Re: Working Hive--> Spark --> HDFS

2016-11-23 Thread Divya Gehlot

can you please share the stacktrace of excpetion you get .


Thanks,
Divya

On 24 November 2016 at 06:33, Joaquin Alzola 
wrote:

> Hi Guys
>
>
>
> Can somebody tell me a workin version of HoSoHDFS.
>
>
>
> So far I have tested:
>
> Hive1.2.1àSparkà1.6.3à Hadoop 2.6
>
>
>
> Hive 2.1 à Spark2.0.2 à Hadoop 2.7
>
>
>
> And both of them give me varios exceptions.
>
> I have to say the first one creates the job in HDFS and finish it
> successfully but give back an error on spark.
>
>
>
> BR
>
>
>
> Joaquin
>
>
> This email is confidential and may be subject to privilege. If you are not
> the intended recipient, please do not copy or disclose its content but
> contact the sender immediately upon receipt.
>

Re: hive union all

2016-11-23 Thread 徐鹏

could you please explain the sql
在 2016年11月24日，下午1:01，C R 
> 写道：

hi all,
all I want to say is here:





it is ok.





I feel a little embarrassed





According to my test,the same hive ql will be executed well in CDH version.

RE: import sql file

2016-11-23 Thread Markovitz, Dudu

Hi Patcharee 
The question is not clear.

Dudu

-Original Message-
From: patcharee [mailto:patcharee.thong...@uni.no] 
Sent: Wednesday, November 23, 2016 11:37 AM
To: user@hive.apache.org
Subject: import sql file

Hi,

How can I import .sql file into hive?

Best, Patcharee

Re: import sql file

2016-11-23 Thread Mich Talebzadeh

put the file into an hdfs directory.

assuming to be some type of csv file from mssql or oracle, create an
external table in hive looking at the hdfs directory where you put the sql
file using LOCATION in Hive.

there are multiple examples if you google it. It is pretty straight forward.

alternatively use Sqoop to read the RDBMS table and create and import data
into Hive table. you need the JAR file for the relevevant RDBMS

HTH

Dr Mich Talebzadeh

LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*

http://talebzadehmich.wordpress.com

*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

On 23 November 2016 at 09:45, patcharee  wrote:

> I exported sql table into .sql file and would like to import this into hive
>
> Best, Patcharee
>
>
> On 23. nov. 2016 10:40, Markovitz, Dudu wrote:
>
>> Hi Patcharee
>> The question is not clear.
>>
>> Dudu
>>
>> -Original Message-
>> From: patcharee [mailto:patcharee.thong...@uni.no]
>> Sent: Wednesday, November 23, 2016 11:37 AM
>> To: user@hive.apache.org
>> Subject: import sql file
>>
>> Hi,
>>
>> How can I import .sql file into hive?
>>
>> Best, Patcharee
>>
>>
>

Re: Hive Serialization issues

2016-11-23 Thread Dana Ram Meghwal

Hey,
Any leads?

On Tue, Nov 22, 2016 at 5:35 PM, Dana Ram Meghwal  wrote:

> Hey All,
>
> I am using Hive 2.0 with external meta-store on EMR-5.0.0 and TEZ as
> execution engine.
> Our data are stored in json format so for serialization and
> deserialization purpose we are planning to use lazy serde
> (classname is  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' ).
>
> My table definition is
>
> CREATE EXTERNAL TABLE IF NOT EXISTS 
> daily_active_users_summary_json_partition_dt_paths_v1
> (uid string, city string, user string, songcount string, songid_list
> array  ) PARTITIONED BY ( dt string)
>
>  ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
>
>  WITH SERDEPROPERTIES ('paths'='uid,city,user,songcount,songid_list')
>
>  LOCATION 's3:///users/daily_active_
> users_summary_json_partition_dt';
>
>
> and data look like this---
>
> {"uid":"xx","listening_user_flag":"non_listening","platform":"android","model":"micromax
> a110q","aquisition_channel":"organic","state":"delhi","app_
> version":"3.2:","country":"IN","city":"new delhi","new_listening_user_
> flag":"non_listening","manufacturer":"Micromax","
> login_mode":"loggedout","new_user_flag":"returning","digital_channel":"Not
> Source"}
>
>
> Note: I have pasted here one record in table.
>
>
> Now, When I do query
>
> select * from daily_active_users_summary_json_partition_dt_paths_v1 limit
> 5;
>
>
> the first field of table takes the complete record and rest of field are
> showing to be NULL.
>
> When I use different serde  'org.apache.hive.hcatalog.data.JsonSerDe'
>
> then I can see the above query works fine and able to serialize data
> perfectly fine. We want to user the lazy serde because our data contains
> non-utf-8 character and the later serde does not support non-utf-8
> character serialization/deserialization.
>
>
> Can you please help me solve this, we mostly want to use lazy serde only
> as we have already experimented with other serde's none of them is working
> for us Is there any configuration which enable
> serialization/deserialization while using lazy Serde.
>
> Or is there any other serde which can fine process non-utf-8 character in
> hive-2 and tez.
>
> Thank you
>
>
> Best Regards,
> Dana Ram Meghwal
> Software Engineer
> dana...@saavn.com
>
>


-- 
Dana Ram Meghwal
Software Engineer
dana...@saavn.com

import sql file

2016-11-23 Thread patcharee


Hi,

How can I import .sql file into hive?

Best, Patcharee

Re: import sql file

2016-11-23 Thread patcharee


I exported sql table into .sql file and would like to import this into hive

Best, Patcharee

On 23. nov. 2016 10:40, Markovitz, Dudu wrote:

Hi Patcharee
The question is not clear.

Dudu

-Original Message-
From: patcharee [mailto:patcharee.thong...@uni.no]
Sent: Wednesday, November 23, 2016 11:37 AM
To: user@hive.apache.org
Subject: import sql file

Hi,

How can I import .sql file into hive?

Best, Patcharee

Interrogating a uniontype

2016-11-23 Thread Elliot West

Can anyone recommend a good approach for interrogating uniontype values in
HQL? I note that the documentation states that the support for such types
is limited to 'look-at-only' which I assume to mean that I may only dump
out the value in its entirety, and extract sub-elements. Using the example
below, is there anyway I can get to field 'b' of union index 3 to extract
only the value 5?

{0:1}
{1:2.0}
{2:["three","four"]}
{3:{"a":5,"b":"five"}}


If not possible with HQL, would it be possible to implement a UDF that can
explode the type into something more navigable, like an array, struct, or
map?

Example when exploded as array:


[1,null,null,null]
[null,2.0,null,null]
[null,null,["three","four"],null]
[null,null,null,{"a":5,"b":"five"}]

Has anyone done this?

Thanks,

Elliot.

Re: Interrogating a uniontype

2016-11-23 Thread Elliot West

Ah, I see that this can't be done with an array as there is no type common
to all union indexes. Perhaps a struct with one field per indexed type?

On Wed, 23 Nov 2016 at 17:29, Elliot West  wrote:

> Can anyone recommend a good approach for interrogating uniontype values in
> HQL? I note that the documentation states that the support for such types
> is limited to 'look-at-only' which I assume to mean that I may only dump
> out the value in its entirety, and extract sub-elements. Using the example
> below, is there anyway I can get to field 'b' of union index 3 to extract
> only the value 5?
>
> {0:1}
> {1:2.0}
> {2:["three","four"]}
> {3:{"a":5,"b":"five"}}
>
>
> If not possible with HQL, would it be possible to implement a UDF that can
> explode the type into something more navigable, like an array, struct, or
> map?
>
> Example when exploded as array:
>
>
> [1,null,null,null]
> [null,2.0,null,null]
> [null,null,["three","four"],null]
> [null,null,null,{"a":5,"b":"five"}]
>
> Has anyone done this?
>
> Thanks,
>
> Elliot.
>
>

Re: Hive Serialization issues

2016-11-23 Thread Edward Capriolo

I believe json itself has encoding rules. What i suggest you do is build
your own input format or serde and escape those fieds possibly by
converting them to hex.

On Wednesday, November 23, 2016, Dana Ram Meghwal  wrote:

> Hey,
> Any leads?
>
> On Tue, Nov 22, 2016 at 5:35 PM, Dana Ram Meghwal  > wrote:
>
>> Hey All,
>>
>> I am using Hive 2.0 with external meta-store on EMR-5.0.0 and TEZ as
>> execution engine.
>> Our data are stored in json format so for serialization and
>> deserialization purpose we are planning to use lazy serde
>> (classname is  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' ).
>>
>> My table definition is
>>
>> CREATE EXTERNAL TABLE IF NOT EXISTS 
>> daily_active_users_summary_json_partition_dt_paths_v1
>> (uid string, city string, user string, songcount string, songid_list
>> array  ) PARTITIONED BY ( dt string)
>>
>>  ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
>>
>>  WITH SERDEPROPERTIES ('paths'='uid,city,user,songcount,songid_list')
>>
>>  LOCATION 's3:///users/daily_active_us
>> ers_summary_json_partition_dt';
>>
>>
>> and data look like this---
>>
>> {"uid":"xx","listening_user_flag":"non_listening","
>> platform":"android","model":"micromax a110q","aquisition_channel":"o
>> rganic","state":"delhi","app_version":"3.2:","country":"IN","city":"new
>> delhi","new_listening_user_flag":"non_listening","manufactur
>> er":"Micromax","login_mode":"loggedout","new_user_flag":"
>> returning","digital_channel":"Not Source"}
>>
>>
>> Note: I have pasted here one record in table.
>>
>>
>> Now, When I do query
>>
>> select * from daily_active_users_summary_json_partition_dt_paths_v1
>> limit 5;
>>
>>
>> the first field of table takes the complete record and rest of field are
>> showing to be NULL.
>>
>> When I use different serde  'org.apache.hive.hcatalog.data.JsonSerDe'
>>
>> then I can see the above query works fine and able to serialize data
>> perfectly fine. We want to user the lazy serde because our data contains
>> non-utf-8 character and the later serde does not support non-utf-8
>> character serialization/deserialization.
>>
>>
>> Can you please help me solve this, we mostly want to use lazy serde only
>> as we have already experimented with other serde's none of them is working
>> for us Is there any configuration which enable
>> serialization/deserialization while using lazy Serde.
>>
>> Or is there any other serde which can fine process non-utf-8 character in
>> hive-2 and tez.
>>
>> Thank you
>>
>>
>> Best Regards,
>> Dana Ram Meghwal
>> Software Engineer
>> dana...@saavn.com 
>>
>>
>
>
> --
> Dana Ram Meghwal
> Software Engineer
> dana...@saavn.com 
>
>

-- 
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.

Re: Interrogating a uniontype

2016-11-23 Thread Per Ullberg

Could you write a UDF that parses it and returns a json object. From there
you can use the standard json support in Hive. I did something similar for
Erlang structs about 3 years ago. I actually kept them on file and wrote a
serde that exposed them as json objects.

regards
/Pelle

On Wed, Nov 23, 2016 at 6:40 PM, Elliot West  wrote:

> Ah, I see that this can't be done with an array as there is no type common
> to all union indexes. Perhaps a struct with one field per indexed type?
>
> On Wed, 23 Nov 2016 at 17:29, Elliot West  wrote:
>
>> Can anyone recommend a good approach for interrogating uniontype values
>> in HQL? I note that the documentation states that the support for such
>> types is limited to 'look-at-only' which I assume to mean that I may only
>> dump out the value in its entirety, and extract sub-elements. Using the
>> example below, is there anyway I can get to field 'b' of union index 3 to
>> extract only the value 5?
>>
>> {0:1}
>> {1:2.0}
>> {2:["three","four"]}
>> {3:{"a":5,"b":"five"}}
>>
>>
>> If not possible with HQL, would it be possible to implement a UDF that
>> can explode the type into something more navigable, like an array, struct,
>> or map?
>>
>> Example when exploded as array:
>>
>>
>> [1,null,null,null]
>> [null,2.0,null,null]
>> [null,null,["three","four"],null]
>> [null,null,null,{"a":5,"b":"five"}]
>>
>> Has anyone done this?
>>
>> Thanks,
>>
>> Elliot.
>>
>>


-- 

*Per Ullberg*
Data Vault Tech Lead
Odin Uppsala
+46 701612693 <+46+701612693>

Klarna AB (publ)
Sveavägen 46, 111 34 Stockholm
Tel: +46 8 120 120 00 <+46812012000>
Reg no: 556737-0431
klarna.com

Working Hive--> Spark --> HDFS

Need advice on tracing errors with batch mode

Re: Need advice on tracing errors with batch mode

Re: Working Hive--> Spark --> HDFS

Re: hive union all

RE: import sql file

Re: import sql file

Re: Hive Serialization issues

import sql file

Re: import sql file

Interrogating a uniontype

Re: Interrogating a uniontype

Re: Hive Serialization issues

Re: Interrogating a uniontype

14 matches

Site Navigation

Mail list logo

Footer information