Re: how to handle variable format data of text file?

2013-03-18 Thread Zhiwen Sun
As u defined in create table hql: fields delimited by blank space. So, the other data is omitted if you wanna contain rest data at the end of line. I suggest you use org.apache.hadoop.hive.contrib.serde2.RegexSerDe row format instead of default delimited format. Zhiwen Sun On Mon, Mar 11,

dfs storage full on all slave machines of 6 machine hive cluster

2013-03-18 Thread Chunky Gupta
Hi, We have a 6 machine hive cluster. We are getting errors while a query is running and it fails. I found that on all 5 slaves storage is nearly full ( 96%, 98%, 100%, 97%, 98% storage used) . On my slaves machines, this folder /mnt/hadoop-fs/dfs/data/current/ is contributing 95% storage used.

Re: dfs storage full on all slave machines of 6 machine hive cluster

2013-03-18 Thread Manish Bhoge
I think these directories belong to task tracker temporary storage. I am not very confident to conclude that go ahead with your clean up. So, wait for similar or an expert's response  Sent from HTC via Rocket! excuse typo.

Re: dfs storage full on all slave machines of 6 machine hive cluster

2013-03-18 Thread Zhiwen Sun
The folder /mnt/hadoop-fs/dfs/data/current/ is the main folder of datanode in hadoop. You can use *hadoop dfs -rmr {nouserdir} *to get more free space in HDFS. *Don't delete file directly in OS file system.* Zhiwen Sun On Mon, Mar 18, 2013 at 6:48 PM, Manish Bhoge

Re: how to handle variable format data of text file?

2013-03-18 Thread Ramki Palle
One way you can try is to make your ldata as a map field as it contains variable formatted data and write a UDF to get whatever information you need get. Regards, Ramki. On Mon, Mar 18, 2013 at 1:23 AM, Zhiwen Sun pens...@gmail.com wrote: As u defined in create table hql: fields delimited by

Re: dfs storage full on all slave machines of 6 machine hive cluster

2013-03-18 Thread Chunky Gupta
Hi Zhiwen, /mnt/hadoop-fs/mapred/local/taskTracker/ Inside this folder there are folders with different user name, can I delete these ?. I do not understand what this {*nouserdir*} you were talking about, can you please explain ?. Thanks, Chunky. On Mon, Mar 18, 2013 at 8:40 PM, Zhiwen Sun

Re: hive-0.10 and nullpointerexception

2013-03-18 Thread Mark Grover
For the sake of readers, I just saw another manifestation of this problem. The issue was that mapreduce.framework.name in mapred-site.xml was set to yarn instead of classic when the intent was to use MR1. Hope that helps, Mark On Mon, Mar 4, 2013 at 8:26 AM, Mark Grover

Use Hive reflect() method to call non-static JDK fuctions

2013-03-18 Thread David Lee
It's relatively straight forward to call static functions in JDK using reflect. For example, select reflect(java.lang.Math, max, 2, 3) from mytable limit 1; However, how do I use reflect to call non-static functions (e.g., indexOf() method in java.lang.String class)? None of the following

Re: Use Hive reflect() method to call non-static JDK fuctions

2013-03-18 Thread Navis류승우
It's implemented in trunk, named as reflect2. https://issues.apache.org/jira/browse/HIVE-4025 2013/3/19 David Lee seek...@gmail.com: It's relatively straight forward to call static functions in JDK using reflect. For example, select reflect(java.lang.Math, max, 2, 3) from mytable limit 1;

Important information required about Hive metastore connections

2013-03-18 Thread Siddharth Tiwari
Hi team, What we have observed is, hive creates so many connections to metastore (mysql). This makes us change the numer of connections to mysql. Can anyone explain me how does hive manages its connections to metastore database. is it one for every Job submitted by every user ? if not then how

Re: Important information required about Hive metastore connections

2013-03-18 Thread Navis류승우
Hive uses DBCP as a connection pool. Ref http://www.datanucleus.org/products/accessplatform_2_2/rdbms/dbcp.html 2013/3/19 Siddharth Tiwari siddharth.tiw...@live.com: Hi team, What we have observed is, hive creates so many connections to metastore (mysql). This makes us change the numer of

Does Hive support collation?

2013-03-18 Thread Jon Klein
Hi, I'm using Hive for dealing with some international characters.Does Hive have collation support so I can specify case sensitivity, ascent sensitivity or width sensitity for string comparison purposes or in group-by clause? Thanks,Jon

Re: Closing long running hive query forcefully

2013-03-18 Thread wzc
' select /* myid bla bla*/ x,y,z ' I can't run above command using cli nor hive -f, could you explain how to add comment in hive query? 2013/2/8 Edward Capriolo edlinuxg...@gmail.com That is a good way to do it. We do it with comment sometimes. select /* myid bla bla*/ x,y,z Edward On