Re: Hive select shows null after successful data load
Do you have any timestamp fields in the table that might contain null value
?
I faced a similar situation sometime back - changing the data type to
string made it work.
But I was working on delimited text files.
Not sure if it applies to JSON .. but its still worth giving a try !!
Richa
On Wed, Jun 19, 2013 at 7:28 AM, Sunita Arvind sunitarv...@gmail.comwrote:
Having the a column name same as the table name, is a problem due to which
I was not able to reference jobs.values.id from jobs. Changing the table
name to jobs1 resolved the semantic error.
However, the query still returns null
hive select jobs.values.position.title from jobs1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201306080116_0036, Tracking URL =
http://node01.expressanalytics.net:50030/jobdetails.jsp?jobid=job_201306080116_0036
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_201306080116_0036
Hadoop job information for Stage-1: number of mappers: 1; number of
reducers: 0
2013-06-18 18:55:52,381 Stage-1 map = 0%, reduce = 0%
2013-06-18 18:55:56,394 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
0.88 sec
2013-06-18 18:55:57,400 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
0.88 sec
2013-06-18 18:55:58,407 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
0.88 sec
MapReduce Total cumulative CPU time: 880 msec
Ended Job = job_201306080116_0036
MapReduce Jobs Launched:
Job 0: Map: 1 Cumulative CPU: 0.88 sec HDFS Read: 35374 HDFS Write: 3
SUCCESS
Total MapReduce CPU Time Spent: 880 msec
OK
null
Time taken: 9.591 seconds
regards
Sunita
On Tue, Jun 18, 2013 at 9:35 PM, Sunita Arvind sunitarv...@gmail.comwrote:
Ok.
The data files are quite small. Around 35 KB and 1 KB each.
[sunita@node01 tables]$ hadoop fs -ls /user/sunita/tables/jobs
Found 1 items
-rw-r--r-- 3 sunita hdfs 35172 2013-06-18 18:31
/user/sunita/tables/jobs/jobs_noSite_parsed.json
[sunita@node01 tables]$ hadoop fs -text
/user/sunita/tables/jobs/jobs_noSite_parsed.json |more
{jobs: {_total: 1812, _count: 20, _start: 0, values:
[{company: {i
d: 21836, name: CyberCoders}, postingDate: {year: 2013, day:
10, mo
nth: 6}, descriptionSnippet: Software Engineer-Hadoop, HDFS, HBase,
Pig- Ve
rtica Analytics Senior Hadoop Engineer - Skills Required - Hadoop, HDFS,
HBase,
Pig, SQL, Industrial Software Development, System Integration, Java,
high perf
ormance, multi-threading, VerticaWe are a well known consumer product
developme
nt company and we are looking to add a Hadoop Engineer to our Engineering
team.
You will be working with the latest , expirationDate: {year: 2013,
day
Its a single line, so used 'more' rather than 'head'. But effectively,
the file exists and has the data.
regards
Sunita
On Tue, Jun 18, 2013 at 8:38 PM, Stephen Sprague sprag...@gmail.comwrote:
As Nitin alluded to its best to confirm the data is definitely in hdfs
using hdfs semantics rather than hive for the first step.
1. how big is it? hadoop fs -ls your hdfs dir
2. cat a bit of it and see if anything is there. hadoop fs -text your
hdfs dir/filename | head -10
do you see any data from step #2?
On Tue, Jun 18, 2013 at 3:58 PM, Sunita Arvind sunitarv...@gmail.comwrote:
I ran some complex queries. Something to the extent of
select jobs from jobs;
which triggers map reduce jobs but does not show errors and produces
the same output null. If I try referencing the struct elements, I get
error which seems to be the root cause.
Attached are the select statement outputs with the corresponding hive
logs.
I have also attached my usage details of another table - try_parsed
which has a subset of the same data which seems to work fine. Also attached
is the input file for this table - try_parsed.json
Thanks for your help
Sunita
On Tue, Jun 18, 2013 at 4:35 PM, Nitin Pawar
nitinpawar...@gmail.comwrote:
can you run a little more complex query
select uniq across columns or do some maths. so we know when it fires
up a mapreduce
On Wed, Jun 19, 2013 at 1:59 AM, Sunita Arvind
sunitarv...@gmail.comwrote:
Thanks for responding Nitin. Yes I am sure that serde is working fine
and json file is being picked based on all the errors that showed up till
this stage. What sort of error are you suspecting. File not present or
serde not parsing it ?
On Tuesday, June 18, 2013, Nitin Pawar wrote:
select * from table is as good as hdfs -cat
are you sure there is any data in the table?
On Tue, Jun 18, 2013 at 11:54 PM, Sunita Arvind
sunitarv...@gmail.com wrote:
Hi,
I am able to parse the input JSON file and load it into hive. I do
not see any errors with create table, so I am assuming that. But when
I try
to read the data, I get null
hive select * from jobs;
OK
null
I have validated the JSON with JSONLint and Notepad++ JSON plugin
and it is a valid JSON. Here
Re: Hive select shows null after successful data load
Thanks for sharing your experience Richa.
I do have timestamps but in the format of year : INT, day : INT, month :
INT.
As per your suggestion, I changed them all to string, but still get null as
the output.
regards
Sunita
On Wed, Jun 19, 2013 at 2:17 AM, Richa Sharma
mailtorichasha...@gmail.comwrote:
Do you have any timestamp fields in the table that might contain null
value ?
I faced a similar situation sometime back - changing the data type to
string made it work.
But I was working on delimited text files.
Not sure if it applies to JSON .. but its still worth giving a try !!
Richa
On Wed, Jun 19, 2013 at 7:28 AM, Sunita Arvind sunitarv...@gmail.comwrote:
Having the a column name same as the table name, is a problem due to
which I was not able to reference jobs.values.id from jobs. Changing the
table name to jobs1 resolved the semantic error.
However, the query still returns null
hive select jobs.values.position.title from jobs1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201306080116_0036, Tracking URL =
http://node01.expressanalytics.net:50030/jobdetails.jsp?jobid=job_201306080116_0036
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_201306080116_0036
Hadoop job information for Stage-1: number of mappers: 1; number of
reducers: 0
2013-06-18 18:55:52,381 Stage-1 map = 0%, reduce = 0%
2013-06-18 18:55:56,394 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
0.88 sec
2013-06-18 18:55:57,400 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
0.88 sec
2013-06-18 18:55:58,407 Stage-1 map = 100%, reduce = 100%, Cumulative
CPU 0.88 sec
MapReduce Total cumulative CPU time: 880 msec
Ended Job = job_201306080116_0036
MapReduce Jobs Launched:
Job 0: Map: 1 Cumulative CPU: 0.88 sec HDFS Read: 35374 HDFS Write: 3
SUCCESS
Total MapReduce CPU Time Spent: 880 msec
OK
null
Time taken: 9.591 seconds
regards
Sunita
On Tue, Jun 18, 2013 at 9:35 PM, Sunita Arvind sunitarv...@gmail.comwrote:
Ok.
The data files are quite small. Around 35 KB and 1 KB each.
[sunita@node01 tables]$ hadoop fs -ls /user/sunita/tables/jobs
Found 1 items
-rw-r--r-- 3 sunita hdfs 35172 2013-06-18 18:31
/user/sunita/tables/jobs/jobs_noSite_parsed.json
[sunita@node01 tables]$ hadoop fs -text
/user/sunita/tables/jobs/jobs_noSite_parsed.json |more
{jobs: {_total: 1812, _count: 20, _start: 0, values:
[{company: {i
d: 21836, name: CyberCoders}, postingDate: {year: 2013, day:
10, mo
nth: 6}, descriptionSnippet: Software Engineer-Hadoop, HDFS, HBase,
Pig- Ve
rtica Analytics Senior Hadoop Engineer - Skills Required - Hadoop, HDFS,
HBase,
Pig, SQL, Industrial Software Development, System Integration, Java,
high perf
ormance, multi-threading, VerticaWe are a well known consumer product
developme
nt company and we are looking to add a Hadoop Engineer to our
Engineering team.
You will be working with the latest , expirationDate: {year:
2013, day
Its a single line, so used 'more' rather than 'head'. But effectively,
the file exists and has the data.
regards
Sunita
On Tue, Jun 18, 2013 at 8:38 PM, Stephen Sprague sprag...@gmail.comwrote:
As Nitin alluded to its best to confirm the data is definitely in hdfs
using hdfs semantics rather than hive for the first step.
1. how big is it? hadoop fs -ls your hdfs dir
2. cat a bit of it and see if anything is there. hadoop fs -text
your hdfs dir/filename | head -10
do you see any data from step #2?
On Tue, Jun 18, 2013 at 3:58 PM, Sunita Arvind
sunitarv...@gmail.comwrote:
I ran some complex queries. Something to the extent of
select jobs from jobs;
which triggers map reduce jobs but does not show errors and produces
the same output null. If I try referencing the struct elements, I get
error which seems to be the root cause.
Attached are the select statement outputs with the corresponding hive
logs.
I have also attached my usage details of another table - try_parsed
which has a subset of the same data which seems to work fine. Also
attached
is the input file for this table - try_parsed.json
Thanks for your help
Sunita
On Tue, Jun 18, 2013 at 4:35 PM, Nitin Pawar
nitinpawar...@gmail.comwrote:
can you run a little more complex query
select uniq across columns or do some maths. so we know when it fires
up a mapreduce
On Wed, Jun 19, 2013 at 1:59 AM, Sunita Arvind sunitarv...@gmail.com
wrote:
Thanks for responding Nitin. Yes I am sure that serde is working
fine and json file is being picked based on all the errors that showed
up
till this stage. What sort of error are you suspecting. File not
present or
serde not parsing it ?
On Tuesday, June 18, 2013, Nitin Pawar wrote:
select * from table is as good as hdfs -cat
are you sure there is any data in the table?
On Tue, Jun 18, 2013 at 11:54 PM, Sunita Arvind
sunitarv...@gmail.com
Re: Hive select shows null after successful data load
try_parsed_json is not trivial imho :)
start with the very, very basic, for example, { jobs : foo }. Get
that to work first. :) When that works add a level of nesting and see
what happens. Keep building on it until you either break it (and then you
know that last thing you added broke it and can concentrate on that) or
you'll have worked out all the bugs and your final example will work.
Nothing fancy here except old school trial and error.
An alternative I keep bringing up when native semantics don't go one's way
is the transform() function. use python, perl, ruby or whatever to parse
the json and go nuts with the rich features of said language. just write
your output to stdout as a delimited serialization of what you want to
store and that's it. That would be another way to get your scalars, arrays
and structs to work.
Don't give up yet though on the JsonSerde! :) Its probably something very
easy that we just can't see.
On Wed, Jun 19, 2013 at 10:00 AM, Sunita Arvind sunitarv...@gmail.comwrote:
Thanks for looking into it Ramki.
Yes I had tried these options. Here is what I get (renamed the table to
have a meaningful name):
hive select jobs.values[1].id from linkedinjobsearch;
..mapreduce task details
OK
NULL
Time taken: 9.586 seconds
hive select jobs.values[0].position.title from linkedinjobsearch;
Total MapReduce jobs = 1
Launching Job 1 out of 1
OK
NULL
Time taken: 9.617 seconds
I am trying to connect btrace to the process to be able to trace the code
but cant get it to respond. Here is what I tried:
[sunita@node01 ~]$ hive --debug, recursive=y, port=7000,mainSuspend=y,
childSuspend=y
ERROR: Cannot load this JVM TI agent twice, check your java command line
for duplicate jdwp options.
Error occurred during initialization of VM
agent library failed to init: jdwp
Tried changing the port also. Any idea regarding the debuggers that can be
used. I also tried explain query and that does not show any issues either.
regards
Sunita
On Wed, Jun 19, 2013 at 12:11 PM, Ramki Palle ramki.pa...@gmail.comwrote:
Can you run some other queries from job1 table and see if any query
returns some data?
I am guessing your query select jobs.values.position.title from jobs1;
may have some issue. May be it should be as
select jobs.values[0].position.title from jobs1;
Regards,
Ramki.
On Wed, Jun 19, 2013 at 8:24 AM, Sunita Arvind sunitarv...@gmail.comwrote:
Thanks Stephen,
That's just what I tried with the try_parsed table. It is exactly same
data with lesser nesting in the structure and lesser number of entries.
Do you mean to say that highly nested jsons can lead to issues? What are
typical solution to such issues? Write UDFs in hive or parse the JSON into
a delimited file?
I have heard of custom serdes also. Not sure if UDFs and custom serdes
are one and the same.
regards
Sunita
On Wed, Jun 19, 2013 at 10:38 AM, Stephen Sprague sprag...@gmail.comwrote:
I think you might have to start small here instead of going for the
home run on the first swing. when all else fails start with a trivial json
object and then build up from there and see what additional step breaks
it. that way you know if the trivial example fails is something
fundamental and not the complexity of your json object that's throwing
things off.
On Wed, Jun 19, 2013 at 4:34 AM, Sunita Arvind
sunitarv...@gmail.comwrote:
Thanks for sharing your experience Richa.
I do have timestamps but in the format of year : INT, day : INT, month
: INT.
As per your suggestion, I changed them all to string, but still get
null as the output.
regards
Sunita
On Wed, Jun 19, 2013 at 2:17 AM, Richa Sharma
mailtorichasha...@gmail.com wrote:
Do you have any timestamp fields in the table that might contain null
value ?
I faced a similar situation sometime back - changing the data type to
string made it work.
But I was working on delimited text files.
Not sure if it applies to JSON .. but its still worth giving a try !!
Richa
On Wed, Jun 19, 2013 at 7:28 AM, Sunita Arvind sunitarv...@gmail.com
wrote:
Having the a column name same as the table name, is a problem due to
which I was not able to reference jobs.values.id from jobs.
Changing the table name to jobs1 resolved the semantic error.
However, the query still returns null
hive select jobs.values.position.title from jobs1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201306080116_0036, Tracking URL =
http://node01.expressanalytics.net:50030/jobdetails.jsp?jobid=job_201306080116_0036
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill
job_201306080116_0036
Hadoop job information for Stage-1: number of mappers: 1; number of
reducers: 0
2013-06-18 18:55:52,381 Stage-1 map = 0%, reduce = 0%
2013-06-18 18:55:56,394 Stage-1 map = 100%, reduce = 0%, Cumulative
CPU 0.88 sec
2013-06-18
Re: Hive select shows null after successful data load
Thanks Stephen,
Let me explore options. I will let you all know once I am successful.
regards
Sunita
On Wed, Jun 19, 2013 at 3:08 PM, Stephen Sprague sprag...@gmail.com wrote:
try_parsed_json is not trivial imho :)
start with the very, very basic, for example, { jobs : foo }. Get
that to work first. :) When that works add a level of nesting and see
what happens. Keep building on it until you either break it (and then you
know that last thing you added broke it and can concentrate on that) or
you'll have worked out all the bugs and your final example will work.
Nothing fancy here except old school trial and error.
An alternative I keep bringing up when native semantics don't go one's way
is the transform() function. use python, perl, ruby or whatever to parse
the json and go nuts with the rich features of said language. just write
your output to stdout as a delimited serialization of what you want to
store and that's it. That would be another way to get your scalars, arrays
and structs to work.
Don't give up yet though on the JsonSerde! :) Its probably something very
easy that we just can't see.
On Wed, Jun 19, 2013 at 10:00 AM, Sunita Arvind sunitarv...@gmail.comwrote:
Thanks for looking into it Ramki.
Yes I had tried these options. Here is what I get (renamed the table to
have a meaningful name):
hive select jobs.values[1].id from linkedinjobsearch;
..mapreduce task details
OK
NULL
Time taken: 9.586 seconds
hive select jobs.values[0].position.title from linkedinjobsearch;
Total MapReduce jobs = 1
Launching Job 1 out of 1
OK
NULL
Time taken: 9.617 seconds
I am trying to connect btrace to the process to be able to trace the code
but cant get it to respond. Here is what I tried:
[sunita@node01 ~]$ hive --debug, recursive=y, port=7000,mainSuspend=y,
childSuspend=y
ERROR: Cannot load this JVM TI agent twice, check your java command line
for duplicate jdwp options.
Error occurred during initialization of VM
agent library failed to init: jdwp
Tried changing the port also. Any idea regarding the debuggers that can
be used. I also tried explain query and that does not show any issues
either.
regards
Sunita
On Wed, Jun 19, 2013 at 12:11 PM, Ramki Palle ramki.pa...@gmail.comwrote:
Can you run some other queries from job1 table and see if any query
returns some data?
I am guessing your query select jobs.values.position.title from
jobs1; may have some issue. May be it should be as
select jobs.values[0].position.title from jobs1;
Regards,
Ramki.
On Wed, Jun 19, 2013 at 8:24 AM, Sunita Arvind sunitarv...@gmail.comwrote:
Thanks Stephen,
That's just what I tried with the try_parsed table. It is exactly same
data with lesser nesting in the structure and lesser number of entries.
Do you mean to say that highly nested jsons can lead to issues? What
are typical solution to such issues? Write UDFs in hive or parse the JSON
into a delimited file?
I have heard of custom serdes also. Not sure if UDFs and custom serdes
are one and the same.
regards
Sunita
On Wed, Jun 19, 2013 at 10:38 AM, Stephen Sprague
sprag...@gmail.comwrote:
I think you might have to start small here instead of going for the
home run on the first swing. when all else fails start with a trivial
json
object and then build up from there and see what additional step breaks
it. that way you know if the trivial example fails is something
fundamental and not the complexity of your json object that's throwing
things off.
On Wed, Jun 19, 2013 at 4:34 AM, Sunita Arvind
sunitarv...@gmail.comwrote:
Thanks for sharing your experience Richa.
I do have timestamps but in the format of year : INT, day : INT,
month : INT.
As per your suggestion, I changed them all to string, but still get
null as the output.
regards
Sunita
On Wed, Jun 19, 2013 at 2:17 AM, Richa Sharma
mailtorichasha...@gmail.com wrote:
Do you have any timestamp fields in the table that might contain
null value ?
I faced a similar situation sometime back - changing the data type
to string made it work.
But I was working on delimited text files.
Not sure if it applies to JSON .. but its still worth giving a try !!
Richa
On Wed, Jun 19, 2013 at 7:28 AM, Sunita Arvind
sunitarv...@gmail.com wrote:
Having the a column name same as the table name, is a problem due
to which I was not able to reference jobs.values.id from jobs.
Changing the table name to jobs1 resolved the semantic error.
However, the query still returns null
hive select jobs.values.position.title from jobs1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201306080116_0036, Tracking URL =
http://node01.expressanalytics.net:50030/jobdetails.jsp?jobid=job_201306080116_0036
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill
job_201306080116_0036
Hadoop job information for Stage-1:
Re: INSERT non-static data to array?
: https://issues.apache.org/jira/browse/HIVE-3238 This might fit the bill. On Wed, Jun 19, 2013 at 3:23 PM, Michael Malak michaelma...@yahoo.comwrote: Is the only way to INSERT data into a column of type array to load data from a pre-existing file, to use hard-coded values in the INSERT statement, or copy an entire array verbatim from another table? I.e. I'm assuming that a) SQL1999 array INSERT via subquery is not (yet) implemented in Hive, and b) there is also no other way to load dynamically generated data into an array column? If my assumption in a) is true, does a Jira item need to be created for it?
Re: INSERT non-static data to array?
The example code for inline_table() there has static data. It's not possible to use a subquery inside the inline_table() or array() is it? The SQL1999 way is described here: http://www.postgresql.org/message-id/20041028232152.ga76...@winnie.fuhr.org CREATE TABLE table_a(a int, b int, c int[]); INSERT INTO table_a SELECT a, b, ARRAY(SELECT c FROM table_c WHERE table_c.parent = table_b.id) FROM table_b From: Edward Capriolo edlinuxg...@gmail.com To: user@hive.apache.org user@hive.apache.org; Michael Malak michaelma...@yahoo.com Sent: Wednesday, June 19, 2013 2:06 PM Subject: Re: INSERT non-static data to array? : https://issues.apache.org/jira/browse/HIVE-3238 This might fit the bill. On Wed, Jun 19, 2013 at 3:23 PM, Michael Malak michaelma...@yahoo.com wrote: Is the only way to INSERT data into a column of type array to load data from a pre-existing file, to use hard-coded values in the INSERT statement, or copy an entire array verbatim from another table? I.e. I'm assuming that a) SQL1999 array INSERT via subquery is not (yet) implemented in Hive, and b) there is also no other way to load dynamically generated data into an array column? If my assumption in a) is true, does a Jira item need to be created for it?
New to hive.
Hi Folks, I am new to hive and need information, tutorials etc that you can point to. I have installed hive to work with MySQL. I can run queries. Now I would like to understand how the map and reduce classes are created and how I can look at the data for the map job and map class the hive query generates. Also is there a way to create custom map classes. I would appreciate if anyone can help me get started. Thanks, Bharati Sent from my iPadFortigate Filtered
Re: New to hive.
Hello ma'am, Hive queries are parsed using ANTLR http://www.antlr.org/ and and are converted into corresponding MR jobs(actually a lot of things happen under the hood). I had answered a similar questionhttp://stackoverflow.com/questions/17090022/hive-query-control-flow/17095756?noredirect=1#comment24873979_17095756few days ago on SO, you might find it helpful. But I would suggest you to go through the original paperhttp://cs.brown.edu/courses/cs227/papers/hive.pdfwhich explains all these things in proper detail. I would also recommend you to go through the book Programming Hive. It's really nice. HTH Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 4:24 AM, Bharati bharati.ad...@mparallelo.comwrote: Hi Folks, I am new to hive and need information, tutorials etc that you can point to. I have installed hive to work with MySQL. I can run queries. Now I would like to understand how the map and reduce classes are created and how I can look at the data for the map job and map class the hive query generates. Also is there a way to create custom map classes. I would appreciate if anyone can help me get started. Thanks, Bharati Sent from my iPad Fortigate Filtered
Re: show table throwing strange error
Did you check in your hive query log under /tmp to see if it says something in the log ? Sent from my iPhone On Jun 19, 2013, at 5:53 PM, Mohammad Tariq donta...@gmail.com wrote: Hello list, I have a hive(0.9.0) setup on my Ubuntu box running hadoop-1.0.4. Everything was going smooth till now. But today when I issued show tables I got some strange error on the CLI. Here is the error : hive show tables; FAILED: Parse Error: line 1:0 character '' not supported here line 1:1 character '' not supported here line 1:2 character '' not supported here line 1:3 character '' not supported here line 1:4 character '' not supported here line 1:5 character '' not supported here line 1:6 character '' not supported here line 1:7 character '' not supported here line 1:8 character '' not supported here line 1:9 character '' not supported here line 1:10 character '' not supported here line 1:11 character '' not supported here line 1:12 character '' not supported here line 1:13 character '' not supported here line 1:14 character '' not supported here line 1:15 character '' not supported here line 1:16 character '' not supported here line 1:17 character '' not supported here line 1:18 character '' not supported here line 1:19 character '' not supported here line 1:20 character '' not supported here line 1:21 character '' not supported here line 1:22 character '' not supported here line 1:23 character '' not supported here line 1:24 character '' not supported here line 1:25 character '' not supported here line 1:26 character '' not supported here line 1:27 character '' not supported here line 1:28 character '' not supported here line 1:29 character '' not supported here line 1:30 character '' not supported here line 1:31 character '' not supported here line 1:32 character '' not supported here line 1:33 character '' not supported here line 1:34 character '' not supported here line 1:35 character '' not supported here line 1:36 character '' not supported here line 1:37 character '' not supported here line 1:38 character '' not supported here line 1:39 character '' not supported here line 1:40 character '' not supported here line 1:41 character '' not supported here line 1:42 character '' not supported here line 1:43 character '' not supported here line 1:44 character '' not supported here line 1:45 character '' not supported here line 1:46 character '' not supported here line 1:47 character '' not supported here line 1:48 character '' not supported here line 1:49 character '' not supported here line 1:50 character '' not supported here line 1:51 character '' not supported here line 1:52 character '' not supported here line 1:53 character '' not supported here line 1:54 character '' not supported here line 1:55 character '' not supported here line 1:56 character '' not supported here line 1:57 character '' not supported here line 1:58 character '' not supported here line 1:59 character '' not supported here line 1:60 character '' not supported here line 1:61 character '' not supported here line 1:62 character '' not supported here line 1:63 character '' not supported here line 1:64 character '' not supported here line 1:65 character '' not supported here line 1:66 character '' not supported here line 1:67 character '' not supported here line 1:68 character '' not supported here line 1:69 character '' not supported here line 1:70 character '' not supported here line 1:71 character '' not supported here line 1:72 character '' not supported here line 1:73 character '' not supported here line 1:74 character '' not supported here line 1:75 character '' not supported here line 1:76 character '' not supported here line 1:77 character '' not supported here line 1:78 character '' not supported here line 1:79 character '' not supported here . . . . . . line 1:378 character '' not supported here line 1:379 character '' not supported here line 1:380 character '' not supported here line 1:381 character '' not supported here Strangely other queries like select foo from pokes where bar = 'tariq'; are working fine. Tried to search over the net but could not find anything useful.Need some help. Thank you so much for your time. Warm Regards, Tariq cloudfront.blogspot.com
Re: show table throwing strange error
Hello Anurag, Thank you for the quick response. Log files is full of such lines along with a trace that says it is some parsing related issue. But the strange thing is that here I can see *'\00' *but on the CLI it was just *' '. *I am wondering what's with wrong with *show tables;* * * line 1:79 character '\00' not supported here line 1:80 character '\00' not supported here at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:446) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:416) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:689) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:557) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Thanks again. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 6:53 AM, Anurag Tangri tangri.anu...@gmail.comwrote: Did you check in your hive query log under /tmp to see if it says something in the log ? Sent from my iPhone On Jun 19, 2013, at 5:53 PM, Mohammad Tariq donta...@gmail.com wrote: Hello list, I have a hive(0.9.0) setup on my Ubuntu box running hadoop-1.0.4. Everything was going smooth till now. But today when I issued *show tables * I got some strange error on the CLI. Here is the error : hive show tables; FAILED: Parse Error: line 1:0 character '' not supported here line 1:1 character '' not supported here line 1:2 character '' not supported here line 1:3 character '' not supported here line 1:4 character '' not supported here line 1:5 character '' not supported here line 1:6 character '' not supported here line 1:7 character '' not supported here line 1:8 character '' not supported here line 1:9 character '' not supported here line 1:10 character '' not supported here line 1:11 character '' not supported here line 1:12 character '' not supported here line 1:13 character '' not supported here line 1:14 character '' not supported here line 1:15 character '' not supported here line 1:16 character '' not supported here line 1:17 character '' not supported here line 1:18 character '' not supported here line 1:19 character '' not supported here line 1:20 character '' not supported here line 1:21 character '' not supported here line 1:22 character '' not supported here line 1:23 character '' not supported here line 1:24 character '' not supported here line 1:25 character '' not supported here line 1:26 character '' not supported here line 1:27 character '' not supported here line 1:28 character '' not supported here line 1:29 character '' not supported here line 1:30 character '' not supported here line 1:31 character '' not supported here line 1:32 character '' not supported here line 1:33 character '' not supported here line 1:34 character '' not supported here line 1:35 character '' not supported here line 1:36 character '' not supported here line 1:37 character '' not supported here line 1:38 character '' not supported here line 1:39 character '' not supported here line 1:40 character '' not supported here line 1:41 character '' not supported here line 1:42 character '' not supported here line 1:43 character '' not supported here line 1:44 character '' not supported here line 1:45 character '' not supported here line 1:46 character '' not supported here line 1:47 character '' not supported here line 1:48 character '' not supported here line 1:49 character '' not supported here line 1:50 character '' not supported here line 1:51 character '' not supported here line 1:52 character '' not supported here line 1:53 character '' not supported here line 1:54 character '' not supported here line 1:55 character '' not supported here line 1:56 character '' not supported here line 1:57 character '' not supported here line 1:58 character '' not supported here line 1:59 character '' not supported here line 1:60 character '' not supported here line 1:61 character '' not supported here line 1:62 character '' not supported here line 1:63 character '' not supported here line 1:64 character '' not supported here line 1:65 character '' not supported here line 1:66 character '' not supported here line 1:67 character '' not supported here line 1:68 character '' not supported here line 1:69 character '' not supported here line 1:70 character '' not supported here line 1:71 character '' not
Re: show table throwing strange error
Can you also check your hive site XML ? Is it properly formatted and connection strings correct ? Sent from my iPhone On Jun 19, 2013, at 6:30 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Anurag, Thank you for the quick response. Log files is full of such lines along with a trace that says it is some parsing related issue. But the strange thing is that here I can see '\00' but on the CLI it was just ' '. I am wondering what's with wrong with show tables; line 1:79 character '\00' not supported here line 1:80 character '\00' not supported here at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:446) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:416) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:689) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:557) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Thanks again. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 6:53 AM, Anurag Tangri tangri.anu...@gmail.com wrote: Did you check in your hive query log under /tmp to see if it says something in the log ? Sent from my iPhone On Jun 19, 2013, at 5:53 PM, Mohammad Tariq donta...@gmail.com wrote: Hello list, I have a hive(0.9.0) setup on my Ubuntu box running hadoop-1.0.4. Everything was going smooth till now. But today when I issued show tables I got some strange error on the CLI. Here is the error : hive show tables; FAILED: Parse Error: line 1:0 character '' not supported here line 1:1 character '' not supported here line 1:2 character '' not supported here line 1:3 character '' not supported here line 1:4 character '' not supported here line 1:5 character '' not supported here line 1:6 character '' not supported here line 1:7 character '' not supported here line 1:8 character '' not supported here line 1:9 character '' not supported here line 1:10 character '' not supported here line 1:11 character '' not supported here line 1:12 character '' not supported here line 1:13 character '' not supported here line 1:14 character '' not supported here line 1:15 character '' not supported here line 1:16 character '' not supported here line 1:17 character '' not supported here line 1:18 character '' not supported here line 1:19 character '' not supported here line 1:20 character '' not supported here line 1:21 character '' not supported here line 1:22 character '' not supported here line 1:23 character '' not supported here line 1:24 character '' not supported here line 1:25 character '' not supported here line 1:26 character '' not supported here line 1:27 character '' not supported here line 1:28 character '' not supported here line 1:29 character '' not supported here line 1:30 character '' not supported here line 1:31 character '' not supported here line 1:32 character '' not supported here line 1:33 character '' not supported here line 1:34 character '' not supported here line 1:35 character '' not supported here line 1:36 character '' not supported here line 1:37 character '' not supported here line 1:38 character '' not supported here line 1:39 character '' not supported here line 1:40 character '' not supported here line 1:41 character '' not supported here line 1:42 character '' not supported here line 1:43 character '' not supported here line 1:44 character '' not supported here line 1:45 character '' not supported here line 1:46 character '' not supported here line 1:47 character '' not supported here line 1:48 character '' not supported here line 1:49 character '' not supported here line 1:50 character '' not supported here line 1:51 character '' not supported here line 1:52 character '' not supported here line 1:53 character '' not supported here line 1:54 character '' not supported here line 1:55 character '' not supported here line 1:56 character '' not supported here line 1:57 character '' not supported here line 1:58 character '' not supported here line 1:59 character '' not supported here line 1:60 character '' not supported here line 1:61 character '' not supported here line 1:62 character '' not supported here line 1:63 character '' not supported here line 1:64 character
Re: show table throwing strange error
It looks OK to me, configuration property namejavax.jdo.option.ConnectionURL/name valuejdbc:mysql://localhost:3306/hive?*createDatabaseIfNotExist*=true /value /property property namejavax.jdo.option.ConnectionDriverName/name valuecom.mysql.jdbc.Driver/value /property property namejavax.jdo.option.ConnectionUserName/name valueapache/value /property property namejavax.jdo.option.ConnectionPassword/name valuepassword/value /property property namehive.metastore.local/name valuetrue/value /property property namehive.exec.scratchdir/name value/hadoop/hive-tmp/value descriptionScratch space for Hive jobs/description /property /configuration Anything wrong here? Thank you. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 7:06 AM, Mapred Learn mapred.le...@gmail.comwrote: Can you also check your hive site XML ? Is it properly formatted and connection strings correct ? Sent from my iPhone On Jun 19, 2013, at 6:30 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Anurag, Thank you for the quick response. Log files is full of such lines along with a trace that says it is some parsing related issue. But the strange thing is that here I can see *'\00' *but on the CLI it was just *' '. *I am wondering what's with wrong with *show tables;* * * line 1:79 character '\00' not supported here line 1:80 character '\00' not supported here at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:446) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:416) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:689) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:557) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Thanks again. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 6:53 AM, Anurag Tangri tangri.anu...@gmail.comwrote: Did you check in your hive query log under /tmp to see if it says something in the log ? Sent from my iPhone On Jun 19, 2013, at 5:53 PM, Mohammad Tariq donta...@gmail.com wrote: Hello list, I have a hive(0.9.0) setup on my Ubuntu box running hadoop-1.0.4. Everything was going smooth till now. But today when I issued *show tables* I got some strange error on the CLI. Here is the error : hive show tables; FAILED: Parse Error: line 1:0 character '' not supported here line 1:1 character '' not supported here line 1:2 character '' not supported here line 1:3 character '' not supported here line 1:4 character '' not supported here line 1:5 character '' not supported here line 1:6 character '' not supported here line 1:7 character '' not supported here line 1:8 character '' not supported here line 1:9 character '' not supported here line 1:10 character '' not supported here line 1:11 character '' not supported here line 1:12 character '' not supported here line 1:13 character '' not supported here line 1:14 character '' not supported here line 1:15 character '' not supported here line 1:16 character '' not supported here line 1:17 character '' not supported here line 1:18 character '' not supported here line 1:19 character '' not supported here line 1:20 character '' not supported here line 1:21 character '' not supported here line 1:22 character '' not supported here line 1:23 character '' not supported here line 1:24 character '' not supported here line 1:25 character '' not supported here line 1:26 character '' not supported here line 1:27 character '' not supported here line 1:28 character '' not supported here line 1:29 character '' not supported here line 1:30 character '' not supported here line 1:31 character '' not supported here line 1:32 character '' not supported here line 1:33 character '' not supported here line 1:34 character '' not supported here line 1:35 character '' not supported here line 1:36 character '' not supported here line 1:37 character '' not supported here line 1:38 character '' not supported here line 1:39 character '' not supported here line 1:40 character '' not supported here line 1:41 character '' not supported here line 1:42 character '' not supported here line 1:43 character '' not supported here line 1:44 character '' not supported here line 1:45 character '' not supported here line 1:46 character '' not supported here
Re: show table throwing strange error
It actually seems to be ignoring hive-site.xml. No effect of the properties set in hive-site.xml file. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 7:12 AM, Mohammad Tariq donta...@gmail.com wrote: It looks OK to me, configuration property namejavax.jdo.option.ConnectionURL/name valuejdbc:mysql://localhost:3306/hive?*createDatabaseIfNotExist*=true /value /property property namejavax.jdo.option.ConnectionDriverName/name valuecom.mysql.jdbc.Driver/value /property property namejavax.jdo.option.ConnectionUserName/name valueapache/value /property property namejavax.jdo.option.ConnectionPassword/name valuepassword/value /property property namehive.metastore.local/name valuetrue/value /property property namehive.exec.scratchdir/name value/hadoop/hive-tmp/value descriptionScratch space for Hive jobs/description /property /configuration Anything wrong here? Thank you. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 7:06 AM, Mapred Learn mapred.le...@gmail.comwrote: Can you also check your hive site XML ? Is it properly formatted and connection strings correct ? Sent from my iPhone On Jun 19, 2013, at 6:30 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Anurag, Thank you for the quick response. Log files is full of such lines along with a trace that says it is some parsing related issue. But the strange thing is that here I can see *'\00' *but on the CLI it was just *' '. *I am wondering what's with wrong with *show tables;* * * line 1:79 character '\00' not supported here line 1:80 character '\00' not supported here at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:446) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:416) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:689) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:557) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Thanks again. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 6:53 AM, Anurag Tangri tangri.anu...@gmail.comwrote: Did you check in your hive query log under /tmp to see if it says something in the log ? Sent from my iPhone On Jun 19, 2013, at 5:53 PM, Mohammad Tariq donta...@gmail.com wrote: Hello list, I have a hive(0.9.0) setup on my Ubuntu box running hadoop-1.0.4. Everything was going smooth till now. But today when I issued *show tables* I got some strange error on the CLI. Here is the error : hive show tables; FAILED: Parse Error: line 1:0 character '' not supported here line 1:1 character '' not supported here line 1:2 character '' not supported here line 1:3 character '' not supported here line 1:4 character '' not supported here line 1:5 character '' not supported here line 1:6 character '' not supported here line 1:7 character '' not supported here line 1:8 character '' not supported here line 1:9 character '' not supported here line 1:10 character '' not supported here line 1:11 character '' not supported here line 1:12 character '' not supported here line 1:13 character '' not supported here line 1:14 character '' not supported here line 1:15 character '' not supported here line 1:16 character '' not supported here line 1:17 character '' not supported here line 1:18 character '' not supported here line 1:19 character '' not supported here line 1:20 character '' not supported here line 1:21 character '' not supported here line 1:22 character '' not supported here line 1:23 character '' not supported here line 1:24 character '' not supported here line 1:25 character '' not supported here line 1:26 character '' not supported here line 1:27 character '' not supported here line 1:28 character '' not supported here line 1:29 character '' not supported here line 1:30 character '' not supported here line 1:31 character '' not supported here line 1:32 character '' not supported here line 1:33 character '' not supported here line 1:34 character '' not supported here line 1:35 character '' not supported here line 1:36 character '' not supported here line 1:37 character '' not supported here line 1:38 character '' not supported here line 1:39 character '' not supported here line 1:40 character '' not supported
Re: show table throwing strange error
Looks like you use MySQL. Can you check if your MySQL still up ? and permissions on your hive metastore db ? Sent from my iPhone On Jun 19, 2013, at 6:44 PM, Mohammad Tariq donta...@gmail.com wrote: It actually seems to be ignoring hive-site.xml. No effect of the properties set in hive-site.xml file. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 7:12 AM, Mohammad Tariq donta...@gmail.com wrote: It looks OK to me, configuration property namejavax.jdo.option.ConnectionURL/name valuejdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true/value /property property namejavax.jdo.option.ConnectionDriverName/name valuecom.mysql.jdbc.Driver/value /property property namejavax.jdo.option.ConnectionUserName/name valueapache/value /property property namejavax.jdo.option.ConnectionPassword/name valuepassword/value /property property namehive.metastore.local/name valuetrue/value /property property namehive.exec.scratchdir/name value/hadoop/hive-tmp/value descriptionScratch space for Hive jobs/description /property /configuration Anything wrong here? Thank you. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 7:06 AM, Mapred Learn mapred.le...@gmail.com wrote: Can you also check your hive site XML ? Is it properly formatted and connection strings correct ? Sent from my iPhone On Jun 19, 2013, at 6:30 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Anurag, Thank you for the quick response. Log files is full of such lines along with a trace that says it is some parsing related issue. But the strange thing is that here I can see '\00' but on the CLI it was just ' '. I am wondering what's with wrong with show tables; line 1:79 character '\00' not supported here line 1:80 character '\00' not supported here at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:446) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:416) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:689) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:557) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Thanks again. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 6:53 AM, Anurag Tangri tangri.anu...@gmail.com wrote: Did you check in your hive query log under /tmp to see if it says something in the log ? Sent from my iPhone On Jun 19, 2013, at 5:53 PM, Mohammad Tariq donta...@gmail.com wrote: Hello list, I have a hive(0.9.0) setup on my Ubuntu box running hadoop-1.0.4. Everything was going smooth till now. But today when I issued show tables I got some strange error on the CLI. Here is the error : hive show tables; FAILED: Parse Error: line 1:0 character '' not supported here line 1:1 character '' not supported here line 1:2 character '' not supported here line 1:3 character '' not supported here line 1:4 character '' not supported here line 1:5 character '' not supported here line 1:6 character '' not supported here line 1:7 character '' not supported here line 1:8 character '' not supported here line 1:9 character '' not supported here line 1:10 character '' not supported here line 1:11 character '' not supported here line 1:12 character '' not supported here line 1:13 character '' not supported here line 1:14 character '' not supported here line 1:15 character '' not supported here line 1:16 character '' not supported here line 1:17 character '' not supported here line 1:18 character '' not supported here line 1:19 character '' not supported here line 1:20 character '' not supported here line 1:21 character '' not supported here line 1:22 character '' not supported here line 1:23 character '' not supported here line 1:24 character '' not supported here line 1:25 character '' not supported here line 1:26 character '' not supported here line 1:27 character '' not supported here line 1:28 character '' not supported here line 1:29 character '' not supported here line 1:30 character '' not supported here line 1:31 character '' not supported here line 1:32 character '' not supported here line 1:33 character '' not supported here line 1:34 character
Re: show table throwing strange error
Yes. It's up and running with all the required permissions. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 7:21 AM, Anurag Tangri tangri.anu...@gmail.comwrote: Looks like you use MySQL. Can you check if your MySQL still up ? and permissions on your hive metastore db ? Sent from my iPhone On Jun 19, 2013, at 6:44 PM, Mohammad Tariq donta...@gmail.com wrote: It actually seems to be ignoring hive-site.xml. No effect of the properties set in hive-site.xml file. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 7:12 AM, Mohammad Tariq donta...@gmail.comwrote: It looks OK to me, configuration property namejavax.jdo.option.ConnectionURL/name valuejdbc:mysql://localhost:3306/hive?*createDatabaseIfNotExist*=true /value /property property namejavax.jdo.option.ConnectionDriverName/name valuecom.mysql.jdbc.Driver/value /property property namejavax.jdo.option.ConnectionUserName/name valueapache/value /property property namejavax.jdo.option.ConnectionPassword/name valuepassword/value /property property namehive.metastore.local/name valuetrue/value /property property namehive.exec.scratchdir/name value/hadoop/hive-tmp/value descriptionScratch space for Hive jobs/description /property /configuration Anything wrong here? Thank you. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 7:06 AM, Mapred Learn mapred.le...@gmail.comwrote: Can you also check your hive site XML ? Is it properly formatted and connection strings correct ? Sent from my iPhone On Jun 19, 2013, at 6:30 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Anurag, Thank you for the quick response. Log files is full of such lines along with a trace that says it is some parsing related issue. But the strange thing is that here I can see *'\00' *but on the CLI it was just *' '. *I am wondering what's with wrong with *show tables;* * * line 1:79 character '\00' not supported here line 1:80 character '\00' not supported here at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:446) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:416) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:689) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:557) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Thanks again. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 6:53 AM, Anurag Tangri tangri.anu...@gmail.comwrote: Did you check in your hive query log under /tmp to see if it says something in the log ? Sent from my iPhone On Jun 19, 2013, at 5:53 PM, Mohammad Tariq donta...@gmail.com wrote: Hello list, I have a hive(0.9.0) setup on my Ubuntu box running hadoop-1.0.4. Everything was going smooth till now. But today when I issued *show tables* I got some strange error on the CLI. Here is the error : hive show tables; FAILED: Parse Error: line 1:0 character '' not supported here line 1:1 character '' not supported here line 1:2 character '' not supported here line 1:3 character '' not supported here line 1:4 character '' not supported here line 1:5 character '' not supported here line 1:6 character '' not supported here line 1:7 character '' not supported here line 1:8 character '' not supported here line 1:9 character '' not supported here line 1:10 character '' not supported here line 1:11 character '' not supported here line 1:12 character '' not supported here line 1:13 character '' not supported here line 1:14 character '' not supported here line 1:15 character '' not supported here line 1:16 character '' not supported here line 1:17 character '' not supported here line 1:18 character '' not supported here line 1:19 character '' not supported here line 1:20 character '' not supported here line 1:21 character '' not supported here line 1:22 character '' not supported here line 1:23 character '' not supported here line 1:24 character '' not supported here line 1:25 character '' not supported here line 1:26 character '' not supported here line 1:27 character '' not supported here line 1:28 character '' not supported here line 1:29 character '' not supported here line 1:30 character '' not supported here line 1:31 character ''
Re: show table throwing strange error
That error is gone after I recreated hive-site.xml and restarted hive. But now there seems to be some problem with metastore settings. It's not going to mysql. Anyways, thank you both for the help. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 7:29 AM, Mohammad Tariq donta...@gmail.com wrote: Yes. It's up and running with all the required permissions. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 7:21 AM, Anurag Tangri tangri.anu...@gmail.comwrote: Looks like you use MySQL. Can you check if your MySQL still up ? and permissions on your hive metastore db ? Sent from my iPhone On Jun 19, 2013, at 6:44 PM, Mohammad Tariq donta...@gmail.com wrote: It actually seems to be ignoring hive-site.xml. No effect of the properties set in hive-site.xml file. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 7:12 AM, Mohammad Tariq donta...@gmail.comwrote: It looks OK to me, configuration property namejavax.jdo.option.ConnectionURL/name valuejdbc:mysql://localhost:3306/hive?*createDatabaseIfNotExist* =true/value /property property namejavax.jdo.option.ConnectionDriverName/name valuecom.mysql.jdbc.Driver/value /property property namejavax.jdo.option.ConnectionUserName/name valueapache/value /property property namejavax.jdo.option.ConnectionPassword/name valuepassword/value /property property namehive.metastore.local/name valuetrue/value /property property namehive.exec.scratchdir/name value/hadoop/hive-tmp/value descriptionScratch space for Hive jobs/description /property /configuration Anything wrong here? Thank you. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 7:06 AM, Mapred Learn mapred.le...@gmail.comwrote: Can you also check your hive site XML ? Is it properly formatted and connection strings correct ? Sent from my iPhone On Jun 19, 2013, at 6:30 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Anurag, Thank you for the quick response. Log files is full of such lines along with a trace that says it is some parsing related issue. But the strange thing is that here I can see *'\00' *but on the CLI it was just *' '. *I am wondering what's with wrong with *show tables;* * * line 1:79 character '\00' not supported here line 1:80 character '\00' not supported here at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:446) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:416) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:689) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:557) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Thanks again. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 6:53 AM, Anurag Tangri tangri.anu...@gmail.com wrote: Did you check in your hive query log under /tmp to see if it says something in the log ? Sent from my iPhone On Jun 19, 2013, at 5:53 PM, Mohammad Tariq donta...@gmail.com wrote: Hello list, I have a hive(0.9.0) setup on my Ubuntu box running hadoop-1.0.4. Everything was going smooth till now. But today when I issued *show tables* I got some strange error on the CLI. Here is the error : hive show tables; FAILED: Parse Error: line 1:0 character '' not supported here line 1:1 character '' not supported here line 1:2 character '' not supported here line 1:3 character '' not supported here line 1:4 character '' not supported here line 1:5 character '' not supported here line 1:6 character '' not supported here line 1:7 character '' not supported here line 1:8 character '' not supported here line 1:9 character '' not supported here line 1:10 character '' not supported here line 1:11 character '' not supported here line 1:12 character '' not supported here line 1:13 character '' not supported here line 1:14 character '' not supported here line 1:15 character '' not supported here line 1:16 character '' not supported here line 1:17 character '' not supported here line 1:18 character '' not supported here line 1:19 character '' not supported here line 1:20 character '' not supported here line 1:21 character '' not supported here line 1:22 character '' not supported here line 1:23 character '' not supported
Re: Hive select shows null after successful data load
Finally I could get it work. The issue resolves once I remove the arrays
within position structure. So that is the limitation of the serde. I
changed 'industries' to string and 'jobfunctions' to Mapstring,string I
can query the table just fine now. Here is the complete DDL for reference:
create external table linkedin_Jobsearch (
jobs STRUCT
values : ARRAYSTRUCT
company : STRUCT
id : STRING,
name : STRING,
postingDate : STRUCT
year : STRING,
day : STRING,
month : STRING,
descriptionSnippet : STRING,
expirationDate : STRUCT
year : STRING,
day : STRING,
month : STRING,
position : STRUCT
jobFunctions : MAPSTRING,STRING,--these were arrays of structure
in my previous attempts
industries : STRING,
title : STRING,
jobType : STRUCT
code : STRING,
name : STRING,
experienceLevel : STRUCT
code : STRING,
name : STRING,
id : STRING,
customerJobCode : STRING,
skillsAndExperience : STRING,
salary : STRING,
jobPoster : STRUCT
id : STRING,
firstName : STRING,
lastName : STRING,
headline : STRING,
referralBonus : STRING,
locationDescription : STRING
)
ROW FORMAT SERDE 'com.cloudera.hive.serde.JSONSerDe'
LOCATION '/user/sunita/tables/jobs';
Thanks Stephen for sharing your thoughts. It helped.
Also if someone /Stephen could help me display this information in a useful
manner, that would be great. Right now all the values show up as arrays.
Here is what I mean:
For a query like this:
hive select jobs.values.company.name, jobs.values.position.title,
jobs.values.locationdescription from linkedin_jobsearch;
This is the output:
[CyberCoders,CyberCoders,CyberCoders,Management Science
Associates,Google,Google,CyberCoders,CyberCoders,HP,Sigmaways,Global
Data Consultancy,Global Data
Consultancy,CyberCoders,CyberCoders,CyberCoders,VMware,CD IT
Recruitment,CD IT Recruitment,Digital Reasoning Systems,AOL]
[Software Engineer-Hadoop, HDFS, HBase, Pig- Vertica Analytics,Software
Engineer-Hadoop, HDFS, HBase, Pig- Vertica Analytics,Software
Engineer-Hadoop, HDFS, HBase, Pig- Vertica Analytics,Data
Architect,Systems Engineer, Site Reliability Engineering,Systems
Engineer, Site Reliability Engineering,NoSQL Engineer - MongoDB for big
data, web crawling - RELO OFFER,NoSQL Engineer - MongoDB for big data,
web crawling - RELO OFFER,Hadoop Database Administrator Medicare,Hadoop
/ Big Data Consultant,Lead Hadoop developer,Head of Big Data -
Hadoop,Hadoop Engineer - Hadoop, Operations, Linux Admin, Java,
Storage,Sr. Hadoop Administrator - Hadoop, MapReduce, HDFS,Sr. Hadoop
Administrator - Hadoop, MapReduce, HDFS,Software Engineer - Big
Data,Hadoop Team Lead Consultant - Global Leader in Big Data
solutions,Hadoop Administrator Consultant - Global Leader in Big Data
solutions,Java Developer,Sr.Software Engineer-Big Data-Hadoop]
[Pittsburgh, PA,Pittsburgh, PA,Harrisburg, PA,Pittsburgh, PA
(Shadyside area near Bakery Square),Pittsburgh, PA, USA,Pittsburgh,
PA,Cleveland, OH,Akron, OH,Herndon, VA,Cupertino, CA,London,
United Kingdom,London, United Kingdom,Mountain View, CA,san jose,
CA,Santa Clara, CA,Palo Alto, CA,Home based - Live anywhere in the UK
or Benelux,Home based - Live anywhere in the UK or Benelux,Herndon,
VA,Dulles, VA]
Time taken: 8.518 seconds
All company names come into an array, all position titles into another
array and all locationdescription into yet another array. I cannot map 1
value to the other.
The below query gives a decent output where individual columns can be
somewhat mapped:
hive select jobs.values[0].company.name, jobs.values[0].position.title,
jobs.values[0].locationdescription from linkedin_jobsearch;
CyberCoders Software Engineer-Hadoop, HDFS, HBase, Pig- Vertica
Analytics Pittsburgh, PA
Time taken: 8.543 seconds
But if I want to get the whole list this does not work. I have tried
setting Input and output formats and setting serde properties to map to
columns, but the output is the same. I haven't tried LATERAL VIEW
json_tuple as yet, I found it cryptic and I hope there is something simpler.
I can think of writing a UDF which loops for the length of the array and I
can access values array incrementally. But the catch here is, what if one
of the records has any of their fields missing.
Is there a better solution?
regards
Sunita
On Wed, Jun 19, 2013 at 3:29 PM, Sunita Arvind sunitarv...@gmail.comwrote:
Thanks Stephen,
Let me explore options. I will let you all know once I am successful.
regards
Sunita
On Wed, Jun 19, 2013 at 3:08 PM, Stephen Sprague sprag...@gmail.comwrote:
try_parsed_json is not trivial imho :)
start with the very, very basic, for example, { jobs : foo }. Get
that to work first. :) When that works add a level of nesting and see
what happens. Keep building on it until you either break it (and then you
know that last thing you added broke it and can concentrate on that) or
you'll have worked out all the bugs and your final example will work.
Nothing fancy here except old school trial and error.
An alternative I keep bringing up when native semantics
Re: show table throwing strange error
Your issue seems familiar. Try logging out of hive session and re-login. Sunita On Wed, Jun 19, 2013 at 8:53 PM, Mohammad Tariq donta...@gmail.com wrote: Hello list, I have a hive(0.9.0) setup on my Ubuntu box running hadoop-1.0.4. Everything was going smooth till now. But today when I issued *show tables * I got some strange error on the CLI. Here is the error : hive show tables; FAILED: Parse Error: line 1:0 character '' not supported here line 1:1 character '' not supported here line 1:2 character '' not supported here line 1:3 character '' not supported here line 1:4 character '' not supported here line 1:5 character '' not supported here line 1:6 character '' not supported here line 1:7 character '' not supported here line 1:8 character '' not supported here line 1:9 character '' not supported here line 1:10 character '' not supported here line 1:11 character '' not supported here line 1:12 character '' not supported here line 1:13 character '' not supported here line 1:14 character '' not supported here line 1:15 character '' not supported here line 1:16 character '' not supported here line 1:17 character '' not supported here line 1:18 character '' not supported here line 1:19 character '' not supported here line 1:20 character '' not supported here line 1:21 character '' not supported here line 1:22 character '' not supported here line 1:23 character '' not supported here line 1:24 character '' not supported here line 1:25 character '' not supported here line 1:26 character '' not supported here line 1:27 character '' not supported here line 1:28 character '' not supported here line 1:29 character '' not supported here line 1:30 character '' not supported here line 1:31 character '' not supported here line 1:32 character '' not supported here line 1:33 character '' not supported here line 1:34 character '' not supported here line 1:35 character '' not supported here line 1:36 character '' not supported here line 1:37 character '' not supported here line 1:38 character '' not supported here line 1:39 character '' not supported here line 1:40 character '' not supported here line 1:41 character '' not supported here line 1:42 character '' not supported here line 1:43 character '' not supported here line 1:44 character '' not supported here line 1:45 character '' not supported here line 1:46 character '' not supported here line 1:47 character '' not supported here line 1:48 character '' not supported here line 1:49 character '' not supported here line 1:50 character '' not supported here line 1:51 character '' not supported here line 1:52 character '' not supported here line 1:53 character '' not supported here line 1:54 character '' not supported here line 1:55 character '' not supported here line 1:56 character '' not supported here line 1:57 character '' not supported here line 1:58 character '' not supported here line 1:59 character '' not supported here line 1:60 character '' not supported here line 1:61 character '' not supported here line 1:62 character '' not supported here line 1:63 character '' not supported here line 1:64 character '' not supported here line 1:65 character '' not supported here line 1:66 character '' not supported here line 1:67 character '' not supported here line 1:68 character '' not supported here line 1:69 character '' not supported here line 1:70 character '' not supported here line 1:71 character '' not supported here line 1:72 character '' not supported here line 1:73 character '' not supported here line 1:74 character '' not supported here line 1:75 character '' not supported here line 1:76 character '' not supported here line 1:77 character '' not supported here line 1:78 character '' not supported here line 1:79 character '' not supported here . . . . . . line 1:378 character '' not supported here line 1:379 character '' not supported here line 1:380 character '' not supported here line 1:381 character '' not supported here Strangely other queries like *select foo from pokes where bar = 'tariq';*are working fine. Tried to search over the net but could not find anything useful.Need some help. Thank you so much for your time. Warm Regards, Tariq cloudfront.blogspot.com
Re: New to hive.
Programming Hive and Hadoop: The Definitive Guide are available at the O'Reilly website (http://oreilly.com/) and on Amazon. But don't forget the Hive wiki: - Hive Home -- https://cwiki.apache.org/confluence/display/Hive/Home - Getting Started -- https://cwiki.apache.org/confluence/display/Hive/GettingStarted - Hive Tutorial -- https://cwiki.apache.org/confluence/display/Hive/Tutorial – Lefty On Wed, Jun 19, 2013 at 7:02 PM, Mohammad Tariq donta...@gmail.com wrote: Hello ma'am, Hive queries are parsed using ANTLR http://www.antlr.org/ and and are converted into corresponding MR jobs(actually a lot of things happen under the hood). I had answered a similar questionhttp://stackoverflow.com/questions/17090022/hive-query-control-flow/17095756?noredirect=1#comment24873979_17095756few days ago on SO, you might find it helpful. But I would suggest you to go through the original paperhttp://cs.brown.edu/courses/cs227/papers/hive.pdfwhich explains all these things in proper detail. I would also recommend you to go through the book Programming Hive. It's really nice. HTH Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jun 20, 2013 at 4:24 AM, Bharati bharati.ad...@mparallelo.comwrote: Hi Folks, I am new to hive and need information, tutorials etc that you can point to. I have installed hive to work with MySQL. I can run queries. Now I would like to understand how the map and reduce classes are created and how I can look at the data for the map job and map class the hive query generates. Also is there a way to create custom map classes. I would appreciate if anyone can help me get started. Thanks, Bharati Sent from my iPad Fortigate Filtered
