[jira] [Resolved] (HIVE-1019) java.io.FileNotFoundException: HIVE_PLAN (No such file or directory)
[ https://issues.apache.org/jira/browse/HIVE-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut resolved HIVE-1019. Resolution: Won't Fix Hiveserver2 doesn't suffer from this. java.io.FileNotFoundException: HIVE_PLAN (No such file or directory) Key: HIVE-1019 URL: https://issues.apache.org/jira/browse/HIVE-1019 Project: Hive Issue Type: Bug Components: Server Infrastructure Affects Versions: 0.6.0 Reporter: Bennie Schut Assignee: Bennie Schut Priority: Minor Attachments: HIVE-1019-1.patch, HIVE-1019-2.patch, HIVE-1019-3.patch, HIVE-1019-4.patch, HIVE-1019-5.patch, HIVE-1019-6.patch, HIVE-1019-7.patch, HIVE-1019-8.patch, HIVE-1019.patch, stacktrace2.txt I keep getting errors like this: java.io.FileNotFoundException: HIVE_PLAN (No such file or directory) and : java.io.IOException: cannot find dir = hdfs://victoria.ebuddy.com:9000/tmp/hive-dwh/801467596/10002 in partToPartitionInfo! when running multiple threads with roughly similar queries. I have a patch for this which works for me. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-1019) java.io.FileNotFoundException: HIVE_PLAN (No such file or directory)
[ https://issues.apache.org/jira/browse/HIVE-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018555#comment-14018555 ] Bennie Schut commented on HIVE-1019: xuanjinlee this is a somewhat prehistoric patch which I forgot to close. Most people have moved to hiveserver2 which doesn't suffer from these threading issues. Unless anyone objects I would actually like to close this issue. java.io.FileNotFoundException: HIVE_PLAN (No such file or directory) Key: HIVE-1019 URL: https://issues.apache.org/jira/browse/HIVE-1019 Project: Hive Issue Type: Bug Components: Server Infrastructure Affects Versions: 0.6.0 Reporter: Bennie Schut Assignee: Bennie Schut Priority: Minor Attachments: HIVE-1019-1.patch, HIVE-1019-2.patch, HIVE-1019-3.patch, HIVE-1019-4.patch, HIVE-1019-5.patch, HIVE-1019-6.patch, HIVE-1019-7.patch, HIVE-1019-8.patch, HIVE-1019.patch, stacktrace2.txt I keep getting errors like this: java.io.FileNotFoundException: HIVE_PLAN (No such file or directory) and : java.io.IOException: cannot find dir = hdfs://victoria.ebuddy.com:9000/tmp/hive-dwh/801467596/10002 in partToPartitionInfo! when running multiple threads with roughly similar queries. I have a patch for this which works for me. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HIVE-1539) Concurrent metastore threading problem
[ https://issues.apache.org/jira/browse/HIVE-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut resolved HIVE-1539. Resolution: Fixed Release Note: We switched to a datanucleus version = 2.2 a long time ago so this is fixed. Concurrent metastore threading problem --- Key: HIVE-1539 URL: https://issues.apache.org/jira/browse/HIVE-1539 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: ClassLoaderResolver.patch, HIVE-1539-1.patch, HIVE-1539.patch, thread_dump_hanging.txt When running hive as a service and running a high number of queries concurrently I end up with multiple threads running at 100% cpu without any progress. Looking at these threads I notice this thread(484e): at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:598) But on a different thread(63a2): at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.java) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-3308) Mixing avro and snappy gives null values
[ https://issues.apache.org/jira/browse/HIVE-3308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13612502#comment-13612502 ] Bennie Schut commented on HIVE-3308: I would really appreciate someone committing this. It has tests showing the issue with correct results after the patch. It makes the serde more consistent with other serdes. Basically anyone using compression combined with avro will hit this bug like we see with HIVE-4195. Mixing avro and snappy gives null values Key: HIVE-3308 URL: https://issues.apache.org/jira/browse/HIVE-3308 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: HIVE-3308.patch1.txt, HIVE-3308.patch2.txt On default hive uses LazySimpleSerDe for output. When I now enable compression and select count(*) from avrotable the output is a file with the .avro extension but this then will display null values since the file is in reality not an avro file but a file created by LazySimpleSerDe using compression so should be a .snappy file. This causes any job (exception select * from avrotable is that not truly a job) to show null values. If you use any serde other then avro you can temporarily fix this by setting set hive.output.file.extension=.snappy and it will correctly work again but this won't work on avro since it overwrites the hive.output.file.extension during initializing. When you dump the query result into a table with create table bla as you can rename the .avro file into .snappy and the select from bla will also magiacally work again. Input and Ouput serdes don't always match so when I use avro as an input format it should not set the hive.output.file.extension. Onces it's set all queries will use it and fail making the connection useless to reuse. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4195) Avro SerDe causes incorrect behavior in unrelated tables
[
https://issues.apache.org/jira/browse/HIVE-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13604936#comment-13604936
]
Bennie Schut commented on HIVE-4195:
I suspect this is a duplicate of HIVE-3308. It's a really simple patch but
hasn't been commited yet.
Avro SerDe causes incorrect behavior in unrelated tables
Key: HIVE-4195
URL: https://issues.apache.org/jira/browse/HIVE-4195
Project: Hive
Issue Type: Bug
Reporter: Skye Wanderman-Milne
When I run a file that first creates an Avro table using the Avro SerDe, then
immediately creates an LZO text table and inserts data into the LZO table,
the resulting LZO table contain Avro data files. When I remove the Avro
CREATE TABLE statement, the LZO table contains .lzo files as expected.
{noformat}
DROP TABLE IF EXISTS avro_table;
CREATE EXTERNAL TABLE avro_table
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS
INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES ('avro.schema.literal' = '{
namespace: testing.hive.avro.serde,
name: test_record,
type: record,
fields: [
{name:int1, type:long},
{name:string1, type:string}
]
}');
DROP TABLE IF EXISTS lzo_table;
CREATE EXTERNAL TABLE lzo_table (
id int,
bool_col boolean,
tinyint_col tinyint,
smallint_col smallint,
int_col int,
bigint_col bigint,
float_col float,
double_col double,
date_string_col string,
string_col string,
timestamp_col timestamp)
STORED AS
INPUTFORMAT 'com.hadoop.mapred.DeprecatedLzoTextInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
;
SET hive.exec.compress.output=true;
SET mapred.output.compression.type=BLOCK;
SET mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec;
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.dynamic.partition=true;
SET mapred.max.split.size=25600;
SET hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
insert overwrite table lzo_table SELECT id, bool_col, tinyint_col,
smallint_col, int_col, bigint_col, float_col, double_col, date_string_col,
string_col, timestamp_col FROM src_table;
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3308) Mixing avro and snappy gives null values
[ https://issues.apache.org/jira/browse/HIVE-3308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-3308: --- Attachment: HIVE-3308.patch2.txt Mixing avro and snappy gives null values Key: HIVE-3308 URL: https://issues.apache.org/jira/browse/HIVE-3308 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Bennie Schut Attachments: HIVE-3308.patch1.txt, HIVE-3308.patch2.txt On default hive uses LazySimpleSerDe for output. When I now enable compression and select count(*) from avrotable the output is a file with the .avro extension but this then will display null values since the file is in reality not an avro file but a file created by LazySimpleSerDe using compression so should be a .snappy file. This causes any job (exception select * from avrotable is that not truly a job) to show null values. If you use any serde other then avro you can temporarily fix this by setting set hive.output.file.extension=.snappy and it will correctly work again but this won't work on avro since it overwrites the hive.output.file.extension during initializing. When you dump the query result into a table with create table bla as you can rename the .avro file into .snappy and the select from bla will also magiacally work again. Input and Ouput serdes don't always match so when I use avro as an input format it should not set the hive.output.file.extension. Onces it's set all queries will use it and fail making the connection useless to reuse. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3308) Mixing avro and snappy gives null values
[ https://issues.apache.org/jira/browse/HIVE-3308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-3308: --- Status: Patch Available (was: Open) Mixing avro and snappy gives null values Key: HIVE-3308 URL: https://issues.apache.org/jira/browse/HIVE-3308 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Bennie Schut Attachments: HIVE-3308.patch1.txt, HIVE-3308.patch2.txt On default hive uses LazySimpleSerDe for output. When I now enable compression and select count(*) from avrotable the output is a file with the .avro extension but this then will display null values since the file is in reality not an avro file but a file created by LazySimpleSerDe using compression so should be a .snappy file. This causes any job (exception select * from avrotable is that not truly a job) to show null values. If you use any serde other then avro you can temporarily fix this by setting set hive.output.file.extension=.snappy and it will correctly work again but this won't work on avro since it overwrites the hive.output.file.extension during initializing. When you dump the query result into a table with create table bla as you can rename the .avro file into .snappy and the select from bla will also magiacally work again. Input and Ouput serdes don't always match so when I use avro as an input format it should not set the hive.output.file.extension. Onces it's set all queries will use it and fail making the connection useless to reuse. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Phabricator issues, libphutil v1 libraries are no longer supported
I've been having issues using Phabricator to create a patch for hive. It looks like some versioning issues with arc_jira_lib and libphutil. I kept getting this error: libphutil v1 libraries are no longer supported. To work around it go to the libphutil directory you checked out of git and revert to an older commit: git checkout 870bcc76434410344d27a3fa4604ac96200bf7f6 The next problem is the arc client: ERR-BAD-VERSION: Your 'arc' client version is '6', which is newer than the server version, '5'. Upgrade your Phabricator install. So also get an older version: git checkout 6cb8d483b23c132da4c82934438cd1af0e5ba32b I'll leave the actual fixing to someone with a bit more php experience. After that I tried to arc diff HIVE-3308 added the message then: Linting... LINT OKAY No lint problems. Running unit tests... No unit test engine is configured for this project. Usage Exception: There are no changes to generate a diff from! In the end I manually uploaded It as D8187. And added the patch to Jira even tough I guess that's what arc is suppose to do. Bennie.
[jira] [Created] (HIVE-3308) Mixing avro and snappy gives null values
Bennie Schut created HIVE-3308: -- Summary: Mixing avro and snappy gives null values Key: HIVE-3308 URL: https://issues.apache.org/jira/browse/HIVE-3308 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Bennie Schut On default hive uses LazySimpleSerDe for output. When I now enable compression and select count(*) from avrotable the output is a file with the .avro extension but this then will display null values since the file is in reality not an avro file but a file created by LazySimpleSerDe using compression so should be a .snappy file. This causes any job (exception select * from avrotable is that not truly a job) to show null values. If you use any serde other then avro you can temporarily fix this by setting set hive.output.file.extension=.snappy and it will correctly work again but this won't work on avro since it overwrites the hive.output.file.extension during initializing. When you dump the query result into a table with create table bla as you can rename the .avro file into .snappy and the select from bla will also magiacally work again. Input and Ouput serdes don't always match so when I use avro as an input format it should not set the hive.output.file.extension. Onces it's set all queries will use it and fail making the connection useless to reuse. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3308) Mixing avro and snappy gives null values
[ https://issues.apache.org/jira/browse/HIVE-3308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-3308: --- Attachment: HIVE-3308.patch1.txt Added a test to show the problem. Result of the test will show: A masked pattern was here POSTHOOK: query: select count(*) from src POSTHOOK: type: QUERY POSTHOOK: Input: default@src A masked pattern was here NULL But should show something like: A masked pattern was here POSTHOOK: query: select count(*) from src POSTHOOK: type: QUERY POSTHOOK: Input: default@src A masked pattern was here 500 Mixing avro and snappy gives null values Key: HIVE-3308 URL: https://issues.apache.org/jira/browse/HIVE-3308 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Bennie Schut Attachments: HIVE-3308.patch1.txt On default hive uses LazySimpleSerDe for output. When I now enable compression and select count(*) from avrotable the output is a file with the .avro extension but this then will display null values since the file is in reality not an avro file but a file created by LazySimpleSerDe using compression so should be a .snappy file. This causes any job (exception select * from avrotable is that not truly a job) to show null values. If you use any serde other then avro you can temporarily fix this by setting set hive.output.file.extension=.snappy and it will correctly work again but this won't work on avro since it overwrites the hive.output.file.extension during initializing. When you dump the query result into a table with create table bla as you can rename the .avro file into .snappy and the select from bla will also magiacally work again. Input and Ouput serdes don't always match so when I use avro as an input format it should not set the hive.output.file.extension. Onces it's set all queries will use it and fail making the connection useless to reuse. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3299) UDF DAYNAME(date) to HIVE
[
https://issues.apache.org/jira/browse/HIVE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422296#comment-13422296
]
Bennie Schut commented on HIVE-3299:
Currently you can get this trough a unix_timestamp function so if you don't
want to wait for an implementation do something like this.
select from_unixtime(unix_timestamp() , '') from dummytable limit 1;
or
select from_unixtime(unix_timestamp('2012-07-25 00:00:00'), '') from
dummytable limit 1;
However simply having a nice udf for it would be nice, like they have in mysql:
http://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html#function_dayname
UDF DAYNAME(date) to HIVE
---
Key: HIVE-3299
URL: https://issues.apache.org/jira/browse/HIVE-3299
Project: Hive
Issue Type: New Feature
Components: UDF
Affects Versions: 0.9.0
Reporter: Namitha Babychan
Fix For: 0.10.0
Current releases of Hive lacks a function which would return the day name
corresponding to a date / timestamp value which might be a part of a column.
The function -DAYNAME (date) would return the day name from a date /
timestamp or column which would be useful while using HiveQL. This would find
its use in various business sectors like retail, which would help in
identifying the trends and sales datails for a particular weekday for entire
year,month or week.
Functionality :-
Function Name: DAYNAME (date)
Returns the name of the weekday for date.
Example: hive SELECT DAYNAME('2012-07-25');
- 'Wednesday'
Usage :-
Case 1 : To find DAY NAME corresponding to a particular date
hive SELECT DAYNAME('2012-07-25');
- 'Wednesday'
Case 2 : To query a table to find details based on a particular day name
Table :-
date |item id|store id|value|unit|price
01/07/2012|110001|00003|0.99|1.00|0.99
02/07/2012|110001|00008|0.99|0.00|0.00
03/07/2012|110001|00009|0.99|0.00|0.00
04/07/2012|110001|001112002|0.99|0.00|0.00
05/07/2012|110001|001112003|0.99|0.00|0.00
06/07/2012|110001|001112006|0.99|1.00|0.99
07/07/2012|110001|001112007|0.99|0.00|0.00
08/07/2012|110001|001112008|0.99|0.00|0.00
09/07/2012|110001|001112009|0.99|0.00|0.00
10/07/2012|110001|001112010|0.99|0.00|0.00
11/07/2012|110001|001113003|0.99|0.00|0.00
12/07/2012|110001|001113006|0.99|0.00|0.00
13/07/2012|110001|001113008|0.99|0.00|0.00
14/07/2012|110001|001113010|0.99|0.00|0.00
15/07/2012|110001|001114002|0.99|0.00|0.00
16/07/2012|110001|001114004|0.99|1.00|0.99
17/07/2012|110001|001114005|0.99|0.00|0.00
18/07/2012|110001|001121004|0.99|0.00|0.00
Query : select * from sales where dayname(date)='wednesday';
Result :-
04/07/2012|110001|001112002|0.99|0.00|0.00
11/07/2012|110001|001113003|0.99|0.00|0.00
18/07/2012|110001|001121004|0.99|0.00|0.00
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1603) support CSV text file format
[
https://issues.apache.org/jira/browse/HIVE-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13260542#comment-13260542
]
Bennie Schut commented on HIVE-1603:
I normally export to csv this way:
set hive.exec.compress.output=false;
create table tmp_test ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES
TERMINATED BY '\n' as select * from sometable;
Then on the command line:
/opt/hadoop/bin/hadoop fs -cat /user/hive/warehouse/tmp_test/00_0 | gzip -c
tmp_test.csv.gz
Wouldn't this be doing the same thing?
support CSV text file format
Key: HIVE-1603
URL: https://issues.apache.org/jira/browse/HIVE-1603
Project: Hive
Issue Type: New Feature
Affects Versions: 0.7.0
Reporter: Ning Zhang
Comma Separated Values (CSV) text format are commonly used in exchanging
relational data between heterogeneous systems. Currently Hive uses TextFile
format when displaying query results. This could cause confusions when column
values contain new lines or tabs. A CSVTextFile format could get around this
problem. This will require a new CSVTextInputFormat, CSVTextOutputFormat, and
CSVSerDe.
A proposed use case is like:
{code}
-- exporting a table to CSV files in a directory
hive set hive.io.output.fileformat=CSVTextFile;
hive insert overwrite local directory '/tmp/CSVrepos/' select * from S where
... ;
-- query result in CSV
hive -e 'set hive.io.output.fileformat=CSVTextFile; select * from T;' |
sql_loader_to_other_systems
-- query CSV files directory from Hive
hive create table T (...) stored as CSVTextFile;
hive load data local inpath '/my/CSVfiles' into table T;
hive select * from T where ...;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2573) Create per-session function registry
[ https://issues.apache.org/jira/browse/HIVE-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149558#comment-13149558 ] Bennie Schut commented on HIVE-2573: This probably fixes HIVE-2544 also. That would be nice. Create per-session function registry - Key: HIVE-2573 URL: https://issues.apache.org/jira/browse/HIVE-2573 Project: Hive Issue Type: Improvement Components: Server Infrastructure Reporter: Navis Priority: Minor Attachments: HIVE-2573.1.patch.txt Currently the function registry is shared resource and could be overrided by other users when using HiveServer. If per-session function registry is provided, this situation could be prevented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HIVE-2544) Nullpointer on registering udfs.
[ https://issues.apache.org/jira/browse/HIVE-2544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-2544 started by Bennie Schut. Nullpointer on registering udfs. Key: HIVE-2544 URL: https://issues.apache.org/jira/browse/HIVE-2544 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Bennie Schut Assignee: Bennie Schut Attachments: HIVE-2544.1.patch.txt Currently the Function registry can throw NullPointers when multiple threads are trying to register the same function. The normal put() will replace the existing registered function object even if it's exactly the same function. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2544) Nullpointer on registering udfs.
Nullpointer on registering udfs. Key: HIVE-2544 URL: https://issues.apache.org/jira/browse/HIVE-2544 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Bennie Schut Assignee: Bennie Schut Currently the Function registry can throw NullPointers when multiple threads are trying to register the same function. The normal put() will replace the existing registered function object even if it's exactly the same function. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2297) Fix NPE in ConditionalResolverSkewJoin
[ https://issues.apache.org/jira/browse/HIVE-2297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070354#comment-13070354 ] Bennie Schut commented on HIVE-2297: From what I know about the java compiler it will simply stop evaluating the rest of the line if fstatus != null is false. So the brackets don't add anything. A lot of code is written with this knowledge in mind. Fix NPE in ConditionalResolverSkewJoin -- Key: HIVE-2297 URL: https://issues.apache.org/jira/browse/HIVE-2297 Project: Hive Issue Type: Bug Reporter: Vaibhav Aggarwal Assignee: Vaibhav Aggarwal Attachments: HIVE-2297.patch, fix_npe.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2191) Allow optional [inner] on equi-join.
[ https://issues.apache.org/jira/browse/HIVE-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-2191: --- Status: Patch Available (was: Open) Allow optional [inner] on equi-join. Key: HIVE-2191 URL: https://issues.apache.org/jira/browse/HIVE-2191 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Bennie Schut Assignee: Bennie Schut Priority: Trivial Fix For: 0.8.0 Attachments: HIVE-2191.1.patch.txt, HIVE-2191.2.patch.txt, HIVE-2191.3.patch.txt Lot's of databases including mysql support an optional inner keyword to explicitely select an equi-join. As shown in the mysql docs: http://dev.mysql.com/doc/refman/5.1/en/join.html For completeness/portability we should allow this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2191) Allow optional [inner] on equi-join.
[ https://issues.apache.org/jira/browse/HIVE-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-2191: --- Attachment: HIVE-2191.2.patch.txt Added a test. Allow optional [inner] on equi-join. Key: HIVE-2191 URL: https://issues.apache.org/jira/browse/HIVE-2191 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Bennie Schut Assignee: Bennie Schut Priority: Trivial Fix For: 0.8.0 Attachments: HIVE-2191.1.patch.txt, HIVE-2191.2.patch.txt Lot's of databases including mysql support an optional inner keyword to explicitely select an equi-join. As shown in the mysql docs: http://dev.mysql.com/doc/refman/5.1/en/join.html For completeness/portability we should allow this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2191) Allow optional [inner] on equi-join.
[ https://issues.apache.org/jira/browse/HIVE-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-2191: --- Attachment: (was: HIVE-2191.2.patch.txt) Allow optional [inner] on equi-join. Key: HIVE-2191 URL: https://issues.apache.org/jira/browse/HIVE-2191 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Bennie Schut Assignee: Bennie Schut Priority: Trivial Fix For: 0.8.0 Attachments: HIVE-2191.1.patch.txt, HIVE-2191.2.patch.txt Lot's of databases including mysql support an optional inner keyword to explicitely select an equi-join. As shown in the mysql docs: http://dev.mysql.com/doc/refman/5.1/en/join.html For completeness/portability we should allow this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2191) Allow optional [inner] on equi-join.
[ https://issues.apache.org/jira/browse/HIVE-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-2191: --- Status: Patch Available (was: Open) Allow optional [inner] on equi-join. Key: HIVE-2191 URL: https://issues.apache.org/jira/browse/HIVE-2191 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Bennie Schut Assignee: Bennie Schut Priority: Trivial Fix For: 0.8.0 Attachments: HIVE-2191.1.patch.txt, HIVE-2191.2.patch.txt Lot's of databases including mysql support an optional inner keyword to explicitely select an equi-join. As shown in the mysql docs: http://dev.mysql.com/doc/refman/5.1/en/join.html For completeness/portability we should allow this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-2191: Allow optional [inner] on equi-join.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/852/ --- (Updated 2011-06-25 21:34:41.720584) Review request for hive. Changes --- Added a test Summary --- HIVE-2191: Allow optional [inner] on equi-join. This addresses bug HIVE-2191. https://issues.apache.org/jira/browse/HIVE-2191 Diffs (updated) - trunk/docs/xdocs/language_manual/joins.xml 1139355 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1139355 trunk/ql/src/test/queries/clientpositive/innerjoin.q PRE-CREATION trunk/ql/src/test/results/clientpositive/innerjoin.q.out PRE-CREATION Diff: https://reviews.apache.org/r/852/diff Testing --- Thanks, Bennie
[jira] [Commented] (HIVE-2230) Hive Client build error
[
https://issues.apache.org/jira/browse/HIVE-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053806#comment-13053806
]
Bennie Schut commented on HIVE-2230:
I talked with Dmytro offline and this line on the wiki should probably changed:
The Hive ODBC driver was developed with Thrift trunk version r790732, but the
latest revision should also be fine.
Hive 0.7 and higher uses thrift 0.5.0. I'm not sure what happens when you mix
with a newer version of thrift but the older version (r790732) doesn't seem to
work. I would probably advice others to use 0.5.0.
Hive Client build error
---
Key: HIVE-2230
URL: https://issues.apache.org/jira/browse/HIVE-2230
Project: Hive
Issue Type: Bug
Components: Clients, ODBC
Environment: hive:
{code}
Path: .
URL: http://svn.apache.org/repos/asf/hive/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1138016
Node Kind: directory
Schedule: normal
Last Changed Author: jvs
Last Changed Rev: 1137839
Last Changed Date: 2011-06-21 03:41:17 +0200 (Tue, 21 Jun 2011)
{code}
thrift:
{code}
Path: .
URL: http://svn.apache.org/repos/asf/thrift/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1138011
Node Kind: directory
Schedule: normal
Last Changed Author: molinaro
Last Changed Rev: 1137870
Last Changed Date: 2011-06-21 08:20:18 +0200 (Tue, 21 Jun 2011)
{code}
Reporter: Dmytro Korochkin
While running ant
{code}
ant compile-cpp -Dthrift.home=/usr/local
{code}
to build Hive Client according to http://wiki.apache.org/hadoop/Hive/HiveODBC
I've got following error message:
{code}
compile-cpp:
[exec] mkdir -p /home/ubuntu/hive/build/metastore/objs
[exec] g++ -Wall -g -fPIC -m32 -DARCH32 -I/usr/local/include/thrift
-I/usr/local/include/thrift/fb303 -I/include
-I/home/ubuntu/hive/service/src/gen/thrift/gen-cpp
-I/home/ubuntu/hive/ql/src/gen/thrift/gen-cpp
-I/home/ubuntu/hive/metastore/src/gen/thrift/gen-cpp
-I/home/ubuntu/hive/odbc/src/cpp -c
/home/ubuntu/hive/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp -o
/home/ubuntu/hive/build/metastore/objs/ThriftHiveMetastore.o
[exec]
/home/ubuntu/hive/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp:
In member function 'virtual bool
Apache::Hadoop::Hive::ThriftHiveMetastoreProcessor::process_fn(apache::thrift::protocol::TProtocol*,
apache::thrift::protocol::TProtocol*, std::string, int32_t)':
[exec]
/home/ubuntu/hive/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp:18014:92:
error: no matching function for call to
'Apache::Hadoop::Hive::ThriftHiveMetastoreProcessor::process_fn(apache::thrift::protocol::TProtocol*,
apache::thrift::protocol::TProtocol*, std::string, int32_t)'
[exec] /usr/local/include/thrift/fb303/FacebookService.h:1299:16: note:
candidate is: virtual bool
facebook::fb303::FacebookServiceProcessor::process_fn(apache::thrift::protocol::TProtocol*,
apache::thrift::protocol::TProtocol*, std::string, int32_t, void*)
[exec] make: ***
[/home/ubuntu/hive/build/metastore/objs/ThriftHiveMetastore.o] Error 1
BUILD FAILED
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2230) Hive Client build error
[
https://issues.apache.org/jira/browse/HIVE-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053086#comment-13053086
]
Bennie Schut commented on HIVE-2230:
Perhaps related to HIVE-1526 where we switched to thrift 0.5.0? The wiki still
seems to claim you need Thrift trunk version r790732
Hive Client build error
---
Key: HIVE-2230
URL: https://issues.apache.org/jira/browse/HIVE-2230
Project: Hive
Issue Type: Bug
Components: Clients, ODBC
Environment: hive:
{code}
Path: .
URL: http://svn.apache.org/repos/asf/hive/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1138016
Node Kind: directory
Schedule: normal
Last Changed Author: jvs
Last Changed Rev: 1137839
Last Changed Date: 2011-06-21 03:41:17 +0200 (Tue, 21 Jun 2011)
{code}
thrift:
{code}
Path: .
URL: http://svn.apache.org/repos/asf/thrift/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1138011
Node Kind: directory
Schedule: normal
Last Changed Author: molinaro
Last Changed Rev: 1137870
Last Changed Date: 2011-06-21 08:20:18 +0200 (Tue, 21 Jun 2011)
{code}
Reporter: Dmytro Korochkin
While running ant
{code}
ant compile-cpp -Dthrift.home=/usr/local
{code}
to build Hive Client according to http://wiki.apache.org/hadoop/Hive/HiveODBC
I've got following error message:
{code}
compile-cpp:
[exec] mkdir -p /home/ubuntu/hive/build/metastore/objs
[exec] g++ -Wall -g -fPIC -m32 -DARCH32 -I/usr/local/include/thrift
-I/usr/local/include/thrift/fb303 -I/include
-I/home/ubuntu/hive/service/src/gen/thrift/gen-cpp
-I/home/ubuntu/hive/ql/src/gen/thrift/gen-cpp
-I/home/ubuntu/hive/metastore/src/gen/thrift/gen-cpp
-I/home/ubuntu/hive/odbc/src/cpp -c
/home/ubuntu/hive/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp -o
/home/ubuntu/hive/build/metastore/objs/ThriftHiveMetastore.o
[exec]
/home/ubuntu/hive/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp:
In member function 'virtual bool
Apache::Hadoop::Hive::ThriftHiveMetastoreProcessor::process_fn(apache::thrift::protocol::TProtocol*,
apache::thrift::protocol::TProtocol*, std::string, int32_t)':
[exec]
/home/ubuntu/hive/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp:18014:92:
error: no matching function for call to
'Apache::Hadoop::Hive::ThriftHiveMetastoreProcessor::process_fn(apache::thrift::protocol::TProtocol*,
apache::thrift::protocol::TProtocol*, std::string, int32_t)'
[exec] /usr/local/include/thrift/fb303/FacebookService.h:1299:16: note:
candidate is: virtual bool
facebook::fb303::FacebookServiceProcessor::process_fn(apache::thrift::protocol::TProtocol*,
apache::thrift::protocol::TProtocol*, std::string, int32_t, void*)
[exec] make: ***
[/home/ubuntu/hive/build/metastore/objs/ThriftHiveMetastore.o] Error 1
BUILD FAILED
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2163) SquirrelSQL 3.2.1 doesn't work with Hive 0.7.0
[
https://issues.apache.org/jira/browse/HIVE-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13043317#comment-13043317
]
Bennie Schut commented on HIVE-2163:
HIVE-2054 removes the dependency to SessionState which includes code which
should be server side so it's most likely fixed by that. Can you try 0.7.1 or
higher?
SquirrelSQL 3.2.1 doesn't work with Hive 0.7.0
--
Key: HIVE-2163
URL: https://issues.apache.org/jira/browse/HIVE-2163
Project: Hive
Issue Type: Bug
Components: JDBC
Affects Versions: 0.7.0
Reporter: Carl Steinbach
I tried using SquirrelSQL 3.2.1 with Hive 0.7.0 following the directions
located here: http://wiki.apache.org/hadoop/Hive/HiveJDBCInterface
I registered the Hive JDBC driver with SquirrelSQL, and then get the
following exception when I try to connect to the Hive Server process:
{noformat}
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: java.io.IOException: failure to login
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:226)
at org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:72)
at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:110)
at
net.sourceforge.squirrel_sql.fw.sql.SQLDriverManager.getConnection(SQLDriverManager.java:133)
at
net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.execute(OpenConnectionCommand.java:97)
at
net.sourceforge.squirrel_sql.client.mainframe.action.ConnectToAliasCommand$SheetHandler.run(ConnectToAliasCommand.java:281)
at
net.sourceforge.squirrel_sql.client.mainframe.action.ConnectToAliasCommand$SheetHandler.performOK(ConnectToAliasCommand.java:238)
at
net.sourceforge.squirrel_sql.client.gui.db.ConnectionInternalFrame.connect(ConnectionInternalFrame.java:311)
at
net.sourceforge.squirrel_sql.client.gui.db.ConnectionInternalFrame.access$300(ConnectionInternalFrame.java:56)
at
net.sourceforge.squirrel_sql.client.gui.db.ConnectionInternalFrame$MyOkClosePanelListener.okPressed(ConnectionInternalFrame.java:461)
at
net.sourceforge.squirrel_sql.client.gui.OkClosePanel.fireButtonPressed(OkClosePanel.java:148)
at
net.sourceforge.squirrel_sql.client.gui.OkClosePanel.access$100(OkClosePanel.java:33)
at
net.sourceforge.squirrel_sql.client.gui.OkClosePanel$1.actionPerformed(OkClosePanel.java:174)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2028)
at
javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2351)
at
javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
at
javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
at java.awt.Component.processMouseEvent(Component.java:6374)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3267)
at java.awt.Component.processEvent(Component.java:6139)
at java.awt.Container.processEvent(Container.java:2085)
at java.awt.Component.dispatchEventImpl(Component.java:4736)
at java.awt.Container.dispatchEventImpl(Container.java:2143)
at java.awt.Component.dispatchEvent(Component.java:4566)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4621)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4282)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4212)
at java.awt.Container.dispatchEventImpl(Container.java:2129)
at java.awt.Window.dispatchEventImpl(Window.java:2478)
at java.awt.Component.dispatchEvent(Component.java:4566)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:680)
at java.awt.EventQueue.access$000(EventQueue.java:86)
at java.awt.EventQueue$1.run(EventQueue.java:639)
at java.awt.EventQueue$1.run(EventQueue.java:637)
at java.security.AccessController.doPrivileged(Native Method)
at
java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at
java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:98)
at java.awt.EventQueue$2.run(EventQueue.java:653)
at java.awt.EventQueue$2.run(EventQueue.java:651)
at java.security.AccessController.doPrivileged(Native Method)
at
java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:650)
at net.sourceforge.squirrel_sql.client.Main$1.dispatchEvent(Main.java:93)
at
java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:296)
at
java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:211)
at
java.awt.EventDispatchThread.pumpEventsForHierarchy
[jira] [Work started] (HIVE-2191) Allow optional [inner] on equi-join.
[ https://issues.apache.org/jira/browse/HIVE-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-2191 started by Bennie Schut. Allow optional [inner] on equi-join. Key: HIVE-2191 URL: https://issues.apache.org/jira/browse/HIVE-2191 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Bennie Schut Assignee: Bennie Schut Priority: Trivial Fix For: 0.8.0 Lot's of databases including mysql support an optional inner keyword to explicitely select an equi-join. As shown in the mysql docs: http://dev.mysql.com/doc/refman/5.1/en/join.html For completeness/portability we should allow this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2191) Allow optional [inner] on equi-join.
[ https://issues.apache.org/jira/browse/HIVE-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-2191: --- Attachment: HIVE-2191.1.patch.txt Created an alias for inner join. Allow optional [inner] on equi-join. Key: HIVE-2191 URL: https://issues.apache.org/jira/browse/HIVE-2191 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Bennie Schut Assignee: Bennie Schut Priority: Trivial Fix For: 0.8.0 Attachments: HIVE-2191.1.patch.txt Lot's of databases including mysql support an optional inner keyword to explicitely select an equi-join. As shown in the mysql docs: http://dev.mysql.com/doc/refman/5.1/en/join.html For completeness/portability we should allow this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2191) Allow optional [inner] on equi-join.
[ https://issues.apache.org/jira/browse/HIVE-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-2191: --- Release Note: Allow optional [inner] on equi-join. Status: Patch Available (was: In Progress) https://reviews.apache.org/r/852/ I wasn't sure if a test should be added for this. I could copy a normal join test but it would add to the total build time, considering it's just an alias to join. But if a test is preferred I don't mind adding it. Allow optional [inner] on equi-join. Key: HIVE-2191 URL: https://issues.apache.org/jira/browse/HIVE-2191 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Bennie Schut Assignee: Bennie Schut Priority: Trivial Fix For: 0.8.0 Attachments: HIVE-2191.1.patch.txt Lot's of databases including mysql support an optional inner keyword to explicitely select an equi-join. As shown in the mysql docs: http://dev.mysql.com/doc/refman/5.1/en/join.html For completeness/portability we should allow this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: HIVE-2191: Allow optional [inner] on equi-join.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/852/ --- Review request for hive. Summary --- HIVE-2191: Allow optional [inner] on equi-join. This addresses bug HIVE-2191. https://issues.apache.org/jira/browse/HIVE-2191 Diffs - trunk/docs/xdocs/language_manual/joins.xml 1131084 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1131084 Diff: https://reviews.apache.org/r/852/diff Testing --- Thanks, Bennie
[jira] [Commented] (HIVE-2054) Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified
[ https://issues.apache.org/jira/browse/HIVE-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13015295#comment-13015295 ] Bennie Schut commented on HIVE-2054: Any chance in getting this committed :) This is rather annoying for windows users. Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified - Key: HIVE-2054 URL: https://issues.apache.org/jira/browse/HIVE-2054 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.8.0 Reporter: Bennie Schut Assignee: Bennie Schut Priority: Minor Fix For: 0.8.0 Attachments: HIVE-2054.1.patch.txt, HIVE-2054.2.patch.txt, HIVE-2054.3.patch.txt It seems something recently changed on the jdbc driver which causes this IOException on windows. java.lang.RuntimeException: java.io.IOException: The system cannot find the path specified at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:237) at org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:73) at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:110) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13010099#comment-13010099 ] Bennie Schut commented on HIVE-1675: I'm unable to reproduce this on the current trunk. I still get errors on local mode but it's something different java.lang.IllegalStateException(Shutdown in progress) after witch the server shuts down. I'll make a new jira for that and I think we can close this one. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2073) java.lang.IllegalStateException(Shutdown in progress) on local mode
java.lang.IllegalStateException(Shutdown in progress) on local mode --- Key: HIVE-2073 URL: https://issues.apache.org/jira/browse/HIVE-2073 Project: Hive Issue Type: Bug Components: Server Infrastructure Affects Versions: 0.8.0 Reporter: Bennie Schut When enabling local mode some queries will run fine. After a while we see an IllegalStateException after which the entire service shuts down. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13010103#comment-13010103 ] Bennie Schut commented on HIVE-1675: Added HIVE-2073 for the different error we now get on local mode. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2073) java.lang.IllegalStateException(Shutdown in progress) on local mode
[ https://issues.apache.org/jira/browse/HIVE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13010102#comment-13010102 ] Bennie Schut commented on HIVE-2073: 2011-03-23 12:40:18,154 ERROR exec.MapRedTask (MapRedTask.java:execute(261)) - Execution failed with exit status: 2 2011-03-23 12:40:18,208 ERROR exec.MapRedTask (MapRedTask.java:execute(261)) - Execution failed with exit status: 2 2011-03-23 12:40:18,342 ERROR ql.Driver (SessionState.java:printError(360)) - FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask 2011-03-23 12:40:18,354 ERROR ql.Driver (SessionState.java:printError(360)) - FAILED: Hive Internal Error: java.lang.IllegalStateException(Shutdown in progress) java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:39) at java.lang.Runtime.addShutdownHook(Runtime.java:192) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1392) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196) at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167) at org.apache.hadoop.hive.ql.Context.getLocalScratchDir(Context.java:168) at org.apache.hadoop.hive.ql.Context.getLocalTmpFileURI(Context.java:286) at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:223) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:344) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:817) at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:151) at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:764) at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:742) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) 2011-03-23 12:40:18,485 ERROR exec.MapRedTask (MapRedTask.java:execute(261)) - Execution failed with exit status: 2 java.lang.IllegalStateException(Shutdown in progress) on local mode --- Key: HIVE-2073 URL: https://issues.apache.org/jira/browse/HIVE-2073 Project: Hive Issue Type: Bug Components: Server Infrastructure Affects Versions: 0.8.0 Reporter: Bennie Schut When enabling local mode some queries will run fine. After a while we see an IllegalStateException after which the entire service shuts down. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2073) java.lang.IllegalStateException(Shutdown in progress) on local mode
[ https://issues.apache.org/jira/browse/HIVE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13010134#comment-13010134 ] Bennie Schut commented on HIVE-2073: It seems we are trying to add some scratchDir stuff to an instance which is shutting down. Perhaps this is an effect and not the initial cause. Still it's not nice it's shutting down the entire service because of this. java.lang.IllegalStateException(Shutdown in progress) on local mode --- Key: HIVE-2073 URL: https://issues.apache.org/jira/browse/HIVE-2073 Project: Hive Issue Type: Bug Components: Server Infrastructure Affects Versions: 0.8.0 Reporter: Bennie Schut When enabling local mode some queries will run fine. After a while we see an IllegalStateException after which the entire service shuts down. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2069) NullPointerException on getSchemas
NullPointerException on getSchemas -- Key: HIVE-2069 URL: https://issues.apache.org/jira/browse/HIVE-2069 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.8.0 Reporter: Bennie Schut Assignee: Bennie Schut Fix For: 0.8.0 Calling getSchemas will cause a nullpointerexception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2069) NullPointerException on getSchemas
[ https://issues.apache.org/jira/browse/HIVE-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13009825#comment-13009825 ] Bennie Schut commented on HIVE-2069: java.lang.NullPointerException at java.util.ArrayList.init(ArrayList.java:131) at org.apache.hadoop.hive.jdbc.HiveMetaDataResultSet.init(HiveMetaDataResultSet.java:32) at org.apache.hadoop.hive.jdbc.HiveDatabaseMetaData$3.init(HiveDatabaseMetaData.java:481) at org.apache.hadoop.hive.jdbc.HiveDatabaseMetaData.getSchemas(HiveDatabaseMetaData.java:480) at org.apache.hadoop.hive.jdbc.HiveDatabaseMetaData.getSchemas(HiveDatabaseMetaData.java:475) at org.apache.hadoop.hive.jdbc.TestJdbcDriver.testMetaDataGetSchemas(TestJdbcDriver.java:488) Probably introduced on HIVE-1126. getCatalogs works correctly but this wasn't tested. NullPointerException on getSchemas -- Key: HIVE-2069 URL: https://issues.apache.org/jira/browse/HIVE-2069 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.8.0 Reporter: Bennie Schut Assignee: Bennie Schut Fix For: 0.8.0 Calling getSchemas will cause a nullpointerexception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2069) NullPointerException on getSchemas
[ https://issues.apache.org/jira/browse/HIVE-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-2069: --- Attachment: HIVE-2069.1.patch.txt This patch includes a fix and a test which can be used to reproduce the nullpointer. NullPointerException on getSchemas -- Key: HIVE-2069 URL: https://issues.apache.org/jira/browse/HIVE-2069 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.8.0 Reporter: Bennie Schut Assignee: Bennie Schut Fix For: 0.8.0 Attachments: HIVE-2069.1.patch.txt Calling getSchemas will cause a nullpointerexception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2069) NullPointerException on getSchemas
[ https://issues.apache.org/jira/browse/HIVE-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-2069: --- Release Note: Fix for NullPointerException on the jdbc driver on getSchemas Status: Patch Available (was: Open) NullPointerException on getSchemas -- Key: HIVE-2069 URL: https://issues.apache.org/jira/browse/HIVE-2069 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.8.0 Reporter: Bennie Schut Assignee: Bennie Schut Fix For: 0.8.0 Attachments: HIVE-2069.1.patch.txt Calling getSchemas will cause a nullpointerexception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: HIVE-2069: NullPointerException on getSchemas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/521/ --- Review request for hive. Summary --- HIVE-2069: NullPointerException on getSchemas This addresses bug HIVE-2069. https://issues.apache.org/jira/browse/HIVE-2069 Diffs - trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveMetaDataResultSet.java 1083926 trunk/jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java 1083926 Diff: https://reviews.apache.org/r/521/diff Testing --- Thanks, Bennie
Re: Review Request: HIVE-2054: fix for IOException on the jdbc driver on windows.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/513/ --- (Updated 2011-03-21 12:50:40.422997) Review request for hive. Changes --- New patch because of changes from HIVE-2062 Summary --- HIVE-2054: fix for IOException on the jdbc driver on windows. This addresses bug HIVE-2054. https://issues.apache.org/jira/browse/HIVE-2054 Diffs (updated) - trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 1083914 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java 1083914 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveStatement.java 1083914 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/JdbcSessionState.java 1083914 Diff: https://reviews.apache.org/r/513/diff Testing --- Thanks, Bennie
[jira] [Updated] (HIVE-2054) Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified
[ https://issues.apache.org/jira/browse/HIVE-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-2054: --- Attachment: HIVE-2054.3.patch.txt New patch because of changes from HIVE-2062 Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified - Key: HIVE-2054 URL: https://issues.apache.org/jira/browse/HIVE-2054 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.8.0 Reporter: Bennie Schut Assignee: Bennie Schut Priority: Minor Fix For: 0.8.0 Attachments: HIVE-2054.1.patch.txt, HIVE-2054.2.patch.txt, HIVE-2054.3.patch.txt It seems something recently changed on the jdbc driver which causes this IOException on windows. java.lang.RuntimeException: java.io.IOException: The system cannot find the path specified at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:237) at org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:73) at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:110) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-2054) Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified
[ https://issues.apache.org/jira/browse/HIVE-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-2054: --- Attachment: HIVE-2054.2.patch.txt New patch because of merge from HIVE-1815. Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified - Key: HIVE-2054 URL: https://issues.apache.org/jira/browse/HIVE-2054 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.8.0 Reporter: Bennie Schut Assignee: Bennie Schut Priority: Minor Fix For: 0.8.0 Attachments: HIVE-2054.1.patch.txt, HIVE-2054.2.patch.txt It seems something recently changed on the jdbc driver which causes this IOException on windows. java.lang.RuntimeException: java.io.IOException: The system cannot find the path specified at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:237) at org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:73) at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:110) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-2054: fix for IOException on the jdbc driver on windows.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/513/ --- (Updated 2011-03-18 01:08:21.290554) Review request for hive. Changes --- New patch because of merge from HIVE-1815. Summary --- HIVE-2054: fix for IOException on the jdbc driver on windows. This addresses bug HIVE-2054. https://issues.apache.org/jira/browse/HIVE-2054 Diffs (updated) - trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 1082843 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java 1082843 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveStatement.java 1082843 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/JdbcSessionState.java 1081782 Diff: https://reviews.apache.org/r/513/diff Testing --- Thanks, Bennie
[jira] Updated: (HIVE-1815) The class HiveResultSet should implement batch fetching.
[ https://issues.apache.org/jira/browse/HIVE-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-1815: --- Attachment: HIVE-1815.2.patch.txt Updated to use an iterator instead of deleting items. The class HiveResultSet should implement batch fetching. Key: HIVE-1815 URL: https://issues.apache.org/jira/browse/HIVE-1815 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.8.0 Environment: Custom Java application using the Hive JDBC driver to connect to a Hive server, execute a Hive query and process the results. Reporter: Guy le Mar Fix For: 0.8.0 Attachments: HIVE-1815.1.patch.txt, HIVE-1815.2.patch.txt When using the Hive JDBC driver, you can execute a Hive query and obtain a HiveResultSet instance that contains the results of the query. Unfortunately, HiveResultSet can then only fetch a single row of these results from the Hive server at a time. As a consequence, it's extremely slow to fetch a resultset of anything other than a trivial size. It would be nice for the HiveResultSet to be able to fetch N rows from the server at a time, so that performance is suitable to support applications that provide human interaction. (From memory, I think it took me around 20 minutes to fetch 4000 rows.) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-2054) Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified
[ https://issues.apache.org/jira/browse/HIVE-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13007848#comment-13007848 ] Bennie Schut commented on HIVE-2054: Yes setting hive.querylog.location makes it work. At the very least we should remove the extends SessionState since it introduces a link to the hive server code which makes no sense at this point in time. However I have a preference for removing it all together since it currently adds no value. On the jdbc side I would expect the HiveConnection to hold the state which it is actually doing right now. Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified - Key: HIVE-2054 URL: https://issues.apache.org/jira/browse/HIVE-2054 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.8.0 Reporter: Bennie Schut Assignee: Bennie Schut Priority: Minor Fix For: 0.8.0 Attachments: HIVE-2054.1.patch.txt It seems something recently changed on the jdbc driver which causes this IOException on windows. java.lang.RuntimeException: java.io.IOException: The system cannot find the path specified at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:237) at org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:73) at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:110) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-1815: The class HiveResultSet should implement batch fetching.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/514/ --- (Updated 2011-03-17 01:06:34.734673) Review request for hive. Changes --- Updated to use an iterator instead of deleting items. Summary --- HIVE-1815: The class HiveResultSet should implement batch fetching. This addresses bug HIVE-1815. https://issues.apache.org/jira/browse/HIVE-1815 Diffs (updated) - trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveQueryResultSet.java 1081785 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveStatement.java 1081785 trunk/jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java 1081785 Diff: https://reviews.apache.org/r/514/diff Testing --- Thanks, Bennie
[jira] Commented: (HIVE-2054) Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified
[
https://issues.apache.org/jira/browse/HIVE-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13007366#comment-13007366
]
Bennie Schut commented on HIVE-2054:
Yes it was this code block:
try {
File tmpFile = File.createTempFile(sessionID, .pipeout, tmpDir);
tmpFile.deleteOnExit();
startSs.setTmpOutputFile(tmpFile);
} catch (IOException e) {
throw new RuntimeException(e);
}
So you are correct it's related to changes from HIVE-818.
Exception on windows when using the jdbc driver. IOException: The system
cannot find the path specified
-
Key: HIVE-2054
URL: https://issues.apache.org/jira/browse/HIVE-2054
Project: Hive
Issue Type: Bug
Components: JDBC
Affects Versions: 0.8.0
Reporter: Bennie Schut
Assignee: Bennie Schut
Priority: Minor
Fix For: 0.8.0
Attachments: HIVE-2054.1.patch.txt
It seems something recently changed on the jdbc driver which causes this
IOException on windows.
java.lang.RuntimeException: java.io.IOException: The system cannot find the
path specified
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:237)
at
org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:73)
at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:110)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1815) The class HiveResultSet should implement batch fetching.
[ https://issues.apache.org/jira/browse/HIVE-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-1815: --- Attachment: HIVE-1815.1.patch.txt This is the simplest implementation I could do. Just changed the fetchOne to fetchN and return the result on each next() call until the list is empty and then do another fetchN. We've used this for a week and the performance increase on large resultsets is significant. You could also do the fetchN on a different thread to keep the queue full but that's a bit more work for just a little more gain. I've added 1 small test to call the setFetchSize and getFetchSize but the jdbc tests should all work like they worked before this test since the functionality doesn't change. The class HiveResultSet should implement batch fetching. Key: HIVE-1815 URL: https://issues.apache.org/jira/browse/HIVE-1815 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.5.0 Environment: Custom Java application using the Hive JDBC driver to connect to a Hive server, execute a Hive query and process the results. Reporter: Guy le Mar Attachments: HIVE-1815.1.patch.txt When using the Hive JDBC driver, you can execute a Hive query and obtain a HiveResultSet instance that contains the results of the query. Unfortunately, HiveResultSet can then only fetch a single row of these results from the Hive server at a time. As a consequence, it's extremely slow to fetch a resultset of anything other than a trivial size. It would be nice for the HiveResultSet to be able to fetch N rows from the server at a time, so that performance is suitable to support applications that provide human interaction. (From memory, I think it took me around 20 minutes to fetch 4000 rows.) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1815) The class HiveResultSet should implement batch fetching.
[ https://issues.apache.org/jira/browse/HIVE-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-1815: --- Fix Version/s: 0.8.0 Affects Version/s: (was: 0.5.0) 0.8.0 Release Note: Use batch fetching on the hive jdbc driver to increase performance. Status: Patch Available (was: Reopened) The class HiveResultSet should implement batch fetching. Key: HIVE-1815 URL: https://issues.apache.org/jira/browse/HIVE-1815 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.8.0 Environment: Custom Java application using the Hive JDBC driver to connect to a Hive server, execute a Hive query and process the results. Reporter: Guy le Mar Fix For: 0.8.0 Attachments: HIVE-1815.1.patch.txt When using the Hive JDBC driver, you can execute a Hive query and obtain a HiveResultSet instance that contains the results of the query. Unfortunately, HiveResultSet can then only fetch a single row of these results from the Hive server at a time. As a consequence, it's extremely slow to fetch a resultset of anything other than a trivial size. It would be nice for the HiveResultSet to be able to fetch N rows from the server at a time, so that performance is suitable to support applications that provide human interaction. (From memory, I think it took me around 20 minutes to fetch 4000 rows.) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1815) The class HiveResultSet should implement batch fetching.
[ https://issues.apache.org/jira/browse/HIVE-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13007386#comment-13007386 ] Bennie Schut commented on HIVE-1815: https://reviews.apache.org/r/514/ The class HiveResultSet should implement batch fetching. Key: HIVE-1815 URL: https://issues.apache.org/jira/browse/HIVE-1815 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.8.0 Environment: Custom Java application using the Hive JDBC driver to connect to a Hive server, execute a Hive query and process the results. Reporter: Guy le Mar Fix For: 0.8.0 Attachments: HIVE-1815.1.patch.txt When using the Hive JDBC driver, you can execute a Hive query and obtain a HiveResultSet instance that contains the results of the query. Unfortunately, HiveResultSet can then only fetch a single row of these results from the Hive server at a time. As a consequence, it's extremely slow to fetch a resultset of anything other than a trivial size. It would be nice for the HiveResultSet to be able to fetch N rows from the server at a time, so that performance is suitable to support applications that provide human interaction. (From memory, I think it took me around 20 minutes to fetch 4000 rows.) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: HIVE-1815: The class HiveResultSet should implement batch fetching.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/514/ --- Review request for hive. Summary --- HIVE-1815: The class HiveResultSet should implement batch fetching. This addresses bug HIVE-1815. https://issues.apache.org/jira/browse/HIVE-1815 Diffs - trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveQueryResultSet.java 1081785 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveStatement.java 1081785 trunk/jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java 1081785 Diff: https://reviews.apache.org/r/514/diff Testing --- Thanks, Bennie
[jira] Created: (HIVE-2054) Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified
Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified - Key: HIVE-2054 URL: https://issues.apache.org/jira/browse/HIVE-2054 Project: Hive Issue Type: Bug Components: JDBC Reporter: Bennie Schut Priority: Minor It seems something recently changed on the jdbc driver which causes this IOException on windows. java.lang.RuntimeException: java.io.IOException: The system cannot find the path specified at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:237) at org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:73) at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:110) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-2054) Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified
[ https://issues.apache.org/jira/browse/HIVE-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13006899#comment-13006899 ] Bennie Schut commented on HIVE-2054: This seems to happen because we use the same SessionState class the cli is using which is now including some temporary output files and history file references. It's rather trivial to remove the SessionState from the jdbc driver to make it work again (just tried this a few minutes ago). We currently have a JdbcSessionState which extends the SessonState but I don't see a need for the JdbcSessionState either. It seems to be there as a placeholder but is not actually used. Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified - Key: HIVE-2054 URL: https://issues.apache.org/jira/browse/HIVE-2054 Project: Hive Issue Type: Bug Components: JDBC Reporter: Bennie Schut Priority: Minor It seems something recently changed on the jdbc driver which causes this IOException on windows. java.lang.RuntimeException: java.io.IOException: The system cannot find the path specified at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:237) at org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:73) at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:110) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-2054) Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified
[ https://issues.apache.org/jira/browse/HIVE-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-2054: --- Attachment: HIVE-2054.1.patch.txt Removing SessionState so the jdbc works correctly on windows. Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified - Key: HIVE-2054 URL: https://issues.apache.org/jira/browse/HIVE-2054 Project: Hive Issue Type: Bug Components: JDBC Reporter: Bennie Schut Priority: Minor Attachments: HIVE-2054.1.patch.txt It seems something recently changed on the jdbc driver which causes this IOException on windows. java.lang.RuntimeException: java.io.IOException: The system cannot find the path specified at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:237) at org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:73) at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:110) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-2054) Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified
[ https://issues.apache.org/jira/browse/HIVE-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-2054: --- Fix Version/s: 0.8.0 Assignee: Bennie Schut Affects Version/s: 0.8.0 Release Note: Fix for IOException on the jdbc driver on windows. Status: Patch Available (was: Open) https://reviews.apache.org/r/513/ Exception on windows when using the jdbc driver. IOException: The system cannot find the path specified - Key: HIVE-2054 URL: https://issues.apache.org/jira/browse/HIVE-2054 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.8.0 Reporter: Bennie Schut Assignee: Bennie Schut Priority: Minor Fix For: 0.8.0 Attachments: HIVE-2054.1.patch.txt It seems something recently changed on the jdbc driver which causes this IOException on windows. java.lang.RuntimeException: java.io.IOException: The system cannot find the path specified at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:237) at org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:73) at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:110) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: HIVE-2054: fix for IOException on the jdbc driver on windows.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/513/ --- Review request for hive. Summary --- HIVE-2054: fix for IOException on the jdbc driver on windows. This addresses bug HIVE-2054. https://issues.apache.org/jira/browse/HIVE-2054 Diffs - trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 1081782 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java 1081782 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveStatement.java 1081782 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/JdbcSessionState.java 1081782 Diff: https://reviews.apache.org/r/513/diff Testing --- Thanks, Bennie
[jira] Commented: (HIVE-1539) Concurrent metastore threading problem
[ https://issues.apache.org/jira/browse/HIVE-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12978698#action_12978698 ] Bennie Schut commented on HIVE-1539: Are we getting errors like these on HIVE-1862 ? : [junit] Exception: java.lang.RuntimeException: The table default__show_idx_full_idx_comment__ is an index table. Please do drop index instead. [junit] org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: The table default__show_idx_full_idx_comment__ is an index table. Please do drop index instead. Or is this something else? Concurrent metastore threading problem --- Key: HIVE-1539 URL: https://issues.apache.org/jira/browse/HIVE-1539 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: ClassLoaderResolver.patch, HIVE-1539-1.patch, HIVE-1539.patch, thread_dump_hanging.txt When running hive as a service and running a high number of queries concurrently I end up with multiple threads running at 100% cpu without any progress. Looking at these threads I notice this thread(484e): at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:598) But on a different thread(63a2): at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.java) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1815) The class HiveResultSet should implement batch fetching.
[ https://issues.apache.org/jira/browse/HIVE-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12978321#action_12978321 ] Bennie Schut commented on HIVE-1815: Thanks for the comment Edward, fetchN is part of the hive server yes. But the jdbc driver doesn't use this. The HiveQueryResultSet call's a client.fetchOne(); It would be nice if the jdbc driver uses the fetchN you suggested in the comment and perhaps keeps a little queue of records and when dropping below a threshold do another fetchN. I've also noticed the same slowness and it makes sence this was probably cause by using fetchOne(). The class HiveResultSet should implement batch fetching. Key: HIVE-1815 URL: https://issues.apache.org/jira/browse/HIVE-1815 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.5.0 Environment: Custom Java application using the Hive JDBC driver to connect to a Hive server, execute a Hive query and process the results. Reporter: Guy le Mar Fix For: 0.6.0 When using the Hive JDBC driver, you can execute a Hive query and obtain a HiveResultSet instance that contains the results of the query. Unfortunately, HiveResultSet can then only fetch a single row of these results from the Hive server at a time. As a consequence, it's extremely slow to fetch a resultset of anything other than a trivial size. It would be nice for the HiveResultSet to be able to fetch N rows from the server at a time, so that performance is suitable to support applications that provide human interaction. (From memory, I think it took me around 20 minutes to fetch 4000 rows.) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1846) change hive assumption that local mode mappers/reducers always run in same jvm
[ https://issues.apache.org/jira/browse/HIVE-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12970118#action_12970118 ] Bennie Schut commented on HIVE-1846: One of the problems I ran into on local mode was a missing plan. If this is solved perhaps it also solves part of HIVE-1675. change hive assumption that local mode mappers/reducers always run in same jvm --- Key: HIVE-1846 URL: https://issues.apache.org/jira/browse/HIVE-1846 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Assignee: Joydeep Sen Sarma Attachments: HIVE-1846.patch we are trying out a version of hadoop local mode that runs multiple mappers/reducers by spawning jvm's for them. In this mode hive mappers fail in reading the plan file. it seems that we assume (in the setMapredWork call) that local mode mappers/reducers will run in the same jvm (we can cache the current plan in a global var and don't serialize to a path). this needs to get fixed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968759#action_12968759 ] Bennie Schut commented on HIVE-1675: manage to cause this with parallel=false so perhaps not such an interesting angle ;-) I've added some more logging to better understand the cause of this: 2010-12-07 15:49:44,697 INFO exec.Utilities (Utilities.java:getMapRedWork(154)) - Getting jobid:9c2eeba4-a602-4d4b-ba0b-60ce815c4ea7 from cache. 2010-12-07 15:49:44,703 INFO lzo.GPLNativeCodeLoader (GPLNativeCodeLoader.java:clinit(34)) - Loaded native gpl library 2010-12-07 15:49:44,705 INFO lzo.LzoCodec (LzoCodec.java:clinit(72)) - Successfully loaded initialized native-lzo library [hadoop-lzo rev c7acdaa96a7ce04538c0716fe699ffaf11836c70] 2010-12-07 15:49:44,712 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(192)) - Total input paths to process : 1 2010-12-07 15:49:44,880 INFO exec.Utilities (Utilities.java:getMapRedWork(154)) - Getting jobid:e8b2dab2-986a-4bb1-947f-00aec5b46a06 from cache. 2010-12-07 15:49:44,882 INFO exec.ExecDriver (SessionState.java:printInfo(268)) - Job running in-process (local Hadoop) 2010-12-07 15:49:44,882 WARN mapred.LocalJobRunner (LocalJobRunner.java:run(256)) - job_local_0001 java.lang.RuntimeException: java.io.FileNotFoundException: HIVE_PLANe8b2dab2-986a-4bb1-947f-00aec5b46a06 (No such file or directory) at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:166) at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:238) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:244) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:139) Caused by: java.io.FileNotFoundException: HIVE_PLANe8b2dab2-986a-4bb1-947f-00aec5b46a06 (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.init(FileInputStream.java:106) at java.io.FileInputStream.init(FileInputStream.java:66) at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:157) ... 3 more First thing I noticed which is different from a successful job would be that it's trying to get a different jobid from the cache Getting jobid:e8b2dab2-986a-4bb1-947f-00aec5b46a06 from cache I'm still confused. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Fix For: 0.7.0 Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12967142#action_12967142 ] Bennie Schut commented on HIVE-1675: The good news is I'm unable to reproduce this once I set hive.exec.parallel=false. Perhaps it's cleaning uo some things during one of the steps while the other steps still need it? hive.exec.parallel works great for non local stuff for us but the combination of local and parallel seems like it has some issues. I was getting all kinds of different exceptions on local mode like the saxparser but also: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_local_0001/attempt_local_0001_m_00_0/output/file.out in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138) at org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:50) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:193) and: java.io.FileNotFoundException: HIVE_PLAN1fe0a487-2e46-44e4-a557-9b3c9dddac07 (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.init(FileInputStream.java:106) at java.io.FileInputStream.init(FileInputStream.java:66) at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:155) at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:238) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:244) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:139) I only get these issues when enabling parallel and local. At least an angle to look at. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Fix For: 0.7.0 Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut reassigned HIVE-1675: -- Assignee: Bennie Schut SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Work started: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-1675 started by Bennie Schut. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-1675: --- Attachment: HIVE-1675.patch We needed to close the OutputStream (or flush it) before another process can read the data. This fixed the SAXParserException we got. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-1675: --- Fix Version/s: 0.7.0 Status: Patch Available (was: In Progress) SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Fix For: 0.7.0 Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1675) SAXParseException on plan.xml during local mode.
[ https://issues.apache.org/jira/browse/HIVE-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-1675: --- Status: Open (was: Patch Available) doesn't look like it actually got solved. Testing it a bit more. SAXParseException on plan.xml during local mode. Key: HIVE-1675 URL: https://issues.apache.org/jira/browse/HIVE-1675 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Fix For: 0.7.0 Attachments: HIVE-1675.patch, local_10005_plan.xml, local_10006_plan.xml When hive switches to local mode (hive.exec.mode.local.auto=true) I receive a sax parser exception on the plan.xml If I set hive.exec.mode.local.auto=false I get the correct results. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1539) Concurrent metastore threading problem
[ https://issues.apache.org/jira/browse/HIVE-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-1539: --- Attachment: HIVE-1539.patch 2.2.0-m2 can now be found on maven so the previous patch can be discarded and you can simple update the ivy/libraries/properties file to use datanucleus 2.2.0-m2. since -m2 is a milestone release I'm not sure if we would want to include this or wait for 2.2.0 release. Concurrent metastore threading problem --- Key: HIVE-1539 URL: https://issues.apache.org/jira/browse/HIVE-1539 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: ClassLoaderResolver.patch, HIVE-1539.patch, thread_dump_hanging.txt When running hive as a service and running a high number of queries concurrently I end up with multiple threads running at 100% cpu without any progress. Looking at these threads I notice this thread(484e): at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:598) But on a different thread(63a2): at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.java) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
