[jira] [Created] (HIVE-6950) Parsing Error in GROUPING SETS
Rohit Agarwal created HIVE-6950:
---
Summary: Parsing Error in GROUPING SETS
Key: HIVE-6950
URL: https://issues.apache.org/jira/browse/HIVE-6950
Project: Hive
Issue Type: Bug
Reporter: Rohit Agarwal
The following query:
{code}
SELECT tab1.a,
tab1.b,
SUM(tab1.c)
FROM tab1
GROUP BY tab1.a,
tab1.b
GROUPING SETS ((tab1.a, tab1.b))
{code}
results in the following error:
{code}
parseexception line 7:22 missing ) at ',' near '' line 7:31 extraneous input
')' expecting eof near ''
{code}
Changing the query to:
{code}
SELECT tab1.a,
tab1.b,
SUM(tab1.c)
FROM tab1
GROUP BY tab1.a,
tab1.b
GROUPING SETS ((a, tab1.b))
{code}
makes it work.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6950) Parsing Error in GROUPING SETS
[
https://issues.apache.org/jira/browse/HIVE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohit Agarwal updated HIVE-6950:
Description:
The following query:
{code}
SELECT tab1.a,
tab1.b,
SUM(tab1.c)
FROM tab1
GROUP BY tab1.a,
tab1.b
GROUPING SETS ((tab1.a, tab1.b))
{code}
results in the following error:
{code}
ParseException line 7:22 missing ) at ',' near 'EOF'
line 7:31 extraneous input ')' expecting EOF near 'EOF'
{code}
Changing the query to:
{code}
SELECT tab1.a,
tab1.b,
SUM(tab1.c)
FROM tab1
GROUP BY tab1.a,
tab1.b
GROUPING SETS ((a, tab1.b))
{code}
makes it work.
was:
The following query:
{code}
SELECT tab1.a,
tab1.b,
SUM(tab1.c)
FROM tab1
GROUP BY tab1.a,
tab1.b
GROUPING SETS ((tab1.a, tab1.b))
{code}
results in the following error:
{code}
parseexception line 7:22 missing ) at ',' near '' line 7:31 extraneous input
')' expecting eof near ''
{code}
Changing the query to:
{code}
SELECT tab1.a,
tab1.b,
SUM(tab1.c)
FROM tab1
GROUP BY tab1.a,
tab1.b
GROUPING SETS ((a, tab1.b))
{code}
makes it work.
Parsing Error in GROUPING SETS
--
Key: HIVE-6950
URL: https://issues.apache.org/jira/browse/HIVE-6950
Project: Hive
Issue Type: Bug
Reporter: Rohit Agarwal
The following query:
{code}
SELECT tab1.a,
tab1.b,
SUM(tab1.c)
FROM tab1
GROUP BY tab1.a,
tab1.b
GROUPING SETS ((tab1.a, tab1.b))
{code}
results in the following error:
{code}
ParseException line 7:22 missing ) at ',' near 'EOF'
line 7:31 extraneous input ')' expecting EOF near 'EOF'
{code}
Changing the query to:
{code}
SELECT tab1.a,
tab1.b,
SUM(tab1.c)
FROM tab1
GROUP BY tab1.a,
tab1.b
GROUPING SETS ((a, tab1.b))
{code}
makes it work.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6944) WebHCat e2e tests broken by HIVE-6432
[ https://issues.apache.org/jira/browse/HIVE-6944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976457#comment-13976457 ] Szehon Ho commented on HIVE-6944: - Sorry about that, looks like a mistake on the new Jenkins, I just fixed the flag 'check no precommit tests=true' on the default job configuration, hopefully new jobs in the queue will now check for NO PRECOMMIT TESTS string (though jobs already queued will still have this issue). Thanks for putting it. WebHCat e2e tests broken by HIVE-6432 - Key: HIVE-6944 URL: https://issues.apache.org/jira/browse/HIVE-6944 Project: Hive Issue Type: Bug Components: WebHCat Affects Versions: 0.14.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Attachments: HIVE-6944.patch HIVE-6432 removed templeton/v/queue REST endpoint and broke webhcat e2e tests NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6932) hive README needs update
[
https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976493#comment-13976493
]
Lefty Leverenz commented on HIVE-6932:
--
Nano-nit: an extra space before your own upgrade script.
{noformat}
+- We have provided upgrade scripts for MySQL, PostgreSQL, Oracle,
+ Microsoft SQL Server, and Derby databases. If you are using a
+ different database for your MetaStore you will need to provide
+ your own upgrade script.
{noformat}
hive README needs update
Key: HIVE-6932
URL: https://issues.apache.org/jira/browse/HIVE-6932
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Attachments: HIVE-6932.1.patch
It needs to be updated to include Tez as a runtime. Also, it talks about
average latency being in minutes, which is very misleading.
NO PRECOMMIT TESTS
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6932) hive README needs update
[
https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976507#comment-13976507
]
Lefty Leverenz commented on HIVE-6932:
--
Nano-nit #2: need another space before frameworks.
{noformat}
+* Query execution via Apache Hadoop MapReduce and using Apache Tez
+ frameworks.
{noformat}
Also a question: Is QL the proper name for the language or HiveQL? The
README says HiveQL twice in the Getting Started section. The wiki is
inconsistent with QL, HiveQL, and Hive QL.
hive README needs update
Key: HIVE-6932
URL: https://issues.apache.org/jira/browse/HIVE-6932
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Attachments: HIVE-6932.1.patch
It needs to be updated to include Tez as a runtime. Also, it talks about
average latency being in minutes, which is very misleading.
NO PRECOMMIT TESTS
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-2621) Allow multiple group bys with the same input data and spray keys to be run on the same reducer.
[ https://issues.apache.org/jira/browse/HIVE-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976525#comment-13976525 ] Lefty Leverenz commented on HIVE-2621: -- Asking again: Is the definition of *hive.multigroupby.singlereducer* correct or was it just held over from *hive.multigroupby.singlemr*? (See previous comment.) Allow multiple group bys with the same input data and spray keys to be run on the same reducer. --- Key: HIVE-2621 URL: https://issues.apache.org/jira/browse/HIVE-2621 Project: Hive Issue Type: New Feature Reporter: Kevin Wilfong Assignee: Kevin Wilfong Fix For: 0.9.0 Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2621.D567.1.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2621.D567.2.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2621.D567.3.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2621.D567.4.patch, HIVE-2621.1.patch.txt Currently, when a user runs a query, such as a multi-insert, where each insertion subclause consists of a simple query followed by a group by, the group bys for each clause are run on a separate reducer. This requires writing the data for each group by clause to an intermediate file, and then reading it back. This uses a significant amount of the total CPU consumed by the query for an otherwise simple query. If the subclauses are grouped by their distinct expressions and group by keys, with all of the group by expressions for a group of subclauses run on a single reducer, this would reduce the amount of reading/writing to intermediate files for some queries. To do this, for each group of subclauses, in the mapper we would execute a the filters for each subclause 'or'd together (provided each subclause has a filter) followed by a reduce sink. In the reducer, the child operators would be each subclauses filter followed by the group by and any subsequent operations. Note that this would require turning off map aggregation, so we would need to make using this type of plan configurable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5538) Turn on vectorization by default.
[
https://issues.apache.org/jira/browse/HIVE-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976533#comment-13976533
]
Hive QA commented on HIVE-5538:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12641155/HIVE-5538.3.patch
{color:red}ERROR:{color} -1 due to 71 failed/errored test(s), 5417 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_diff_part_cols
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_diff_part_cols2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_predicate_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_create
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partInit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_quote1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
[jira] [Created] (HIVE-6952) Hive 0.13 HiveOutputFormat breaks backwards compatibility
Costin Leau created HIVE-6952: - Summary: Hive 0.13 HiveOutputFormat breaks backwards compatibility Key: HIVE-6952 URL: https://issues.apache.org/jira/browse/HIVE-6952 Project: Hive Issue Type: Bug Components: File Formats, Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Costin Leau Priority: Blocker Hive 0.13 changed the signature of HiveOutputFormat (through commit r1527149) breaking backwards compatibility with previous releases; the return type of getHiveRecordWriter has been changed from RecordWriter to FSRecordWriter. FSRecordWriter introduces one new method on top of RecordWriter however it does not extend the previous interface and it lives in a completely new package. Thus code running fine on Hive 0.12 breaks on Hive 0.13. After the upgrade, code running on HIve 0.13, will break on anything lower than this. This could have easily been avoided by extending the existing interface or introducing a new one that RecordWriter could have extended going forward. By changing the signature, the existing contract (and compatibility) has been voided. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6951) Hive 0.13 HiveOutputFormat breaks backwards compatibility
Costin Leau created HIVE-6951: - Summary: Hive 0.13 HiveOutputFormat breaks backwards compatibility Key: HIVE-6951 URL: https://issues.apache.org/jira/browse/HIVE-6951 Project: Hive Issue Type: Bug Components: File Formats, Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Costin Leau Priority: Blocker Hive 0.13 changed the signature of HiveOutputFormat (through commit r1527149) breaking backwards compatibility with previous releases; the return type of getHiveRecordWriter has been changed from RecordWriter to FSRecordWriter. FSRecordWriter introduces one new method on top of RecordWriter however it does not extend the previous interface and it lives in a completely new package. Thus code running fine on Hive 0.12 breaks on Hive 0.13. After the upgrade, code running on HIve 0.13, will break on anything lower than this. This could have easily been avoided by extending the existing interface or introducing a new one that RecordWriter could have extended going forward. By changing the signature, the existing contract (and compatibility) has been voided. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6952) Hive 0.13 HiveOutputFormat breaks backwards compatibility
[ https://issues.apache.org/jira/browse/HIVE-6952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976591#comment-13976591 ] Costin Leau commented on HIVE-6952: --- Actually taking a closer look indicates that FSRecordWriter and RecordWriter are identical - they both implement the same two methods, with the _exact_ same signature which makes the breaking change even more puzzling. Hive 0.13 HiveOutputFormat breaks backwards compatibility - Key: HIVE-6952 URL: https://issues.apache.org/jira/browse/HIVE-6952 Project: Hive Issue Type: Bug Components: File Formats, Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Costin Leau Priority: Blocker Hive 0.13 changed the signature of HiveOutputFormat (through commit r1527149) breaking backwards compatibility with previous releases; the return type of getHiveRecordWriter has been changed from RecordWriter to FSRecordWriter. FSRecordWriter introduces one new method on top of RecordWriter however it does not extend the previous interface and it lives in a completely new package. Thus code running fine on Hive 0.12 breaks on Hive 0.13. After the upgrade, code running on HIve 0.13, will break on anything lower than this. This could have easily been avoided by extending the existing interface or introducing a new one that RecordWriter could have extended going forward. By changing the signature, the existing contract (and compatibility) has been voided. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: 49 config params without descriptions
Found two more from HIVE-5522https://issues.apache.org/jira/browse/HIVE-5522 (also HIVE-6098 https://issues.apache.org/jira/browse/HIVE-6098, Merge Tez branch into trunk) so the current total is 51 configs that don't have descriptions in 0.13.0: *Release 13 * hive.compute.splits.in.am hive.rpc.query.plan But these both have comments in HiveConf.java that can be used as descriptions, although they aren't included in hive-default.xml.template. I missed them because I was working from the patch for HIVE-6037https://issues.apache.org/jira/browse/HIVE-6037 and Navis had used the HiveConf comments for descriptions. (That means there could be more parameters missing from the 0.13.0 template file.) -- Lefty On Mon, Apr 14, 2014 at 1:53 AM, Lefty Leverenz leftylever...@gmail.comwrote: Here's a list of 49 configuration parameters in RC0 (and trunk) that don't have descriptions in hive-default.xml.template: *Release 1 or 2 * hive.exec.submitviachild hive.metastore.metadb.dir hive.jar.path hive.aux.jars.path hive.table.name hive.partition.name hive.alias *Release 3 * hive.cli.errors.ignore *Release 4 * hive.added.files.path hive.added.jars.path *Release 5 * hive.intermediate.compression.codec hive.intermediate.compression.type hive.added.archives.path *Release 6 * hive.metastore.archive.intermediate.archived hive.metastore.archive.intermediate.extracted hive.mapred.partitioner hive.exec.script.trust hive.hadoop.supports.splittable.combineinputformat *Release 7 * hive.lockmgr.zookeeper.default.partition.name hive.metastore.fs.handler.class hive.query.result.fileformat hive.hashtable.initialCapacity hive.hashtable.loadfactor hive.debug.localtask hive.lock.manager hive.outerjoin.supports.filters hive.semantic.analyzer.hook *Release 8 * hive.exec.job.debug.timeout hive.exec.tasklog.debug.timeout hive.merge.rcfile.block.level hive.merge.input.format.block.level hive.merge.current.job.has.dynamic.partitions hive.stats.collect.rawdatasize *Release 8.1 * hive.optimize.metadataonly *Release 9 * *Release 10 * *Release 11 * hive.exec.rcfile.use.sync.cache hive.stats.key.prefix--- *internal* *Release 12 * hive.scratch.dir.permission datanucleus.fixedDatastore datanucleus.rdbms.useLegacyNativeValueStrategy hive.optimize.sampling.orderby --- *internal?* hive.optimize.sampling.orderby.number hive.optimize.sampling.orderby.percent hive.server2.authentication.ldap.Domain hive.server2.session.hook hive.typecheck.on.insert *Release 13 * hive.metastore.expression.proxy hive.txn.manager hive.stageid.rearrange hive.explain.dependency.append.tasktype What's the best way to deal with these? 1. Ignore them (or identify those that can be ignored). 2. Add some descriptions in Hive 0.13.0 RC1. 3. Deal with them after HIVE-6037https://issues.apache.org/jira/browse/HIVE-6037gets committed. - Try to cover all of them by Hive 0.14.0: - Put the list in a JIRA and create a common HiveConf.java patch, which can be appended until release 0.14.0 is ready. - Accumulate descriptions in JIRA comments, then create a patch from the comments. - Deal with them as soon as possible: - Put the list in an umbrella JIRA and use sub-task JIRAs to add descriptions individually or in small groups. 4. Deal with them in the wiki, then patch HiveConf.java before release 0.14.0. 5. [Your idea goes here.] -- Lefty
[jira] [Commented] (HIVE-6947) More fixes for tests on hadoop-2
[
https://issues.apache.org/jira/browse/HIVE-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976633#comment-13976633
]
Hive QA commented on HIVE-6947:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12641157/HIVE-6947.patch
{color:red}ERROR:{color} -1 due to 53 failed/errored test(s), 5483 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_count
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_load_dyn_part1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union7
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/3/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/3/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 53 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12641157
More fixes
[jira] [Created] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
Amareshwari Sriramadasu created HIVE-6953:
-
Summary: All CompactorTest failing with Table/View 'NEXT_TXN_ID'
does not exist
Key: HIVE-6953
URL: https://issues.apache.org/jira/browse/HIVE-6953
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.0
Reporter: Amareshwari Sriramadasu
When I'm running all tests through the command 'mvn clean install -Phadoop-1',
all CompactorTest classes TestInitiator, TestWorker, TestCleaner fail with
following exception :
{noformat}
org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from
transaction database java.sql.SQLSyntaxErrorException: Table/View 'NEXT_TXN_ID'
does not exist.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
{noformat}
This is happening on branch-0.13. Has anyone faced this problem?
[~owen.omalley] or someone else help me solve this. Do i have to set anything?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
[
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976740#comment-13976740
]
Amareshwari Sriramadasu commented on HIVE-6953:
---
There are no failures in trunk, all tests pass. [~rhbutani], do you think
anything is missing in branch-0.13? Looking at commits, i couldnt figure out.
All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
--
Key: HIVE-6953
URL: https://issues.apache.org/jira/browse/HIVE-6953
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.0
Reporter: Amareshwari Sriramadasu
When I'm running all tests through the command 'mvn clean install
-Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner
fail with following exception :
{noformat}
org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from
transaction database java.sql.SQLSyntaxErrorException: Table/View
'NEXT_TXN_ID' does not exist.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
{noformat}
This is happening on branch-0.13. Has anyone faced this problem?
[~owen.omalley] or someone else help me solve this. Do i have to set anything?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-6954) After ALTER FILEFORMAT, DESCRIBE throwing exception
Chinna Rao Lalam created HIVE-6954:
--
Summary: After ALTER FILEFORMAT, DESCRIBE throwing exception
Key: HIVE-6954
URL: https://issues.apache.org/jira/browse/HIVE-6954
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.14.0
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
After ALTER TABLE FILEFORMAT, DESCRIBE EXTENDS TABLE throwing exception.
{quote}
CREATE TABLE alter_test ( id int, str string,mp MAPSTRING,STRING,lst
ARRAYSTRING,strct STRUCTA:STRING,B:STRING) PARTITIONED BY (part string)
STORED AS TEXTFILE;
DESCRIBE EXTENDED alter_test;
ALTER TABLE alter_test SET FILEFORMAT PARQUET;
DESCRIBE EXTENDED alter_test;
SELECT * FROM alter_test;
{quote}
hive DESCRIBE EXTENDED alter_test;
FAILED: SemanticException : Table not found alter_test
hive SELECT * FROM alter_test;
FAILED: SemanticException must specify an InputFormat class
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6927) Add support for MSSQL in schematool
[
https://issues.apache.org/jira/browse/HIVE-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976762#comment-13976762
]
Hive QA commented on HIVE-6927:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12640686/HIVE-6927.patch
{color:red}ERROR:{color} -1 due to 58 failed/errored test(s), 5483 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_count
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_load_dyn_part1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union7
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testNameMethods
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testPartition
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/4/testReport
Console output:
Re: Hive Contributor
Thank you Ashutosh. On Tue, Apr 22, 2014 at 12:44 AM, Ashutosh Chauhan hashut...@apache.orgwrote: Welcome aboard, Naveen! I have added you as contributor to project. Looking forward to your contributions to Hive. Ashutosh On Mon, Apr 21, 2014 at 7:18 PM, Naveen Gangam ngan...@cloudera.comwrote: Dear Hive PMC, I would like to contribute to the HIVE community. Could you please grant me the contributor role? My apache username is ngangam. Thank you in advance and I am looking forward to becoming a part of the Hive community. -- Thanks, Naveen :) -- Thanks, Naveen :)
[jira] [Commented] (HIVE-2540) LATERAL VIEW with EXPLODE produces ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976820#comment-13976820
]
Lars Francke commented on HIVE-2540:
We're still seeing this issue with Hive 0.12 (slightly patched as it's the one
from CDH5):
{quote}
2014-04-22 16:18:50,304 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
{product_id:140719,active:null,avg_product_price:null,base_cats:[[1247,818,821,902,905],[1247,818,821,902,906],[1247,818,821,902,909]]}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row
{product_id:140719,active:null,avg_product_price:null,base_cats:[[1247,818,821,902,905],[1247,818,821,902,906],[1247,818,821,902,909]]}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
... 8 more
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
at java.util.ArrayList$Itr.next(ArrayList.java:831)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFExplode.process(GenericUDTFExplode.java:92)
at
org.apache.hadoop.hive.ql.exec.UDTFOperator.processOp(UDTFOperator.java:113)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:844)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:844)
at
org.apache.hadoop.hive.ql.exec.LateralViewForwardOperator.processOp(LateralViewForwardOperator.java:37)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:844)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:844)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:519)
... 9 more
{quote}
I haven't had time to check whether this patch is actually included but I'd
expect it to be. Shall I open a new issue?
LATERAL VIEW with EXPLODE produces ConcurrentModificationException
--
Key: HIVE-2540
URL: https://issues.apache.org/jira/browse/HIVE-2540
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.7.1, 0.9.0
Reporter: David Phillips
Assignee: Navis
Fix For: 0.10.0
Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2540.D2805.1.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2540.D2805.2.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2540.D2805.3.patch
The following produces {{ConcurrentModificationException}} on the {{for}}
loop inside EXPLODE:
{code}
create table foo as select array(1, 2) a from src limit 1;
select a, x.b from foo lateral view explode(a) x as b;
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HIVE-6893) out of sequence error in HiveMetastore server
[
https://issues.apache.org/jira/browse/HIVE-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam reassigned HIVE-6893:
---
Assignee: Naveen Gangam
out of sequence error in HiveMetastore server
-
Key: HIVE-6893
URL: https://issues.apache.org/jira/browse/HIVE-6893
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Reporter: Romain Rigaux
Assignee: Naveen Gangam
Calls listing databases or tables fail. It seems to be a concurrency problem.
{code}
014-03-06 05:34:00,785 ERROR hive.log:
org.apache.thrift.TApplicationException: get_databases failed: out of
sequence response
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:472)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:459)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:648)
at
org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:66)
at
org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:278)
at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:582)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57)
at com.sun.proxy.$Proxy9.getSchemas(Unknown Source)
at
org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:192)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:263)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6927) Add support for MSSQL in schematool
[ https://issues.apache.org/jira/browse/HIVE-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6927: --- Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Failures are hadoop-2 related. Committed to trunk. Thanks, Deepesh! Add support for MSSQL in schematool --- Key: HIVE-6927 URL: https://issues.apache.org/jira/browse/HIVE-6927 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0 Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal Fix For: 0.14.0 Attachments: HIVE-6927.patch Schematool is the preferred way of initializing schema for Hive. Since HIVE-6862 provided the script for MSSQL it would be nice to add the support for it in schematool. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
[
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates reassigned HIVE-6953:
Assignee: Alan Gates
All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
--
Key: HIVE-6953
URL: https://issues.apache.org/jira/browse/HIVE-6953
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.0
Reporter: Amareshwari Sriramadasu
Assignee: Alan Gates
When I'm running all tests through the command 'mvn clean install
-Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner
fail with following exception :
{noformat}
org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from
transaction database java.sql.SQLSyntaxErrorException: Table/View
'NEXT_TXN_ID' does not exist.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
{noformat}
This is happening on branch-0.13. Has anyone faced this problem?
[~owen.omalley] or someone else help me solve this. Do i have to set anything?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6932) hive README needs update
[ https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6932: Attachment: HIVE-6932.2.patch hive README needs update Key: HIVE-6932 URL: https://issues.apache.org/jira/browse/HIVE-6932 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-6932.1.patch, HIVE-6932.2.patch It needs to be updated to include Tez as a runtime. Also, it talks about average latency being in minutes, which is very misleading. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6932) hive README needs update
[ https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6932: Attachment: (was: HIVE-6932.2.patch) hive README needs update Key: HIVE-6932 URL: https://issues.apache.org/jira/browse/HIVE-6932 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-6932.1.patch It needs to be updated to include Tez as a runtime. Also, it talks about average latency being in minutes, which is very misleading. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6932) hive README needs update
[ https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6932: Attachment: README.txt hive README needs update Key: HIVE-6932 URL: https://issues.apache.org/jira/browse/HIVE-6932 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-6932.1.patch It needs to be updated to include Tez as a runtime. Also, it talks about average latency being in minutes, which is very misleading. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6932) hive README needs update
[ https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6932: Attachment: (was: README.txt) hive README needs update Key: HIVE-6932 URL: https://issues.apache.org/jira/browse/HIVE-6932 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-6932.1.patch It needs to be updated to include Tez as a runtime. Also, it talks about average latency being in minutes, which is very misleading. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6932) hive README needs update
[ https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6932: Attachment: HIVE-6932.2.patch [~leftylev] Updated patch addressing the comments. Also changed the wording regarding query execution. hive README needs update Key: HIVE-6932 URL: https://issues.apache.org/jira/browse/HIVE-6932 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-6932.1.patch, HIVE-6932.2.patch It needs to be updated to include Tez as a runtime. Also, it talks about average latency being in minutes, which is very misleading. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6932) hive README needs update
[
https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977082#comment-13977082
]
Hive QA commented on HIVE-6932:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12641158/HIVE-6932.1.patch
{color:red}ERROR:{color} -1 due to 43 failed/errored test(s), 5416 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/5/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/5/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 43 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12641158
hive README needs update
Key: HIVE-6932
URL: https://issues.apache.org/jira/browse/HIVE-6932
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Attachments: HIVE-6932.1.patch, HIVE-6932.2.patch
It needs to be updated to include Tez as a runtime. Also, it talks about
average latency being in minutes, which is very misleading.
NO PRECOMMIT TESTS
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
[
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977087#comment-13977087
]
Alan Gates commented on HIVE-6953:
--
[~amareshwari], I can't reproduce this. Tests pass for me in 0.13 and trunk.
These tests should be creating the tables they need as part of the tests (see
TxnHandler.checkQFileTestHack()). Is there anything in your logs indicating it
tried to create the tables and failed? Are you doing anything in your build to
turn off the hive.in.test config value? This would prevent the tables being
created.
All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
--
Key: HIVE-6953
URL: https://issues.apache.org/jira/browse/HIVE-6953
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.0
Reporter: Amareshwari Sriramadasu
Assignee: Alan Gates
When I'm running all tests through the command 'mvn clean install
-Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner
fail with following exception :
{noformat}
org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from
transaction database java.sql.SQLSyntaxErrorException: Table/View
'NEXT_TXN_ID' does not exist.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
{noformat}
This is happening on branch-0.13. Has anyone faced this problem?
[~owen.omalley] or someone else help me solve this. Do i have to set anything?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6945) issues with dropping partitions on Oracle
[ https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-6945: --- Description: 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is translated by DN into decimal rather than long. This appears to be specific to some cases because it seemed to have worked before (different version of Oracle? JDBC? DN? Maybe depends on whether db was auto-created). 2) When partition dropping code falls back to JDO, it creates objects to return, then drops partitions. It appears that dropping makes DN objects invalid. We create metastore partition objects out of DN objects before drop, however the list of partition column values is re-used, rather than copied, into these. DN appears to clear this list during drop, so the returned object becomes invalid and the exception is thrown. issues with dropping partitions on Oracle - Key: HIVE-6945 URL: https://issues.apache.org/jira/browse/HIVE-6945 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is translated by DN into decimal rather than long. This appears to be specific to some cases because it seemed to have worked before (different version of Oracle? JDBC? DN? Maybe depends on whether db was auto-created). 2) When partition dropping code falls back to JDO, it creates objects to return, then drops partitions. It appears that dropping makes DN objects invalid. We create metastore partition objects out of DN objects before drop, however the list of partition column values is re-used, rather than copied, into these. DN appears to clear this list during drop, so the returned object becomes invalid and the exception is thrown. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6945) issues with dropping partitions on Oracle
[ https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977109#comment-13977109 ] Sergey Shelukhin commented on HIVE-6945: Updated issues with dropping partitions on Oracle - Key: HIVE-6945 URL: https://issues.apache.org/jira/browse/HIVE-6945 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is translated by DN into decimal rather than long. This appears to be specific to some cases because it seemed to have worked before (different version of Oracle? JDBC? DN? Maybe depends on whether db was auto-created). 2) When partition dropping code falls back to JDO, it creates objects to return, then drops partitions. It appears that dropping makes DN objects invalid. We create metastore partition objects out of DN objects before drop, however the list of partition column values is re-used, rather than copied, into these. DN appears to clear this list during drop, so the returned object becomes invalid and the exception is thrown. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977120#comment-13977120 ] Ashutosh Chauhan commented on HIVE-5771: Cool. As you can see from last test run on .8 patch, only handful of failures are remaining which needs to be looked into it. Constant propagation optimizer for Hive --- Key: HIVE-5771 URL: https://issues.apache.org/jira/browse/HIVE-5771 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Ted Xu Assignee: Ted Xu Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.patch Currently there is no constant folding/propagation optimizer, all expressions are evaluated at runtime. HIVE-2470 did a great job on evaluating constants on UDF initializing phase, however, it is still a runtime evaluation and it doesn't propagate constants from a subquery to outside. It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6939) TestExecDriver.testMapRedPlan3 fails on hadoop-2
[ https://issues.apache.org/jira/browse/HIVE-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6939: --- Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Committed to trunk. TestExecDriver.testMapRedPlan3 fails on hadoop-2 Key: HIVE-6939 URL: https://issues.apache.org/jira/browse/HIVE-6939 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Fix For: 0.14.0 Attachments: HIVE-6939.1.patch Passes on hadoop-1, but fails on hadoop-2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HIVE-6951) Hive 0.13 HiveOutputFormat breaks backwards compatibility
[ https://issues.apache.org/jira/browse/HIVE-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-6951. Resolution: Fixed Dupe of HIVE-6952 Hive 0.13 HiveOutputFormat breaks backwards compatibility - Key: HIVE-6951 URL: https://issues.apache.org/jira/browse/HIVE-6951 Project: Hive Issue Type: Bug Components: File Formats, Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Costin Leau Priority: Blocker Hive 0.13 changed the signature of HiveOutputFormat (through commit r1527149) breaking backwards compatibility with previous releases; the return type of getHiveRecordWriter has been changed from RecordWriter to FSRecordWriter. FSRecordWriter introduces one new method on top of RecordWriter however it does not extend the previous interface and it lives in a completely new package. Thus code running fine on Hive 0.12 breaks on Hive 0.13. After the upgrade, code running on HIve 0.13, will break on anything lower than this. This could have easily been avoided by extending the existing interface or introducing a new one that RecordWriter could have extended going forward. By changing the signature, the existing contract (and compatibility) has been voided. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HIVE-6951) Hive 0.13 HiveOutputFormat breaks backwards compatibility
[ https://issues.apache.org/jira/browse/HIVE-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-6951. Resolution: Duplicate Hive 0.13 HiveOutputFormat breaks backwards compatibility - Key: HIVE-6951 URL: https://issues.apache.org/jira/browse/HIVE-6951 Project: Hive Issue Type: Bug Components: File Formats, Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Costin Leau Priority: Blocker Hive 0.13 changed the signature of HiveOutputFormat (through commit r1527149) breaking backwards compatibility with previous releases; the return type of getHiveRecordWriter has been changed from RecordWriter to FSRecordWriter. FSRecordWriter introduces one new method on top of RecordWriter however it does not extend the previous interface and it lives in a completely new package. Thus code running fine on Hive 0.12 breaks on Hive 0.13. After the upgrade, code running on HIve 0.13, will break on anything lower than this. This could have easily been avoided by extending the existing interface or introducing a new one that RecordWriter could have extended going forward. By changing the signature, the existing contract (and compatibility) has been voided. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (HIVE-6951) Hive 0.13 HiveOutputFormat breaks backwards compatibility
[ https://issues.apache.org/jira/browse/HIVE-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reopened HIVE-6951: Hive 0.13 HiveOutputFormat breaks backwards compatibility - Key: HIVE-6951 URL: https://issues.apache.org/jira/browse/HIVE-6951 Project: Hive Issue Type: Bug Components: File Formats, Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Costin Leau Priority: Blocker Hive 0.13 changed the signature of HiveOutputFormat (through commit r1527149) breaking backwards compatibility with previous releases; the return type of getHiveRecordWriter has been changed from RecordWriter to FSRecordWriter. FSRecordWriter introduces one new method on top of RecordWriter however it does not extend the previous interface and it lives in a completely new package. Thus code running fine on Hive 0.12 breaks on Hive 0.13. After the upgrade, code running on HIve 0.13, will break on anything lower than this. This could have easily been avoided by extending the existing interface or introducing a new one that RecordWriter could have extended going forward. By changing the signature, the existing contract (and compatibility) has been voided. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6756) alter table set fileformat should set serde too
[ https://issues.apache.org/jira/browse/HIVE-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977139#comment-13977139 ] Chinna Rao Lalam commented on HIVE-6756: With out the patch, current code is taken care for the RC,ORC and PARQUET file formats (ALTER TATBLE SET FILEFORMT configuring the proper serde for RC,ORC and PARQUET file formats) TEXTFILE, SEQUENCE file formats are not handled. This patch will address by configuring LazySimpleSerde for these file formats. Apart from this in ALTER TATBLE SET FILEFORMT can use INPUTFORMAT,OUTPUTFORMAT classes. In this scenario not sure which serde need to be configure? If throws exception he cannot use INPUTFORMAT,OUTPUTFORMAT classes in ALTER TATBLE SET FILEFORMT. Any suggestions.. alter table set fileformat should set serde too --- Key: HIVE-6756 URL: https://issues.apache.org/jira/browse/HIVE-6756 Project: Hive Issue Type: Bug Reporter: Owen O'Malley Assignee: Chinna Rao Lalam Attachments: HIVE-6756.patch Currently doing alter table set fileformat doesn't change the serde. This is unexpected by customers because the serdes are largely file format specific. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6955) ExprNodeColDesc isSame doesn't account for tabAlias: this affects trait Propagation in Joins
Harish Butani created HIVE-6955:
---
Summary: ExprNodeColDesc isSame doesn't account for tabAlias: this
affects trait Propagation in Joins
Key: HIVE-6955
URL: https://issues.apache.org/jira/browse/HIVE-6955
Project: Hive
Issue Type: Bug
Reporter: Harish Butani
For tpcds Q15:
{code}
explain
select ca_zip, sum(cs_sales_price)
from catalog_sales, customer, customer_address, date_dim
where catalog_sales.cs_bill_customer_sk = customer.c_customer_sk
and customer.c_current_addr_sk = customer_address.ca_address_sk
and (substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475',
'85392', '85460', '80348', '81792')
or ca_state in ('CA','WA','GA')
or cs_sales_price 500)
and catalog_sales.cs_sold_date_sk = date_dim.d_date_sk
and d_qoy = 2 and d_year = 2001
group by ca_zip
order by ca_zip
limit 100;
{code}
The Traits setup for the Operators are:
{code}
FIL[23]: bucketCols=[[]],numBuckets=-1
RS[11]: bucketCols=[[VALUE._col0]],numBuckets=-1
JOIN[12]: bucketCols=[[_col71], [_col71]],numBuckets=-1
FIL[13]: bucketCols=[[_col71], [_col71]],numBuckets=-1
SEL[14]: bucketCols=[[_col71], [_col71]],numBuckets=-1
GBY[15]: bucketCols=[[_col0]],numBuckets=-1
RS[16]: bucketCols=[[KEY._col0]],numBuckets=-1
GBY[17]: bucketCols=[[_col0]],numBuckets=-1
SEL[18]: bucketCols=[[_col0]],numBuckets=-1
LIM[21]: bucketCols=[[_col0]],numBuckets=-1
FS[22]: bucketCols=[[_col0]],numBuckets=-1
TS[3]: bucketCols=[[]],numBuckets=-1
RS[5]: bucketCols=[[VALUE._col0]],numBuckets=-1
JOIN[6]: bucketCols=[[_col3], [_col36]],numBuckets=-1
RS[7]: bucketCols=[[VALUE._col40]],numBuckets=-1
JOIN[9]: bucketCols=[[_col40], [_col0]],numBuckets=-1
RS[10]: bucketCols=[[VALUE._col0]],numBuckets=-1
TS[1]: bucketCols=[[]],numBuckets=-1
RS[8]: bucketCols=[[VALUE._col0]],numBuckets=-1
TS[0]: bucketCols=[[]],numBuckets=-1
RS[4]: bucketCols=[[VALUE._col3]],numBuckets=-1
{code}
This is incorrect:
Join[9] joins ca join (cs join cust). In this case both sides of join have a
'_col0' column. The reverse mapping of trait propagation relies on
ExprNodeColumnDesc.isSame; since this doesn't account for the tabAlias we end
up with Join[9] being bucketed on cs_sold_date_sk; Join[12] has the same issue,
only compounds the error.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HIVE-6955) ExprNodeColDesc isSame doesn't account for tabAlias: this affects trait Propagation in Joins
[
https://issues.apache.org/jira/browse/HIVE-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Butani reassigned HIVE-6955:
---
Assignee: Harish Butani
ExprNodeColDesc isSame doesn't account for tabAlias: this affects trait
Propagation in Joins
Key: HIVE-6955
URL: https://issues.apache.org/jira/browse/HIVE-6955
Project: Hive
Issue Type: Bug
Reporter: Harish Butani
Assignee: Harish Butani
For tpcds Q15:
{code}
explain
select ca_zip, sum(cs_sales_price)
from catalog_sales, customer, customer_address, date_dim
where catalog_sales.cs_bill_customer_sk = customer.c_customer_sk
and customer.c_current_addr_sk = customer_address.ca_address_sk
and (substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475',
'85392', '85460', '80348', '81792')
or ca_state in ('CA','WA','GA')
or cs_sales_price 500)
and catalog_sales.cs_sold_date_sk = date_dim.d_date_sk
and d_qoy = 2 and d_year = 2001
group by ca_zip
order by ca_zip
limit 100;
{code}
The Traits setup for the Operators are:
{code}
FIL[23]: bucketCols=[[]],numBuckets=-1
RS[11]: bucketCols=[[VALUE._col0]],numBuckets=-1
JOIN[12]: bucketCols=[[_col71], [_col71]],numBuckets=-1
FIL[13]: bucketCols=[[_col71], [_col71]],numBuckets=-1
SEL[14]: bucketCols=[[_col71], [_col71]],numBuckets=-1
GBY[15]: bucketCols=[[_col0]],numBuckets=-1
RS[16]: bucketCols=[[KEY._col0]],numBuckets=-1
GBY[17]: bucketCols=[[_col0]],numBuckets=-1
SEL[18]: bucketCols=[[_col0]],numBuckets=-1
LIM[21]: bucketCols=[[_col0]],numBuckets=-1
FS[22]: bucketCols=[[_col0]],numBuckets=-1
TS[3]: bucketCols=[[]],numBuckets=-1
RS[5]: bucketCols=[[VALUE._col0]],numBuckets=-1
JOIN[6]: bucketCols=[[_col3], [_col36]],numBuckets=-1
RS[7]: bucketCols=[[VALUE._col40]],numBuckets=-1
JOIN[9]: bucketCols=[[_col40], [_col0]],numBuckets=-1
RS[10]: bucketCols=[[VALUE._col0]],numBuckets=-1
TS[1]: bucketCols=[[]],numBuckets=-1
RS[8]: bucketCols=[[VALUE._col0]],numBuckets=-1
TS[0]: bucketCols=[[]],numBuckets=-1
RS[4]: bucketCols=[[VALUE._col3]],numBuckets=-1
{code}
This is incorrect:
Join[9] joins ca join (cs join cust). In this case both sides of join have a
'_col0' column. The reverse mapping of trait propagation relies on
ExprNodeColumnDesc.isSame; since this doesn't account for the tabAlias we end
up with Join[9] being bucketed on cs_sold_date_sk; Join[12] has the same
issue, only compounds the error.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6945) issues with dropping partitions on Oracle
[ https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-6945: --- Status: Patch Available (was: Open) issues with dropping partitions on Oracle - Key: HIVE-6945 URL: https://issues.apache.org/jira/browse/HIVE-6945 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-6945.patch 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is translated by DN into decimal rather than long. This appears to be specific to some cases because it seemed to have worked before (different version of Oracle? JDBC? DN? Maybe depends on whether db was auto-created). 2) When partition dropping code falls back to JDO, it creates objects to return, then drops partitions. It appears that dropping makes DN objects invalid. We create metastore partition objects out of DN objects before drop, however the list of partition column values is re-used, rather than copied, into these. DN appears to clear this list during drop, so the returned object becomes invalid and the exception is thrown. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6945) issues with dropping partitions on Oracle
[ https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-6945: --- Attachment: HIVE-6945.patch issues with dropping partitions on Oracle - Key: HIVE-6945 URL: https://issues.apache.org/jira/browse/HIVE-6945 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-6945.patch 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is translated by DN into decimal rather than long. This appears to be specific to some cases because it seemed to have worked before (different version of Oracle? JDBC? DN? Maybe depends on whether db was auto-created). 2) When partition dropping code falls back to JDO, it creates objects to return, then drops partitions. It appears that dropping makes DN objects invalid. We create metastore partition objects out of DN objects before drop, however the list of partition column values is re-used, rather than copied, into these. DN appears to clear this list during drop, so the returned object becomes invalid and the exception is thrown. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6955) ExprNodeColDesc isSame doesn't account for tabAlias: this affects trait Propagation in Joins
[
https://issues.apache.org/jira/browse/HIVE-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Butani updated HIVE-6955:
Status: Patch Available (was: Open)
ExprNodeColDesc isSame doesn't account for tabAlias: this affects trait
Propagation in Joins
Key: HIVE-6955
URL: https://issues.apache.org/jira/browse/HIVE-6955
Project: Hive
Issue Type: Bug
Reporter: Harish Butani
Assignee: Harish Butani
Attachments: HIVE-6955.1.patch
For tpcds Q15:
{code}
explain
select ca_zip, sum(cs_sales_price)
from catalog_sales, customer, customer_address, date_dim
where catalog_sales.cs_bill_customer_sk = customer.c_customer_sk
and customer.c_current_addr_sk = customer_address.ca_address_sk
and (substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475',
'85392', '85460', '80348', '81792')
or ca_state in ('CA','WA','GA')
or cs_sales_price 500)
and catalog_sales.cs_sold_date_sk = date_dim.d_date_sk
and d_qoy = 2 and d_year = 2001
group by ca_zip
order by ca_zip
limit 100;
{code}
The Traits setup for the Operators are:
{code}
FIL[23]: bucketCols=[[]],numBuckets=-1
RS[11]: bucketCols=[[VALUE._col0]],numBuckets=-1
JOIN[12]: bucketCols=[[_col71], [_col71]],numBuckets=-1
FIL[13]: bucketCols=[[_col71], [_col71]],numBuckets=-1
SEL[14]: bucketCols=[[_col71], [_col71]],numBuckets=-1
GBY[15]: bucketCols=[[_col0]],numBuckets=-1
RS[16]: bucketCols=[[KEY._col0]],numBuckets=-1
GBY[17]: bucketCols=[[_col0]],numBuckets=-1
SEL[18]: bucketCols=[[_col0]],numBuckets=-1
LIM[21]: bucketCols=[[_col0]],numBuckets=-1
FS[22]: bucketCols=[[_col0]],numBuckets=-1
TS[3]: bucketCols=[[]],numBuckets=-1
RS[5]: bucketCols=[[VALUE._col0]],numBuckets=-1
JOIN[6]: bucketCols=[[_col3], [_col36]],numBuckets=-1
RS[7]: bucketCols=[[VALUE._col40]],numBuckets=-1
JOIN[9]: bucketCols=[[_col40], [_col0]],numBuckets=-1
RS[10]: bucketCols=[[VALUE._col0]],numBuckets=-1
TS[1]: bucketCols=[[]],numBuckets=-1
RS[8]: bucketCols=[[VALUE._col0]],numBuckets=-1
TS[0]: bucketCols=[[]],numBuckets=-1
RS[4]: bucketCols=[[VALUE._col3]],numBuckets=-1
{code}
This is incorrect:
Join[9] joins ca join (cs join cust). In this case both sides of join have a
'_col0' column. The reverse mapping of trait propagation relies on
ExprNodeColumnDesc.isSame; since this doesn't account for the tabAlias we end
up with Join[9] being bucketed on cs_sold_date_sk; Join[12] has the same
issue, only compounds the error.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6955) ExprNodeColDesc isSame doesn't account for tabAlias: this affects trait Propagation in Joins
[
https://issues.apache.org/jira/browse/HIVE-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Butani updated HIVE-6955:
Attachment: HIVE-6955.1.patch
ExprNodeColDesc isSame doesn't account for tabAlias: this affects trait
Propagation in Joins
Key: HIVE-6955
URL: https://issues.apache.org/jira/browse/HIVE-6955
Project: Hive
Issue Type: Bug
Reporter: Harish Butani
Assignee: Harish Butani
Attachments: HIVE-6955.1.patch
For tpcds Q15:
{code}
explain
select ca_zip, sum(cs_sales_price)
from catalog_sales, customer, customer_address, date_dim
where catalog_sales.cs_bill_customer_sk = customer.c_customer_sk
and customer.c_current_addr_sk = customer_address.ca_address_sk
and (substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475',
'85392', '85460', '80348', '81792')
or ca_state in ('CA','WA','GA')
or cs_sales_price 500)
and catalog_sales.cs_sold_date_sk = date_dim.d_date_sk
and d_qoy = 2 and d_year = 2001
group by ca_zip
order by ca_zip
limit 100;
{code}
The Traits setup for the Operators are:
{code}
FIL[23]: bucketCols=[[]],numBuckets=-1
RS[11]: bucketCols=[[VALUE._col0]],numBuckets=-1
JOIN[12]: bucketCols=[[_col71], [_col71]],numBuckets=-1
FIL[13]: bucketCols=[[_col71], [_col71]],numBuckets=-1
SEL[14]: bucketCols=[[_col71], [_col71]],numBuckets=-1
GBY[15]: bucketCols=[[_col0]],numBuckets=-1
RS[16]: bucketCols=[[KEY._col0]],numBuckets=-1
GBY[17]: bucketCols=[[_col0]],numBuckets=-1
SEL[18]: bucketCols=[[_col0]],numBuckets=-1
LIM[21]: bucketCols=[[_col0]],numBuckets=-1
FS[22]: bucketCols=[[_col0]],numBuckets=-1
TS[3]: bucketCols=[[]],numBuckets=-1
RS[5]: bucketCols=[[VALUE._col0]],numBuckets=-1
JOIN[6]: bucketCols=[[_col3], [_col36]],numBuckets=-1
RS[7]: bucketCols=[[VALUE._col40]],numBuckets=-1
JOIN[9]: bucketCols=[[_col40], [_col0]],numBuckets=-1
RS[10]: bucketCols=[[VALUE._col0]],numBuckets=-1
TS[1]: bucketCols=[[]],numBuckets=-1
RS[8]: bucketCols=[[VALUE._col0]],numBuckets=-1
TS[0]: bucketCols=[[]],numBuckets=-1
RS[4]: bucketCols=[[VALUE._col3]],numBuckets=-1
{code}
This is incorrect:
Join[9] joins ca join (cs join cust). In this case both sides of join have a
'_col0' column. The reverse mapping of trait propagation relies on
ExprNodeColumnDesc.isSame; since this doesn't account for the tabAlias we end
up with Join[9] being bucketed on cs_sold_date_sk; Join[12] has the same
issue, only compounds the error.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Review Request 20565: HIVE-6945 issues with dropping partitions on Oracle
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20565/ --- Review request for hive, Ashutosh Chauhan and Thejas Nair. Repository: hive-git Description --- see JIRA Diffs - metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java a5c7daf metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 6a0eabe metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java fe15101 Diff: https://reviews.apache.org/r/20565/diff/ Testing --- Thanks, Sergey Shelukhin
[jira] [Updated] (HIVE-5376) Hive does not honor type for partition columns when altering column type
[ https://issues.apache.org/jira/browse/HIVE-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5376: Status: Open (was: Patch Available) Hive does not honor type for partition columns when altering column type Key: HIVE-5376 URL: https://issues.apache.org/jira/browse/HIVE-5376 Project: Hive Issue Type: Bug Components: CLI Reporter: Sergey Shelukhin Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5376.1.patch, HIVE-5376.2.patch, HIVE-5376.3.patch Followup for HIVE-5297. If partition column of type string is changed to int, the data is not verified. The values for partition columns are all in metastore db, so it's easy to check and fail the type change. alter_partition_coltype.q (or some other test?) checks this behavior right now. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6945) issues with dropping partitions on Oracle
[ https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977200#comment-13977200 ] Sergey Shelukhin commented on HIVE-6945: https://reviews.apache.org/r/20565/ issues with dropping partitions on Oracle - Key: HIVE-6945 URL: https://issues.apache.org/jira/browse/HIVE-6945 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-6945.patch 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is translated by DN into decimal rather than long. This appears to be specific to some cases because it seemed to have worked before (different version of Oracle? JDBC? DN? Maybe depends on whether db was auto-created). 2) When partition dropping code falls back to JDO, it creates objects to return, then drops partitions. It appears that dropping makes DN objects invalid. We create metastore partition objects out of DN objects before drop, however the list of partition column values is re-used, rather than copied, into these. DN appears to clear this list during drop, so the returned object becomes invalid and the exception is thrown. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5376) Hive does not honor type for partition columns when altering column type
[ https://issues.apache.org/jira/browse/HIVE-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5376: Status: Patch Available (was: Open) Hive does not honor type for partition columns when altering column type Key: HIVE-5376 URL: https://issues.apache.org/jira/browse/HIVE-5376 Project: Hive Issue Type: Bug Components: CLI Reporter: Sergey Shelukhin Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5376.1.patch, HIVE-5376.2.patch, HIVE-5376.3.patch Followup for HIVE-5297. If partition column of type string is changed to int, the data is not verified. The values for partition columns are all in metastore db, so it's easy to check and fail the type change. alter_partition_coltype.q (or some other test?) checks this behavior right now. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6893) out of sequence error in HiveMetastore server
[
https://issues.apache.org/jira/browse/HIVE-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-6893:
Fix Version/s: 0.13.0
Status: Patch Available (was: Open)
out of sequence error in HiveMetastore server
-
Key: HIVE-6893
URL: https://issues.apache.org/jira/browse/HIVE-6893
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Reporter: Romain Rigaux
Assignee: Naveen Gangam
Fix For: 0.13.0
Attachments: HIVE-6893.1.patch
Calls listing databases or tables fail. It seems to be a concurrency problem.
{code}
014-03-06 05:34:00,785 ERROR hive.log:
org.apache.thrift.TApplicationException: get_databases failed: out of
sequence response
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:472)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:459)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:648)
at
org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:66)
at
org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:278)
at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:582)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57)
at com.sun.proxy.$Proxy9.getSchemas(Unknown Source)
at
org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:192)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:263)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6893) out of sequence error in HiveMetastore server
[
https://issues.apache.org/jira/browse/HIVE-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-6893:
Attachment: HIVE-6893.1.patch
This proposed fix makes HS2 use a synchronized wrapper for the metastore client
when the HMS is running as a standalone process. In embedded mode, we would not
be reading sequenced responses over a socket.
This fix does NOT synchronize all the calls to the HMS. For example,
Hive.getMSC() returns an metastore client instance that is not thread-safe.
There is a bit of risk synchronizing these calls, as it might have a regressive
performance impact.
I will be measure performance impact of using a synchronized wrapper all over
and see if the reward outweighs the risk.
out of sequence error in HiveMetastore server
-
Key: HIVE-6893
URL: https://issues.apache.org/jira/browse/HIVE-6893
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Reporter: Romain Rigaux
Assignee: Naveen Gangam
Fix For: 0.13.0
Attachments: HIVE-6893.1.patch
Calls listing databases or tables fail. It seems to be a concurrency problem.
{code}
014-03-06 05:34:00,785 ERROR hive.log:
org.apache.thrift.TApplicationException: get_databases failed: out of
sequence response
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:472)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:459)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:648)
at
org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:66)
at
org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:278)
at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:582)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57)
at com.sun.proxy.$Proxy9.getSchemas(Unknown Source)
at
org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:192)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:263)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6893) out of sequence error in HiveMetastore server
[
https://issues.apache.org/jira/browse/HIVE-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977213#comment-13977213
]
Vaibhav Gumashta commented on HIVE-6893:
[~ngangam] Can you also upload the patch to review board?
out of sequence error in HiveMetastore server
-
Key: HIVE-6893
URL: https://issues.apache.org/jira/browse/HIVE-6893
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Reporter: Romain Rigaux
Assignee: Naveen Gangam
Fix For: 0.13.0
Attachments: HIVE-6893.1.patch
Calls listing databases or tables fail. It seems to be a concurrency problem.
{code}
014-03-06 05:34:00,785 ERROR hive.log:
org.apache.thrift.TApplicationException: get_databases failed: out of
sequence response
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:472)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:459)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:648)
at
org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:66)
at
org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:278)
at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:582)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57)
at com.sun.proxy.$Proxy9.getSchemas(Unknown Source)
at
org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:192)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:263)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6947) More fixes for tests on hadoop-2
[ https://issues.apache.org/jira/browse/HIVE-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977218#comment-13977218 ] Jason Dere commented on HIVE-6947: -- Just a bit of explanation on auto_sortmerge_join_16: the query results had been changed by HIVE-6455, looks like a subsequent fix has restored the result set back to its original results. +1 More fixes for tests on hadoop-2 - Key: HIVE-6947 URL: https://issues.apache.org/jira/browse/HIVE-6947 Project: Hive Issue Type: Bug Components: Tests Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-6947.patch Few more fixes for test cases on hadoop-2 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5376) Hive does not honor type for partition columns when altering column type
[ https://issues.apache.org/jira/browse/HIVE-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-5376: Resolution: Fixed Status: Resolved (was: Patch Available) thanks Hari, Sergey Hive does not honor type for partition columns when altering column type Key: HIVE-5376 URL: https://issues.apache.org/jira/browse/HIVE-5376 Project: Hive Issue Type: Bug Components: CLI Reporter: Sergey Shelukhin Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5376.1.patch, HIVE-5376.2.patch, HIVE-5376.3.patch Followup for HIVE-5297. If partition column of type string is changed to int, the data is not verified. The values for partition columns are all in metastore db, so it's easy to check and fail the type change. alter_partition_coltype.q (or some other test?) checks this behavior right now. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5376) Hive does not honor type for partition columns when altering column type
[ https://issues.apache.org/jira/browse/HIVE-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5376: --- Fix Version/s: 0.14.0 Hive does not honor type for partition columns when altering column type Key: HIVE-5376 URL: https://issues.apache.org/jira/browse/HIVE-5376 Project: Hive Issue Type: Bug Components: CLI Reporter: Sergey Shelukhin Assignee: Hari Sankar Sivarama Subramaniyan Fix For: 0.14.0 Attachments: HIVE-5376.1.patch, HIVE-5376.2.patch, HIVE-5376.3.patch Followup for HIVE-5297. If partition column of type string is changed to int, the data is not verified. The values for partition columns are all in metastore db, so it's easy to check and fail the type change. alter_partition_coltype.q (or some other test?) checks this behavior right now. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6893) out of sequence error in HiveMetastore server
[
https://issues.apache.org/jira/browse/HIVE-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977259#comment-13977259
]
Naveen Gangam commented on HIVE-6893:
-
Review posted to the review board.
https://reviews.apache.org/r/20569/
out of sequence error in HiveMetastore server
-
Key: HIVE-6893
URL: https://issues.apache.org/jira/browse/HIVE-6893
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Reporter: Romain Rigaux
Assignee: Naveen Gangam
Fix For: 0.13.0
Attachments: HIVE-6893.1.patch
Calls listing databases or tables fail. It seems to be a concurrency problem.
{code}
014-03-06 05:34:00,785 ERROR hive.log:
org.apache.thrift.TApplicationException: get_databases failed: out of
sequence response
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:472)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:459)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:648)
at
org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:66)
at
org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:278)
at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:582)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57)
at com.sun.proxy.$Proxy9.getSchemas(Unknown Source)
at
org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:192)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:263)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6946) Make it easier to run WebHCat e2e tests
[ https://issues.apache.org/jira/browse/HIVE-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977283#comment-13977283 ] Eugene Koifman commented on HIVE-6946: -- hcatalog/src/test/e2e/templeton/deployers/README.txt in the attached patch explains in detail what's added in the patch Make it easier to run WebHCat e2e tests --- Key: HIVE-6946 URL: https://issues.apache.org/jira/browse/HIVE-6946 Project: Hive Issue Type: Bug Components: WebHCat Affects Versions: 0.14.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Attachments: HIVE-6946.patch Right now hcatalog/src/test/e2e/templeton/README.txt explains the steps to set up WebHCat e2e tests but it's cumbersome and error prone. Need to make some improvements here. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6932) hive README needs update
[ https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977287#comment-13977287 ] Lefty Leverenz commented on HIVE-6932: -- Good, I'd wondered about that phrase but moved on to minutiae. hive README needs update Key: HIVE-6932 URL: https://issues.apache.org/jira/browse/HIVE-6932 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-6932.1.patch, HIVE-6932.2.patch It needs to be updated to include Tez as a runtime. Also, it talks about average latency being in minutes, which is very misleading. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6835) Reading of partitioned Avro data fails if partition schema does not match table schema
[
https://issues.apache.org/jira/browse/HIVE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977337#comment-13977337
]
Anthony Hsu commented on HIVE-6835:
---
I started looking into this alternative and encountered an issue. Most calls
to serde.initialize() are treating serde as a Deserializer (interface). I
would either have to change the interface (and change all the implementations)
or cast the Deserializer as an AbstractSerDe (whenever I want to use the new
initialize() method), neither of which seems like a great solution. So I am
back to supporting my original table. prefix approach. Any thoughts on this?
Reading of partitioned Avro data fails if partition schema does not match
table schema
--
Key: HIVE-6835
URL: https://issues.apache.org/jira/browse/HIVE-6835
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Anthony Hsu
Assignee: Anthony Hsu
Attachments: HIVE-6835.1.patch, HIVE-6835.2.patch, HIVE-6835.3.patch
To reproduce:
{code}
create table testarray (a arraystring);
load data local inpath '/home/ahsu/test/array.txt' into table testarray;
# create partitioned Avro table with one array column
create table avroarray partitioned by (y string) row format serde
'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties
('avro.schema.literal'='{namespace:test,name:avroarray,type:
record, fields: [ { name:a, type:{type:array,items:string}
} ] }') STORED as INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
insert into table avroarray partition(y=1) select * from testarray;
# add an int column with a default value of 0
alter table avroarray set serde
'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with
serdeproperties('avro.schema.literal'='{namespace:test,name:avroarray,type:
record, fields: [ {name:intfield,type:int,default:0},{
name:a, type:{type:array,items:string} } ] }');
# fails with ClassCastException
select * from avroarray;
{code}
The select * fails with:
{code}
Failed with exception java.io.IOException:java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
cannot be cast to
org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6950) Parsing Error in GROUPING SETS
[
https://issues.apache.org/jira/browse/HIVE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977346#comment-13977346
]
Harish Butani commented on HIVE-6950:
-
Yes there s an ambiguity in the grammar.
The dfa for the groupingSetExpression chooses to interpret the '((tab1.a,
tab1.b))' as
'((tab1.a), (tab1.b))' and so it complains that the ')' is missing.
In the case of '((a, tab1.b))' the dfa chooses the right path.
The fix is to add Syntactic predicate for this rule.
Parsing Error in GROUPING SETS
--
Key: HIVE-6950
URL: https://issues.apache.org/jira/browse/HIVE-6950
Project: Hive
Issue Type: Bug
Reporter: Rohit Agarwal
The following query:
{code}
SELECT tab1.a,
tab1.b,
SUM(tab1.c)
FROM tab1
GROUP BY tab1.a,
tab1.b
GROUPING SETS ((tab1.a, tab1.b))
{code}
results in the following error:
{code}
ParseException line 7:22 missing ) at ',' near 'EOF'
line 7:31 extraneous input ')' expecting EOF near 'EOF'
{code}
Changing the query to:
{code}
SELECT tab1.a,
tab1.b,
SUM(tab1.c)
FROM tab1
GROUP BY tab1.a,
tab1.b
GROUPING SETS ((a, tab1.b))
{code}
makes it work.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6924) MapJoinKeyBytes::hashCode() should use Murmur hash
[ https://issues.apache.org/jira/browse/HIVE-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-6924: --- Attachment: HIVE-6924.01.patch Updated patch to use Apache implementation. I am assuming +1 still stands as the patch is essentially the same. Running tests MapJoinKeyBytes::hashCode() should use Murmur hash -- Key: HIVE-6924 URL: https://issues.apache.org/jira/browse/HIVE-6924 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-6924.01.patch, HIVE-6924.patch Existing hashCode is bad, causes HashMap to cluster -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6945) issues with dropping partitions on Oracle
[ https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6945: Affects Version/s: 0.13.0 issues with dropping partitions on Oracle - Key: HIVE-6945 URL: https://issues.apache.org/jira/browse/HIVE-6945 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-6945.patch 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is translated by DN into decimal rather than long. This appears to be specific to some cases because it seemed to have worked before (different version of Oracle? JDBC? DN? Maybe depends on whether db was auto-created). 2) When partition dropping code falls back to JDO, it creates objects to return, then drops partitions. It appears that dropping makes DN objects invalid. We create metastore partition objects out of DN objects before drop, however the list of partition column values is re-used, rather than copied, into these. DN appears to clear this list during drop, so the returned object becomes invalid and the exception is thrown. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6956) Duplicate partitioning column for union when dynamic partition sort optimization is enabled
Prasanth J created HIVE-6956: Summary: Duplicate partitioning column for union when dynamic partition sort optimization is enabled Key: HIVE-6956 URL: https://issues.apache.org/jira/browse/HIVE-6956 Project: Hive Issue Type: Bug Affects Versions: 0.13.0, 0.14.0 Reporter: Prasanth J Assignee: Prasanth J HIVE-6455 optimization uses PreOrder graph walker resulting in double invocation of SortedDynPartitionOptimizer(). Because of this there are duplicate partitioning columns in RSOp for few test cases (union_remove_17.q and similar test cases..).. HIVE-6455 optimization does not actually need a PreOrder graph walker. The fix is to revert it to DefaultGraphWalker. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6946) Make it easier to run WebHCat e2e tests
[
https://issues.apache.org/jira/browse/HIVE-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977442#comment-13977442
]
Hive QA commented on HIVE-6946:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12641165/HIVE-6946.patch
{color:red}ERROR:{color} -1 due to 43 failed/errored test(s), 5417 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/6/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/6/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 43 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12641165
Make it easier to run WebHCat e2e tests
---
Key: HIVE-6946
URL: https://issues.apache.org/jira/browse/HIVE-6946
Project: Hive
Issue Type: Bug
Components: WebHCat
Affects Versions: 0.14.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Attachments: HIVE-6946.patch
Right now hcatalog/src/test/e2e/templeton/README.txt explains the steps to
set up WebHCat e2e tests but it's cumbersome and error prone. Need to make
some improvements here.
NO PRECOMMIT TESTS
--
This message was sent by Atlassian JIRA
(v6.2#6252)
java.lang.NoSuchMethodError: org/apache/hadoop/conf/Configuration.unset(Ljava/lang/String;)V
Team: I am getting the exception below on attempt to show tables; in Hive 0.13.0. What is the minimum version of Hadoop that Hive 0.13.0 can be used with? Hive 0.12.0 worked with Hadoop 0.20.x, 0.23.x.y, 1.x.y, 2.x.y Exception in thread main java.lang.NoSuchMethodError: org/apache/hadoop/conf/Configuration.unset(Ljava/lang/String;)V at org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3416) at org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3401) at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:214) at org.apache.hadoop.hive.ql.Context.getLocalScratchDir(Context.java:241) at org.apache.hadoop.hive.ql.Context.getLocalTmpPath(Context.java:333) at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:296) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:391) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:291) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:944) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1009) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:880) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:870) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:600) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Regards Dmitry Vasilenko
[jira] [Created] (HIVE-6957) SQL authorization does not work with HS2 binary mode and Kerberos auth
Thejas M Nair created HIVE-6957: --- Summary: SQL authorization does not work with HS2 binary mode and Kerberos auth Key: HIVE-6957 URL: https://issues.apache.org/jira/browse/HIVE-6957 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2 Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-6957.1.patch In HiveServer2, when Kerberos auth and binary transport modes are used, the user name that gets passed on to authorization is the long kerberos username. The username that is used in grant/revoke statements tend to be the short usernames. This also fails in authorizing statements that involve URI, as the authorization mode checks the file system permissions for given user. It does not recognize that the given long username actually owns the file or belongs to the group that owns the file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6957) SQL authorization does not work with HS2 binary mode and Kerberos auth
[ https://issues.apache.org/jira/browse/HIVE-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977461#comment-13977461 ] Thejas M Nair commented on HIVE-6957: - A workaround is to use the http transport mode for HS2. SQL authorization does not work with HS2 binary mode and Kerberos auth -- Key: HIVE-6957 URL: https://issues.apache.org/jira/browse/HIVE-6957 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2 Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-6957.1.patch In HiveServer2, when Kerberos auth and binary transport modes are used, the user name that gets passed on to authorization is the long kerberos username. The username that is used in grant/revoke statements tend to be the short usernames. This also fails in authorizing statements that involve URI, as the authorization mode checks the file system permissions for given user. It does not recognize that the given long username actually owns the file or belongs to the group that owns the file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6957) SQL authorization does not work with HS2 binary mode and Kerberos auth
[ https://issues.apache.org/jira/browse/HIVE-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6957: Attachment: HIVE-6957.1.patch SQL authorization does not work with HS2 binary mode and Kerberos auth -- Key: HIVE-6957 URL: https://issues.apache.org/jira/browse/HIVE-6957 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2 Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-6957.1.patch In HiveServer2, when Kerberos auth and binary transport modes are used, the user name that gets passed on to authorization is the long kerberos username. The username that is used in grant/revoke statements tend to be the short usernames. This also fails in authorizing statements that involve URI, as the authorization mode checks the file system permissions for given user. It does not recognize that the given long username actually owns the file or belongs to the group that owns the file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6957) SQL authorization does not work with HS2 binary mode and Kerberos auth
[ https://issues.apache.org/jira/browse/HIVE-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977462#comment-13977462 ] Thejas M Nair commented on HIVE-6957: - The long username is not of any significance within hive. We always use the short username for all purposes including the owner in metastore. This patch changes the username that gets set for HS2 purposes, to the short username. SQL authorization does not work with HS2 binary mode and Kerberos auth -- Key: HIVE-6957 URL: https://issues.apache.org/jira/browse/HIVE-6957 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2 Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-6957.1.patch In HiveServer2, when Kerberos auth and binary transport modes are used, the user name that gets passed on to authorization is the long kerberos username. The username that is used in grant/revoke statements tend to be the short usernames. This also fails in authorizing statements that involve URI, as the authorization mode checks the file system permissions for given user. It does not recognize that the given long username actually owns the file or belongs to the group that owns the file. -- This message was sent by Atlassian JIRA (v6.2#6252)
Review Request 20578: HIVE-6957 - SQL authorization does not work with HS2 binary mode and Kerberos auth
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20578/ --- Review request for hive, Ashutosh Chauhan and Vaibhav Gumashta. Bugs: HIVE-6957 https://issues.apache.org/jira/browse/HIVE-6957 Repository: hive-git Description --- In HiveServer2, when Kerberos auth and binary transport modes are used, the user name that gets passed on to authorization is the long kerberos username. The username that is used in grant/revoke statements tend to be the short usernames. This also fails in authorizing statements that involve URI, as the authorization mode checks the file system permissions for given user. It does not recognize that the given long username actually owns the file or belongs to the group that owns the file. Diffs - itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/MiniHiveKdc.java f7ec93d itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java 62bfa1e itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdcSQLAuth.java PRE-CREATION shims/common-secure/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java 9e296de Diff: https://reviews.apache.org/r/20578/diff/ Testing --- Unit test included. Thanks, Thejas Nair
[jira] [Commented] (HIVE-6957) SQL authorization does not work with HS2 binary mode and Kerberos auth
[
https://issues.apache.org/jira/browse/HIVE-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977468#comment-13977468
]
Thejas M Nair commented on HIVE-6957:
-
Error looks like this
{code}
java.sql.SQLException: Error while compiling statement: FAILED:
HiveAccessControlException Permission denied. Principal
[name=us...@example.com, type=USER] does not have following privileges on
Object [type=TABLE_OR_VIEW, name=default.test_jdbc_sql_auth2] : [SELECT]
{code}
SQL authorization does not work with HS2 binary mode and Kerberos auth
--
Key: HIVE-6957
URL: https://issues.apache.org/jira/browse/HIVE-6957
Project: Hive
Issue Type: Bug
Components: Authorization, HiveServer2
Affects Versions: 0.13.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Attachments: HIVE-6957.1.patch
In HiveServer2, when Kerberos auth and binary transport modes are used, the
user name that gets passed on to authorization is the long kerberos username.
The username that is used in grant/revoke statements tend to be the short
usernames.
This also fails in authorizing statements that involve URI, as the
authorization mode checks the file system permissions for given user. It does
not recognize that the given long username actually owns the file or belongs
to the group that owns the file.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6947) More fixes for tests on hadoop-2
[ https://issues.apache.org/jira/browse/HIVE-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-6947: - Attachment: HIVE-6947.1.patch Remove union_remove_17.q as its not an expected output. The duplication of partition columns in RSOp will be fixed in HIVE-6956. More fixes for tests on hadoop-2 - Key: HIVE-6947 URL: https://issues.apache.org/jira/browse/HIVE-6947 Project: Hive Issue Type: Bug Components: Tests Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-6947.1.patch, HIVE-6947.patch Few more fixes for test cases on hadoop-2 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5072) [WebHCat]Enable directly invoke Sqoop job through Templeton
[ https://issues.apache.org/jira/browse/HIVE-5072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977476#comment-13977476 ] Thejas M Nair commented on HIVE-5072: - +1 [WebHCat]Enable directly invoke Sqoop job through Templeton --- Key: HIVE-5072 URL: https://issues.apache.org/jira/browse/HIVE-5072 Project: Hive Issue Type: Improvement Components: WebHCat Affects Versions: 0.12.0 Reporter: Shuaishuai Nie Assignee: Shuaishuai Nie Attachments: HIVE-5072.1.patch, HIVE-5072.2.patch, HIVE-5072.3.patch, HIVE-5072.4.patch, HIVE-5072.5.patch, Templeton-Sqoop-Action.pdf Now it is hard to invoke a Sqoop job through templeton. The only way is to use the classpath jar generated by a sqoop job and use the jar delegator in Templeton. We should implement Sqoop Delegator to enable directly invoke Sqoop job through Templeton. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6945) issues with dropping partitions on Oracle
[ https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-6945: --- Attachment: HIVE-6945.01.patch some missed places on sql path issues with dropping partitions on Oracle - Key: HIVE-6945 URL: https://issues.apache.org/jira/browse/HIVE-6945 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-6945.01.patch, HIVE-6945.patch 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is translated by DN into decimal rather than long. This appears to be specific to some cases because it seemed to have worked before (different version of Oracle? JDBC? DN? Maybe depends on whether db was auto-created). 2) When partition dropping code falls back to JDO, it creates objects to return, then drops partitions. It appears that dropping makes DN objects invalid. We create metastore partition objects out of DN objects before drop, however the list of partition column values is re-used, rather than copied, into these. DN appears to clear this list during drop, so the returned object becomes invalid and the exception is thrown. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6957) SQL authorization does not work with HS2 binary mode and Kerberos auth
[ https://issues.apache.org/jira/browse/HIVE-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6957: Status: Patch Available (was: Open) SQL authorization does not work with HS2 binary mode and Kerberos auth -- Key: HIVE-6957 URL: https://issues.apache.org/jira/browse/HIVE-6957 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2 Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-6957.1.patch In HiveServer2, when Kerberos auth and binary transport modes are used, the user name that gets passed on to authorization is the long kerberos username. The username that is used in grant/revoke statements tend to be the short usernames. This also fails in authorizing statements that involve URI, as the authorization mode checks the file system permissions for given user. It does not recognize that the given long username actually owns the file or belongs to the group that owns the file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6936) Provide table properties to InputFormats
[ https://issues.apache.org/jira/browse/HIVE-6936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-6936: Attachment: HIVE-6936.patch I've added a test case where I use a custom input format and run select * to ensure that the properties are correctly passed down to the input format. Provide table properties to InputFormats Key: HIVE-6936 URL: https://issues.apache.org/jira/browse/HIVE-6936 Project: Hive Issue Type: Bug Components: File Formats Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.14.0 Attachments: HIVE-6936.patch, HIVE-6936.patch Some advanced file formats need the table properties made available to them. Additionally, it would be convenient to provide a unique id for fetch operators and the complete list of directories. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6945) issues with dropping partitions on Oracle
[ https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-6945: --- Attachment: HIVE-6945.01.patch issues with dropping partitions on Oracle - Key: HIVE-6945 URL: https://issues.apache.org/jira/browse/HIVE-6945 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-6945.01.patch, HIVE-6945.patch 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is translated by DN into decimal rather than long. This appears to be specific to some cases because it seemed to have worked before (different version of Oracle? JDBC? DN? Maybe depends on whether db was auto-created). 2) When partition dropping code falls back to JDO, it creates objects to return, then drops partitions. It appears that dropping makes DN objects invalid. We create metastore partition objects out of DN objects before drop, however the list of partition column values is re-used, rather than copied, into these. DN appears to clear this list during drop, so the returned object becomes invalid and the exception is thrown. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6945) issues with dropping partitions on Oracle
[ https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-6945: --- Attachment: (was: HIVE-6945.01.patch) issues with dropping partitions on Oracle - Key: HIVE-6945 URL: https://issues.apache.org/jira/browse/HIVE-6945 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-6945.01.patch, HIVE-6945.patch 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is translated by DN into decimal rather than long. This appears to be specific to some cases because it seemed to have worked before (different version of Oracle? JDBC? DN? Maybe depends on whether db was auto-created). 2) When partition dropping code falls back to JDO, it creates objects to return, then drops partitions. It appears that dropping makes DN objects invalid. We create metastore partition objects out of DN objects before drop, however the list of partition column values is re-used, rather than copied, into these. DN appears to clear this list during drop, so the returned object becomes invalid and the exception is thrown. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 20565: HIVE-6945 issues with dropping partitions on Oracle
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20565/ --- (Updated April 22, 2014, 9:58 p.m.) Review request for hive, Ashutosh Chauhan and Thejas Nair. Repository: hive-git Description --- see JIRA Diffs (updated) - metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java a5c7daf metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 6a0eabe metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java fe15101 Diff: https://reviews.apache.org/r/20565/diff/ Testing --- Thanks, Sergey Shelukhin
[jira] [Created] (HIVE-6958) update union_remove_*, other tests for hadoop-2
Jason Dere created HIVE-6958: Summary: update union_remove_*, other tests for hadoop-2 Key: HIVE-6958 URL: https://issues.apache.org/jira/browse/HIVE-6958 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Update q.out files to match totalSize for Linux platform. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6950) Parsing Error in GROUPING SETS
[
https://issues.apache.org/jira/browse/HIVE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977545#comment-13977545
]
Lefty Leverenz commented on HIVE-6950:
--
Should this be documented in the wiki?
* [GROUPING SETS clause
|https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C+Grouping+and+Rollup#EnhancedAggregation,Cube,GroupingandRollup-GROUPINGSETSclause]
Parsing Error in GROUPING SETS
--
Key: HIVE-6950
URL: https://issues.apache.org/jira/browse/HIVE-6950
Project: Hive
Issue Type: Bug
Reporter: Rohit Agarwal
The following query:
{code}
SELECT tab1.a,
tab1.b,
SUM(tab1.c)
FROM tab1
GROUP BY tab1.a,
tab1.b
GROUPING SETS ((tab1.a, tab1.b))
{code}
results in the following error:
{code}
ParseException line 7:22 missing ) at ',' near 'EOF'
line 7:31 extraneous input ')' expecting EOF near 'EOF'
{code}
Changing the query to:
{code}
SELECT tab1.a,
tab1.b,
SUM(tab1.c)
FROM tab1
GROUP BY tab1.a,
tab1.b
GROUPING SETS ((a, tab1.b))
{code}
makes it work.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6958) update union_remove_*, other tests for hadoop-2
[ https://issues.apache.org/jira/browse/HIVE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-6958: - Attachment: HIVE-6958.1.patch update union_remove_*, other tests for hadoop-2 --- Key: HIVE-6958 URL: https://issues.apache.org/jira/browse/HIVE-6958 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-6958.1.patch Update q.out files to match totalSize for Linux platform. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6958) update union_remove_*, other tests for hadoop-2
[ https://issues.apache.org/jira/browse/HIVE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-6958: - Status: Patch Available (was: Open) update union_remove_*, other tests for hadoop-2 --- Key: HIVE-6958 URL: https://issues.apache.org/jira/browse/HIVE-6958 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-6958.1.patch Update q.out files to match totalSize for Linux platform. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6956) Duplicate partitioning column for union when dynamic partition sort optimization is enabled
[ https://issues.apache.org/jira/browse/HIVE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-6956: - Status: Patch Available (was: Open) Duplicate partitioning column for union when dynamic partition sort optimization is enabled --- Key: HIVE-6956 URL: https://issues.apache.org/jira/browse/HIVE-6956 Project: Hive Issue Type: Bug Affects Versions: 0.13.0, 0.14.0 Reporter: Prasanth J Assignee: Prasanth J Labels: optimization Attachments: HIVE-6956.1.patch HIVE-6455 optimization uses PreOrder graph walker resulting in double invocation of SortedDynPartitionOptimizer(). Because of this there are duplicate partitioning columns in RSOp for few test cases (union_remove_17.q and similar test cases..).. HIVE-6455 optimization does not actually need a PreOrder graph walker. The fix is to revert it to DefaultGraphWalker. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6934) PartitionPruner doesn't handle top level constant expression correctly
[
https://issues.apache.org/jira/browse/HIVE-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-6934:
Attachment: HIVE-6934.1.patch
PartitionPruner doesn't handle top level constant expression correctly
--
Key: HIVE-6934
URL: https://issues.apache.org/jira/browse/HIVE-6934
Project: Hive
Issue Type: Bug
Reporter: Harish Butani
Assignee: Hari Sankar Sivarama Subramaniyan
Attachments: HIVE-6934.1.patch
You hit this error indirectly, because how we handle invalid constant
comparisons. Consider:
{code}
create table x(key int, value string) partitioned by (dt int, ts string);
-- both these queries hit this issue
select * from x where key = 'abc';
select * from x where dt = 'abc';
-- the issue is the comparison get converted to the constant false
-- and the PartitionPruner doesn't handle top level constant exprs corrcetly
{code}
Thanks to [~hsubramaniyan] for uncovering this as part of adding tests for
HIVE-5376
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6934) PartitionPruner doesn't handle top level constant expression correctly
[
https://issues.apache.org/jira/browse/HIVE-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-6934:
Status: Patch Available (was: Open)
PartitionPruner doesn't handle top level constant expression correctly
--
Key: HIVE-6934
URL: https://issues.apache.org/jira/browse/HIVE-6934
Project: Hive
Issue Type: Bug
Reporter: Harish Butani
Assignee: Hari Sankar Sivarama Subramaniyan
Attachments: HIVE-6934.1.patch
You hit this error indirectly, because how we handle invalid constant
comparisons. Consider:
{code}
create table x(key int, value string) partitioned by (dt int, ts string);
-- both these queries hit this issue
select * from x where key = 'abc';
select * from x where dt = 'abc';
-- the issue is the comparison get converted to the constant false
-- and the PartitionPruner doesn't handle top level constant exprs corrcetly
{code}
Thanks to [~hsubramaniyan] for uncovering this as part of adding tests for
HIVE-5376
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6956) Duplicate partitioning column for union when dynamic partition sort optimization is enabled
[ https://issues.apache.org/jira/browse/HIVE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-6956: - Attachment: HIVE-6956.1.patch The test diffs are valid since SEL-RS-EX-SEL-FS is same as SEL-RS-EX-FS.. The diffs have the redundant SEL in reducer removed after changing the graph walker. Duplicate partitioning column for union when dynamic partition sort optimization is enabled --- Key: HIVE-6956 URL: https://issues.apache.org/jira/browse/HIVE-6956 Project: Hive Issue Type: Bug Affects Versions: 0.13.0, 0.14.0 Reporter: Prasanth J Assignee: Prasanth J Labels: optimization Attachments: HIVE-6956.1.patch HIVE-6455 optimization uses PreOrder graph walker resulting in double invocation of SortedDynPartitionOptimizer(). Because of this there are duplicate partitioning columns in RSOp for few test cases (union_remove_17.q and similar test cases..).. HIVE-6455 optimization does not actually need a PreOrder graph walker. The fix is to revert it to DefaultGraphWalker. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6934) PartitionPruner doesn't handle top level constant expression correctly
[
https://issues.apache.org/jira/browse/HIVE-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977554#comment-13977554
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-6934:
-
encountered another hive issue as part of this fix.
explain select * from table where 1;
The above query passes where as ' select * from table where 1;' fails when
trying to convert 1 to boolean. We should be able to catch this exception
semantically.
PartitionPruner doesn't handle top level constant expression correctly
--
Key: HIVE-6934
URL: https://issues.apache.org/jira/browse/HIVE-6934
Project: Hive
Issue Type: Bug
Reporter: Harish Butani
Assignee: Hari Sankar Sivarama Subramaniyan
Attachments: HIVE-6934.1.patch
You hit this error indirectly, because how we handle invalid constant
comparisons. Consider:
{code}
create table x(key int, value string) partitioned by (dt int, ts string);
-- both these queries hit this issue
select * from x where key = 'abc';
select * from x where dt = 'abc';
-- the issue is the comparison get converted to the constant false
-- and the PartitionPruner doesn't handle top level constant exprs corrcetly
{code}
Thanks to [~hsubramaniyan] for uncovering this as part of adding tests for
HIVE-5376
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6958) update union_remove_*, other tests for hadoop-2
[ https://issues.apache.org/jira/browse/HIVE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977553#comment-13977553 ] Szehon Ho commented on HIVE-6958: - I'm just curious, are these the same file size when run on hadoop-1, on linux? update union_remove_*, other tests for hadoop-2 --- Key: HIVE-6958 URL: https://issues.apache.org/jira/browse/HIVE-6958 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-6958.1.patch Update q.out files to match totalSize for Linux platform. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6959) Remove vectorization related constant expression folding code once Constant propagation optimizer for Hive is committed
Hari Sankar Sivarama Subramaniyan created HIVE-6959: --- Summary: Remove vectorization related constant expression folding code once Constant propagation optimizer for Hive is committed Key: HIVE-6959 URL: https://issues.apache.org/jira/browse/HIVE-6959 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan HIVE-5771 covers Constant propagation optimizer for Hive. We should remove any vectorization related code which duplicates this feature once HIVE-5771 is committed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977563#comment-13977563 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-5771: - [~ashutoshc] Created HIVE-6959 to remove vectorization related constant expression folding code once HIVE-5771 is committed. Constant propagation optimizer for Hive --- Key: HIVE-5771 URL: https://issues.apache.org/jira/browse/HIVE-5771 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Ted Xu Assignee: Ted Xu Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.patch Currently there is no constant folding/propagation optimizer, all expressions are evaluated at runtime. HIVE-2470 did a great job on evaluating constants on UDF initializing phase, however, it is still a runtime evaluation and it doesn't propagate constants from a subquery to outside. It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6956) Duplicate partitioning column for union when dynamic partition sort optimization is enabled
[ https://issues.apache.org/jira/browse/HIVE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977564#comment-13977564 ] Ashutosh Chauhan commented on HIVE-6956: +1 Duplicate partitioning column for union when dynamic partition sort optimization is enabled --- Key: HIVE-6956 URL: https://issues.apache.org/jira/browse/HIVE-6956 Project: Hive Issue Type: Bug Affects Versions: 0.13.0, 0.14.0 Reporter: Prasanth J Assignee: Prasanth J Labels: optimization Attachments: HIVE-6956.1.patch HIVE-6455 optimization uses PreOrder graph walker resulting in double invocation of SortedDynPartitionOptimizer(). Because of this there are duplicate partitioning columns in RSOp for few test cases (union_remove_17.q and similar test cases..).. HIVE-6455 optimization does not actually need a PreOrder graph walker. The fix is to revert it to DefaultGraphWalker. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977566#comment-13977566 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-5771: - Hi [~tedxu] I have a minor comment on the implementation. Is there a reason for these rules sharing same val. +opRules.put(new RuleRegExp(R5, UNION%), ConstantPropagateProcFactory +.getUnionProc()); +opRules.put(new RuleRegExp(R5, RS%), ConstantPropagateProcFactory +.getReduceSinkProc()); Thanks Hari Constant propagation optimizer for Hive --- Key: HIVE-5771 URL: https://issues.apache.org/jira/browse/HIVE-5771 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Ted Xu Assignee: Ted Xu Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.patch Currently there is no constant folding/propagation optimizer, all expressions are evaluated at runtime. HIVE-2470 did a great job on evaluating constants on UDF initializing phase, however, it is still a runtime evaluation and it doesn't propagate constants from a subquery to outside. It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6958) update union_remove_*, other tests for hadoop-2
[ https://issues.apache.org/jira/browse/HIVE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977570#comment-13977570 ] Prasanth J commented on HIVE-6958: -- I suspect the new diffs are showing folder size as opposed to the files inside them. I will take a look at it in my linux box and will report back. update union_remove_*, other tests for hadoop-2 --- Key: HIVE-6958 URL: https://issues.apache.org/jira/browse/HIVE-6958 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-6958.1.patch Update q.out files to match totalSize for Linux platform. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HIVE-5092) Fix hiveserver2 mapreduce local job on Windows
[
https://issues.apache.org/jira/browse/HIVE-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan reassigned HIVE-5092:
---
Assignee: Hari Sankar Sivarama Subramaniyan (was: Daniel Dai)
Fix hiveserver2 mapreduce local job on Windows
--
Key: HIVE-5092
URL: https://issues.apache.org/jira/browse/HIVE-5092
Project: Hive
Issue Type: Bug
Components: HiveServer2, Windows
Reporter: Daniel Dai
Assignee: Hari Sankar Sivarama Subramaniyan
Attachments: HIVE-5092-1.patch
Hiveserver2 fail on Mapreduce local job fail. For example:
{code}
select /*+ MAPJOIN(v) */ registration from studenttab10k s join votertab10k v
on (s.name = v.name);
{code}
The root cause is class not found in the local hadoop job
(MapredLocalTask.execute). HADOOP_CLASSPATH does not include $HIVE_HOME/lib.
Set HADOOP_CLASSPATH correctly will fix the issue.
However, there is one complexity in Windows. We start Hiveserver2 using
Windows service console (services.msc), which takes hiveserver2.xml generated
by hive.cmd. There is no way to pass environment variable in hiveserver2.xml
(weird but reality). I attach a patch which pass it through command line
arguments and relay to HADOOP_CLASSPATH in Hive code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6950) Parsing Error in GROUPING SETS
[
https://issues.apache.org/jira/browse/HIVE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977595#comment-13977595
]
Harish Butani commented on HIVE-6950:
-
My 2 cents would be: no. This is a bug. Should get fixed.
Documenting every bug and then updating when we fix, would be asking too much...
Parsing Error in GROUPING SETS
--
Key: HIVE-6950
URL: https://issues.apache.org/jira/browse/HIVE-6950
Project: Hive
Issue Type: Bug
Reporter: Rohit Agarwal
The following query:
{code}
SELECT tab1.a,
tab1.b,
SUM(tab1.c)
FROM tab1
GROUP BY tab1.a,
tab1.b
GROUPING SETS ((tab1.a, tab1.b))
{code}
results in the following error:
{code}
ParseException line 7:22 missing ) at ',' near 'EOF'
line 7:31 extraneous input ')' expecting EOF near 'EOF'
{code}
Changing the query to:
{code}
SELECT tab1.a,
tab1.b,
SUM(tab1.c)
FROM tab1
GROUP BY tab1.a,
tab1.b
GROUPING SETS ((a, tab1.b))
{code}
makes it work.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5092) Fix hiveserver2 mapreduce local job on Windows
[
https://issues.apache.org/jira/browse/HIVE-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-5092:
Attachment: HIVE-5092.2.patch
cc-ing [~daijy] for review. I have rebased the original patch without the
change in hiveserver2.cmd changes. We will have the hiveserver2.cmd once we
introduce windows scripts for Hive installation. The current changes provide a
mechanism for accepting hive.hadoop.classpath as part of hive.
Thanks
Hari
Fix hiveserver2 mapreduce local job on Windows
--
Key: HIVE-5092
URL: https://issues.apache.org/jira/browse/HIVE-5092
Project: Hive
Issue Type: Bug
Components: HiveServer2, Windows
Reporter: Daniel Dai
Assignee: Hari Sankar Sivarama Subramaniyan
Attachments: HIVE-5092-1.patch, HIVE-5092.2.patch
Hiveserver2 fail on Mapreduce local job fail. For example:
{code}
select /*+ MAPJOIN(v) */ registration from studenttab10k s join votertab10k v
on (s.name = v.name);
{code}
The root cause is class not found in the local hadoop job
(MapredLocalTask.execute). HADOOP_CLASSPATH does not include $HIVE_HOME/lib.
Set HADOOP_CLASSPATH correctly will fix the issue.
However, there is one complexity in Windows. We start Hiveserver2 using
Windows service console (services.msc), which takes hiveserver2.xml generated
by hive.cmd. There is no way to pass environment variable in hiveserver2.xml
(weird but reality). I attach a patch which pass it through command line
arguments and relay to HADOOP_CLASSPATH in Hive code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5092) Fix hiveserver2 mapreduce local job on Windows
[
https://issues.apache.org/jira/browse/HIVE-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-5092:
Status: Patch Available (was: Open)
Fix hiveserver2 mapreduce local job on Windows
--
Key: HIVE-5092
URL: https://issues.apache.org/jira/browse/HIVE-5092
Project: Hive
Issue Type: Bug
Components: HiveServer2, Windows
Reporter: Daniel Dai
Assignee: Hari Sankar Sivarama Subramaniyan
Attachments: HIVE-5092-1.patch, HIVE-5092.2.patch
Hiveserver2 fail on Mapreduce local job fail. For example:
{code}
select /*+ MAPJOIN(v) */ registration from studenttab10k s join votertab10k v
on (s.name = v.name);
{code}
The root cause is class not found in the local hadoop job
(MapredLocalTask.execute). HADOOP_CLASSPATH does not include $HIVE_HOME/lib.
Set HADOOP_CLASSPATH correctly will fix the issue.
However, there is one complexity in Windows. We start Hiveserver2 using
Windows service console (services.msc), which takes hiveserver2.xml generated
by hive.cmd. There is no way to pass environment variable in hiveserver2.xml
(weird but reality). I attach a patch which pass it through command line
arguments and relay to HADOOP_CLASSPATH in Hive code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Plan: permanently move hive builds from bigtop
Patch upload auto-trigger is now enabled, thanks Ashutosh for helping me fix the Jenkins url from Apache-Infra side. Now there's no need to manually submit Pre-commit builds, unless it got missed somehow. And in this Jenkins, you can easily see all the JIRA's in the queue: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/ Sorry, please use this URL and not the first one, as job was renamed due to the auto-trigger expecting this format. And to repeat, Test Reporting and the 'Logs located at:' link is still broken and tracked in HIVE-6937. For now, to see the logs of a failed build, you have to manually navigate. Example, for PreCommit-Hive-BUILD#6, logs are at: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-6/http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-6/. The HiveQA comment on the JIRA should mention the build number. Thanks, Szehon On Sun, Apr 20, 2014 at 11:07 PM, Szehon Ho sze...@cloudera.com wrote: It looks great, thanks Lefty! On Sun, Apr 20, 2014 at 2:22 PM, Lefty Leverenz leftylever...@gmail.comwrote: Nice doc, Szehon. I did some minor editing so you might want to make sure I didn't introduce any errors. https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure -- Lefty On Sat, Apr 19, 2014 at 9:45 PM, Szehon Ho sze...@cloudera.com wrote: Migration is done, I updated the wiki to add all the details of the new setup: https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure New Jenkins URL to submit pre-commit jobs: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/precommit-hive/ Again, this has to be done manually for time being, by clicking on 'build with parameters', and entering the issue number as a parameter. I've submitted some already. I'll reach out to some committers to get the auto-trigger working. As I mentioned, there is some work to fix the test-reporting, due to the framework using old url scheme. I am tracking it at HIVE-6937https://issues.apache.org/jira/browse/HIVE-6937. For now I am hosting log directory separately, if you want to see test logs, you have to manually go the url corresponding to your build, like: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/precommit-hive-11/for run#11. Sorry about that. Let me know if you see other issues, thanks! Szehon On Fri, Apr 18, 2014 at 2:11 PM, Thejas Nair the...@hortonworks.com wrote: Sounds good. Thanks Szehon! On Fri, Apr 18, 2014 at 10:17 AM, Ashutosh Chauhan hashut...@apache.org wrote: +1 Thanks Szehon! On Fri, Apr 18, 2014 at 6:29 AM, Xuefu Zhang xzh...@cloudera.com wrote: +1. Thanks for taking care of this. On Thu, Apr 17, 2014 at 11:00 PM, Szehon Ho sze...@cloudera.com wrote: Hi, This week the machine running Hive builds at http://bigtop01.cloudera.org:8080/view/Hive/? ran out of space, so new jobs like Precommit tests stopped. Its still not resolved there, there was another email today on Bigtop list, but there's very few people with root access to that host, and they still haven't responded. I chatted with Brock, he has also seen various issues with Bigtop jenkins in the past, so I am thinking to move the Jenkins jobs to the PTest master itself, where some PMC already have access and can admin if needed. Currently I am hosting the pre-commit Jenkins job on my own EC2 instance as stop-gap. Other advantages of hosting our own Jenkins: 1. No need to wait for other Bigtop jobs to run. 2. Bigtop is using a version of Jenkins that doesnt show parameters like JIRA number for queued jobs, so impossible to tell whether a patch got picked up and where it is in queue. 3. Eliminate network hop from Bigtop box to our PTest master. The disadvantage is: 1. We don't have much experience doing Jenkins admin, but it doesn't look too bad. Mostly, restart if there's issue and clean up if out of space. I wonder what people think, and if there's any objections to this? If not, I'll try setting up this weekend. Then, there is some follow-up work, like changing the Jenkins url's displayed in the test report. Thanks! Szehon -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this
[jira] [Commented] (HIVE-6927) Add support for MSSQL in schematool
[ https://issues.apache.org/jira/browse/HIVE-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977607#comment-13977607 ] Lefty Leverenz commented on HIVE-6927: -- This doesn't seem to need any documentation in the wiki ... or does it? * [Hive Metastore Administration |https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin] * [Hive Schema Tool |https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool] Add support for MSSQL in schematool --- Key: HIVE-6927 URL: https://issues.apache.org/jira/browse/HIVE-6927 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0 Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal Fix For: 0.14.0 Attachments: HIVE-6927.patch Schematool is the preferred way of initializing schema for Hive. Since HIVE-6862 provided the script for MSSQL it would be nice to add the support for it in schematool. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6960) Set Hive pom to use Hadoop-2.4
Jason Dere created HIVE-6960: Summary: Set Hive pom to use Hadoop-2.4 Key: HIVE-6960 URL: https://issues.apache.org/jira/browse/HIVE-6960 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.14.0 Reporter: Jason Dere A number of the hadoop-2 unit test failures are due to HADOOP-10425, fixed in Hadoop 2.4. Perhaps we should move onto that version. - org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3 - org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4 - org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source - org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format - org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database - org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6835) Reading of partitioned Avro data fails if partition schema does not match table schema
[
https://issues.apache.org/jira/browse/HIVE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977616#comment-13977616
]
Xuefu Zhang commented on HIVE-6835:
---
Not sure if I understand your problem correctly, but I do understand that the
scope of the proposed change has got bigger than your original approach. For
any caller of serde initialization, we should be able to find whether serde
instance extends AbstractSerde. If so, we cast the serde instance to
AbstractSerde and call initialize(arg1, arg2, arg3). Otherwise, call
serde.initialize(arg1, arg2). Does this solve the problem?
Reading of partitioned Avro data fails if partition schema does not match
table schema
--
Key: HIVE-6835
URL: https://issues.apache.org/jira/browse/HIVE-6835
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Anthony Hsu
Assignee: Anthony Hsu
Attachments: HIVE-6835.1.patch, HIVE-6835.2.patch, HIVE-6835.3.patch
To reproduce:
{code}
create table testarray (a arraystring);
load data local inpath '/home/ahsu/test/array.txt' into table testarray;
# create partitioned Avro table with one array column
create table avroarray partitioned by (y string) row format serde
'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties
('avro.schema.literal'='{namespace:test,name:avroarray,type:
record, fields: [ { name:a, type:{type:array,items:string}
} ] }') STORED as INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
insert into table avroarray partition(y=1) select * from testarray;
# add an int column with a default value of 0
alter table avroarray set serde
'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with
serdeproperties('avro.schema.literal'='{namespace:test,name:avroarray,type:
record, fields: [ {name:intfield,type:int,default:0},{
name:a, type:{type:array,items:string} } ] }');
# fails with ClassCastException
select * from avroarray;
{code}
The select * fails with:
{code}
Failed with exception java.io.IOException:java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
cannot be cast to
org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6958) update union_remove_*, other tests for hadoop-2
[ https://issues.apache.org/jira/browse/HIVE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977619#comment-13977619 ] Jason Dere commented on HIVE-6958: -- Szehon, almost all of the tests (except stats_partialscan_autogether.q) were hadoop-2 only tests. I'll take a look at that stats_partialscan_autogether.q and see what's up with that one. Prasanth, you are correct that in these cases the difference in size is due to different folder file size on the different platforms. update union_remove_*, other tests for hadoop-2 --- Key: HIVE-6958 URL: https://issues.apache.org/jira/browse/HIVE-6958 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-6958.1.patch Update q.out files to match totalSize for Linux platform. -- This message was sent by Atlassian JIRA (v6.2#6252)
