[jira] [Created] (HIVE-18267) LLAP: Eagerly allocate cache arenas
Prasanth Jayachandran created HIVE-18267: Summary: LLAP: Eagerly allocate cache arenas Key: HIVE-18267 URL: https://issues.apache.org/jira/browse/HIVE-18267 Project: Hive Issue Type: Bug Affects Versions: 3.0.0 Reporter: Prasanth Jayachandran When LLAP starts it will be good to eagerly allocate all arenas required by cache allocator to avoid OOM at runtime. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: Review Request 64524: HIVE-18250
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64524/ --- (Updated Dec. 12, 2017, 11:17 p.m.) Review request for hive and Ashutosh Chauhan. Changes --- Q test files updates. Bugs: HIVE-18250 https://issues.apache.org/jira/browse/HIVE-18250 Repository: hive-git Description --- HIVE-18250 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java ba64f97105eaaa0a5d3f2ad1883db42608c95ae8 ql/src/test/queries/clientpositive/groupby_multialias.q PRE-CREATION ql/src/test/results/clientpositive/complex_alias.q.out 5182153856bba9e0678b816cd435c013a94460b6 ql/src/test/results/clientpositive/groupby_multialias.q.out PRE-CREATION ql/src/test/results/clientpositive/order3.q.out d3db1b9ca6c250fc94b13e343253750f8f059e58 ql/src/test/results/clientpositive/perf/spark/query19.q.out 6a70ddcbd641f576a4802b9946e5aea37efaae18 ql/src/test/results/clientpositive/perf/spark/query55.q.out c6119185725730fbae435798bfb5e0ae8d91bf0b ql/src/test/results/clientpositive/perf/spark/query71.q.out 73af3565b6d90ee46a3ab6fd95d2a71c1b2e2041 ql/src/test/results/clientpositive/perf/tez/query19.q.out 363425fe65b5f305224de926a3743963039de90d ql/src/test/results/clientpositive/perf/tez/query55.q.out 27ea1c3698335eb7c5232f351860971b70342506 ql/src/test/results/clientpositive/perf/tez/query71.q.out bd48e5621be347167180f6b5ccb8745bdaf0d03a Diff: https://reviews.apache.org/r/64524/diff/3/ Changes: https://reviews.apache.org/r/64524/diff/2-3/ Testing --- Thanks, Jesús Camacho Rodríguez
Review Request 64555: HIVE-18230 create plan like plan, and replace plan commands for easy modification
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64555/ --- Review request for hive, Harish Jaiprakash and Prasanth_J. Repository: hive-git Description --- see jira Diffs - itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 62c9172ef5d9ae74b158d1b4f1b8c5c0eca4e375 ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 55ef8de9a5c7144931d0a6ff13224765ee737fea ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 50bdce89a44a8dc87a97e394d00e5dadebbbd351 ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java a09b7961c2dbc26b4d2fa912d0be7037885f63e4 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 1dcfe9dd2834ada2f2bb75bc67e436017c21f71f ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 13789502c235be478659f9267b107e8580594bc8 ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g f1ca30109fc40b452f0e1423a586818bcb783796 ql/src/java/org/apache/hadoop/hive/ql/parse/ResourcePlanParser.g 95c87259c5bdb91a4533d7897d5d32a7c4f56503 ql/src/java/org/apache/hadoop/hive/ql/plan/AlterResourcePlanDesc.java d70d52054dbef5ecbe5be55fd24df25c5381c8d7 ql/src/java/org/apache/hadoop/hive/ql/plan/CreateResourcePlanDesc.java efdd05c5f270e26b84a0e022eac0445ff368969b ql/src/test/queries/clientpositive/resourceplan.q fc924a2f95a849b6e14065cb62c2910db40d1b47 ql/src/test/results/clientpositive/llap/resourceplan.q.out 7f3e784457f87ff88c5f746a9fc2b74ea28c2a6e standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 6e0da5781ed8d536852613bad1793d27dd17dfd4 standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java fc254c6f5338ebcfe5a19bea847ff8141552f15f standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 573ac0173dd51b81cb37902cc144983393ab1f51 standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java 2c92bb20561d09998e400b458036f8e4a960a469 standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java 75fbfa23d25c589826a44be493eb84f3cd375e5e standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java da518ab6e337cbd5befb9f7c1df1781335bd72f4 standalone-metastore/src/main/thrift/hive_metastore.thrift 1085ce566a5fba632ae2acb074ce538143bac6ed standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 24c59f2f1b0b017cff3b26f86c3f903b498d2553 standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1e4fe5d973542d6b0e3ae0760ca5c06ea9888089 Diff: https://reviews.apache.org/r/64555/diff/1/ Testing --- Thanks, Sergey Shelukhin
[jira] [Created] (HIVE-18266) LLAP: /system references wrong file for THP
Prasanth Jayachandran created HIVE-18266: Summary: LLAP: /system references wrong file for THP Key: HIVE-18266 URL: https://issues.apache.org/jira/browse/HIVE-18266 Project: Hive Issue Type: Bug Affects Versions: 3.0.0 Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran copy paste error in /system endpoint. THP references same files again. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: Review Request 64326: HIVE-18208
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64326/ --- (Updated Dec. 12, 2017, 7:37 p.m.) Review request for hive and Jason Dere. Changes --- Rebased the patch Repository: hive-git Description --- SMB Join : Fix the unit tests to run SMB Joins. Updated tests and result files. Diffs (updated) - ql/src/test/queries/clientpositive/auto_sortmerge_join_1.q a1d5249448 ql/src/test/queries/clientpositive/auto_sortmerge_join_10.q e65344dd6d ql/src/test/queries/clientpositive/auto_sortmerge_join_11.q 11499f8eab ql/src/test/queries/clientpositive/auto_sortmerge_join_12.q b512cc5c74 ql/src/test/queries/clientpositive/auto_sortmerge_join_13.q b5706be3d5 ql/src/test/queries/clientpositive/auto_sortmerge_join_14.q dd59c74fc0 ql/src/test/queries/clientpositive/auto_sortmerge_join_15.q 1480b15488 ql/src/test/queries/clientpositive/auto_sortmerge_join_2.q e77d937991 ql/src/test/queries/clientpositive/auto_sortmerge_join_3.q 183f03335a ql/src/test/queries/clientpositive/auto_sortmerge_join_4.q 21f273a17b ql/src/test/queries/clientpositive/auto_sortmerge_join_7.q cf12331e13 ql/src/test/queries/clientpositive/auto_sortmerge_join_8.q 5ec4e26d4b ql/src/test/queries/clientpositive/auto_sortmerge_join_9.q f95631f429 ql/src/test/queries/clientpositive/bucketsortoptimize_insert_2.q 4a14587857 ql/src/test/queries/clientpositive/bucketsortoptimize_insert_6.q ec0c2dc254 ql/src/test/queries/clientpositive/bucketsortoptimize_insert_7.q 45635c1209 ql/src/test/queries/clientpositive/quotedid_smb.q 25d1f0eee7 ql/src/test/queries/clientpositive/smb_cache.q e415e51053 ql/src/test/results/clientpositive/auto_sortmerge_join_10.q.out 22ac2a201a ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out 243a49b45f ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 3d0559a47c ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 36bfac3f4c ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out c1459d53ef ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 37d97d2252 ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 655573650b ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out b78a517a74 ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 72d2c62e5b ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 91b1d8ea67 ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 90d362e981 ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 365f63c0ad ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 8ee44b3493 ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 83d5a968b7 ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 0e0428481b ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 37c07afbf1 ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out ce248323ce ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_6.q.out 3ba8a65772 ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_7.q.out 0af21e0e26 ql/src/test/results/clientpositive/llap/quotedid_smb.q.out 8e850f50ce ql/src/test/results/clientpositive/llap/smb_cache.q.out defade5c08 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_1.q.out e6038b857d ql/src/test/results/clientpositive/spark/auto_sortmerge_join_12.q.out ff9a0f4fa4 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_14.q.out 8c0d506b26 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_15.q.out b005bda331 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out 025d0d29c5 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_3.q.out 3ad950a107 ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out 60437ec56d ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out 16ecabe05d ql/src/test/results/clientpositive/spark/auto_sortmerge_join_8.q.out e180471dcb ql/src/test/results/clientpositive/spark/auto_sortmerge_join_9.q.out 4d0476f9ee ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_2.q.out 814553d81a ql/src/test/results/clientpositive/spark/quotedid_smb.q.out 7b8777f9d6 Diff: https://reviews.apache.org/r/64326/diff/3/ Changes: https://reviews.apache.org/r/64326/diff/2-3/ Testing --- Thanks, Deepak Jaiswal
[jira] [Created] (HIVE-18265) desc formatted/extended or show create table can not fully display the result when field or table comment contains tab character
Hui Huang created HIVE-18265:
Summary: desc formatted/extended or show create table can not
fully display the result when field or table comment contains tab character
Key: HIVE-18265
URL: https://issues.apache.org/jira/browse/HIVE-18265
Project: Hive
Issue Type: Bug
Components: CLI
Affects Versions: 3.0.0
Reporter: Hui Huang
Assignee: Hui Huang
Fix For: 3.0.0
Here are some examples:
create table test_comment (id1 string comment 'full_\tname1', id2 string
comment 'full_\tname2', id3 string comment 'full_\tname3') stored as textfile;
When execute `show create table test_comment`, we can see the following content
in the console,
{quote}
createtab_stmt
CREATE TABLE `test_comment`(
`id1` string COMMENT 'full_
`id2` string COMMENT 'full_
`id3` string COMMENT 'full_
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'hdfs://xxx/user/huanghui/warehouse/huanghuitest.db/test_comment'
TBLPROPERTIES (
'transient_lastDdlTime'='1513095570')
{quote}
And the output of `desc formatted table ` is a little similar,
{quote}
col_namedata_type comment
\# col_name data_type comment
id1 string full_
id2 string full_
id3 string full_
\# Detailed Table Information
(ignore)...
{quote}
When execute `desc extended test_comment`, the problem is more obvious,
{quote}
col_namedata_type comment
id1 string full_
id2 string full_
id3 string full_
Detailed Table Information Table(tableName:test_comment,
dbName:huanghuitest, owner:huanghui, createTime:1513095570, lastAccessTime:0,
retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:id1, type:string,
comment:full_name1), FieldSchema(name:id2, type:string, comment:full_
{quote}
*the rest of the content is lost*.
The content is not really lost, it's just can not display normal. Because hive
store the result in LazyStruct, and LazyStruct use '\t' as field separator:
{code:java}
// LazyStruct.java#parse()
// Go through all bytes in the byte[]
while (fieldByteEnd <= structByteEnd) {
if (fieldByteEnd == structByteEnd || bytes[fieldByteEnd] == separator) {
// Reached the end of a field?
if (lastColumnTakesRest && fieldId == fields.length - 1) {
fieldByteEnd = structByteEnd;
}
startPosition[fieldId] = fieldByteBegin;
fieldId++;
if (fieldId == fields.length || fieldByteEnd == structByteEnd) {
// All fields have been parsed, or bytes have been parsed.
// We need to set the startPosition of fields.length to ensure we
// can use the same formula to calculate the length of each field.
// For missing fields, their starting positions will all be the same,
// which will make their lengths to be -1 and uncheckedGetField will
// return these fields as NULLs.
for (int i = fieldId; i <= fields.length; i++) {
startPosition[i] = fieldByteEnd + 1;
}
break;
}
fieldByteBegin = fieldByteEnd + 1;
fieldByteEnd++;
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Adding Hive Metastore functions to add and alter partitions for multiple tables
Hi dev, I'm wondering if Hive community have ever considered support adding and altering multiple partitions for multiple tables? I'm using Hive Metastore to manage the metadata for Presto querying. Our business requires that we should publish some partitions of data for multiple tables at the same time in an atomic transaction to keep the data consistency. Currently Hive Metastore only supports adding and altering multiple tables for one table. I drafted AddPartitionsForTables and AlterPartitionsForTables function to achieve this based on existing AddPartition and AlterPartition logic and we are testing it on our system. I'm wondering if community have considered these functionality. I would like to contribute the functionality if you have interest. Thank you! -Kaijie Tel: +86-13810485829 E-mail: daniel...@gmail.com
