[jira] [Commented] (HIVE-7099) Add Decimal datatype support for Windowing
[
https://issues.apache.org/jira/browse/HIVE-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004404#comment-14004404
]
Hive QA commented on HIVE-7099:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12645831/HIVE-7099.1.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 5527 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_decimal
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_load_dyn_part1
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.common.metrics.TestMetrics.testScopeConcurrency
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/251/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/251/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-251/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12645831
Add Decimal datatype support for Windowing
--
Key: HIVE-7099
URL: https://issues.apache.org/jira/browse/HIVE-7099
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Harish Butani
Assignee: Harish Butani
Attachments: HIVE-7099.1.patch
Decimal datatype is not handled by Windowing
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7025) Support retention on hive tables
[ https://issues.apache.org/jira/browse/HIVE-7025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-7025: Attachment: HIVE-7025.2.patch.txt Support retention on hive tables Key: HIVE-7025 URL: https://issues.apache.org/jira/browse/HIVE-7025 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-7025.1.patch.txt, HIVE-7025.2.patch.txt Add self destruction properties for temporary tables. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7025) Support retention on hive tables
[ https://issues.apache.org/jira/browse/HIVE-7025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-7025: Summary: Support retention on hive tables (was: TTL on hive tables) Support retention on hive tables Key: HIVE-7025 URL: https://issues.apache.org/jira/browse/HIVE-7025 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-7025.1.patch.txt, HIVE-7025.2.patch.txt Add self destruction properties for temporary tables. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7025) Support retention on hive tables
[ https://issues.apache.org/jira/browse/HIVE-7025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004410#comment-14004410 ] Navis commented on HIVE-7025: - [~appodictic] Re-implemented to use the retention property in table object and now it supports partitioned tables also (drops old partitions instead of the table). Might be better then previous version, but I cannot make RB entry for this (RB throws The file 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java' (r0c7c0d7) could not be found in the repository). Support retention on hive tables Key: HIVE-7025 URL: https://issues.apache.org/jira/browse/HIVE-7025 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-7025.1.patch.txt, HIVE-7025.2.patch.txt Add self destruction properties for temporary tables. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7075) JsonSerde raises NullPointerException when object key is not lower case
[
https://issues.apache.org/jira/browse/HIVE-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-7075:
Status: Patch Available (was: Open)
JsonSerde raises NullPointerException when object key is not lower case
---
Key: HIVE-7075
URL: https://issues.apache.org/jira/browse/HIVE-7075
Project: Hive
Issue Type: Bug
Components: HCatalog, Serializers/Deserializers
Affects Versions: 0.12.0
Reporter: Yibing Shi
Attachments: HIVE-7075.1.patch.txt
We have noticed that the JsonSerde produces a NullPointerException if a JSON
object has a key value that is not lower case. For example. Assume we have
the file one.json:
{ empId : 123, name : John }
{ empId : 456, name : Jane }
hive CREATE TABLE emps (empId INT, name STRING)
ROW FORMAT SERDE org.apache.hive.hcatalog.data.JsonSerDe;
hive LOAD DATA LOCAL INPATH 'one.json' INTO TABLE emps;
hive SELECT * FROM emps;
Failed with exception java.io.IOException:java.lang.NullPointerException
Notice, it seems to work if the keys are lower case. Assume we have the file
'two.json':
{ empid : 123, name : John }
{ empid : 456, name : Jane }
hive DROP TABLE emps;
hive CREATE TABLE emps (empId INT, name STRING)
ROW FORMAT SERDE org.apache.hive.hcatalog.data.JsonSerDe;
hive LOAD DATA LOCAL INPATH 'two.json' INTO TABLE emps;
hive SELECT * FROM emps;
OK
123 John
456 Jane
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7075) JsonSerde raises NullPointerException when object key is not lower case
[
https://issues.apache.org/jira/browse/HIVE-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-7075:
Attachment: HIVE-7075.1.patch.txt
Cannot sure it's right to use lower cased field names. But this is the patch.
JsonSerde raises NullPointerException when object key is not lower case
---
Key: HIVE-7075
URL: https://issues.apache.org/jira/browse/HIVE-7075
Project: Hive
Issue Type: Bug
Components: HCatalog, Serializers/Deserializers
Affects Versions: 0.12.0
Reporter: Yibing Shi
Attachments: HIVE-7075.1.patch.txt
We have noticed that the JsonSerde produces a NullPointerException if a JSON
object has a key value that is not lower case. For example. Assume we have
the file one.json:
{ empId : 123, name : John }
{ empId : 456, name : Jane }
hive CREATE TABLE emps (empId INT, name STRING)
ROW FORMAT SERDE org.apache.hive.hcatalog.data.JsonSerDe;
hive LOAD DATA LOCAL INPATH 'one.json' INTO TABLE emps;
hive SELECT * FROM emps;
Failed with exception java.io.IOException:java.lang.NullPointerException
Notice, it seems to work if the keys are lower case. Assume we have the file
'two.json':
{ empid : 123, name : John }
{ empid : 456, name : Jane }
hive DROP TABLE emps;
hive CREATE TABLE emps (empId INT, name STRING)
ROW FORMAT SERDE org.apache.hive.hcatalog.data.JsonSerDe;
hive LOAD DATA LOCAL INPATH 'two.json' INTO TABLE emps;
hive SELECT * FROM emps;
OK
123 John
456 Jane
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Is there a bug in hiveserver2 metastore caused hold huge objects
There are 18908 PM-s. I post it on http://jack-meng.qiniudn.com/hive-heap2.pnghttp://jack-meng.qiniudn.com/hive-heap.png Thanks. 2014-05-20 0:41 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: What is the count of PM-s that you are seeing when it crashes? From that screenshot I don't see what the problem is or I'm not reading it right. On Mon, May 19, 2014 at 1:17 AM, Meng QingPing mqingp...@gmail.com wrote: Thanks. I post it on http://jack-meng.qiniudn.com/hive-heap.png 2014-05-06 23:14 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: I don't think the dev list supports attachment. Can you post to some image sharing service? On Tue, May 6, 2014 at 3:20 AM, Meng QingPing mqingp...@gmail.com wrote: Weird, the picture attached in sent mail. Anyway send again. 2014-05-05 12:17 GMT+08:00 Chandra Reddy chandu...@gmail.com: seems you have missed attachment. -Chandra On Sun, May 4, 2014 at 6:57 PM, Meng QingPing mqingp...@gmail.com wrote: I run hiveserver2 with metastore in mysql. The hiveserver2 OOM and the heap dump show huge objects hold by org.datanucleus.api.jdo. JDOPersistenceManagerFactory as attached. It seems not release org.datanucleus.api.jdo.JDOPersistenceManager. Hive version is 0.13. Thanks, Jack -- Thanks, -Chandra. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping
Re: Is there a bug in hiveserver2 metastore caused hold huge objects
There are 18908 PM-s. I post it on http://jack-meng.qiniudn.com/hive-heap2.png Thanks. 2014-05-20 0:41 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: What is the count of PM-s that you are seeing when it crashes? From that screenshot I don't see what the problem is or I'm not reading it right. On Mon, May 19, 2014 at 1:17 AM, Meng QingPing mqingp...@gmail.com wrote: Thanks. I post it on http://jack-meng.qiniudn.com/hive-heap.png 2014-05-06 23:14 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: I don't think the dev list supports attachment. Can you post to some image sharing service? On Tue, May 6, 2014 at 3:20 AM, Meng QingPing mqingp...@gmail.com wrote: Weird, the picture attached in sent mail. Anyway send again. 2014-05-05 12:17 GMT+08:00 Chandra Reddy chandu...@gmail.com: seems you have missed attachment. -Chandra On Sun, May 4, 2014 at 6:57 PM, Meng QingPing mqingp...@gmail.com wrote: I run hiveserver2 with metastore in mysql. The hiveserver2 OOM and the heap dump show huge objects hold by org.datanucleus.api.jdo. JDOPersistenceManagerFactory as attached. It seems not release org.datanucleus.api.jdo.JDOPersistenceManager. Hive version is 0.13. Thanks, Jack -- Thanks, -Chandra. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping
[jira] [Updated] (HIVE-6584) Add HiveHBaseTableSnapshotInputFormat
[
https://issues.apache.org/jira/browse/HIVE-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Teng Yutong updated HIVE-6584:
--
Attachment: HIVE-6584.1.patch
this patch is based on the newest patch(HBASE-11137.02-0.98.patch) related
HBASE-11137.
Add HiveHBaseTableSnapshotInputFormat
-
Key: HIVE-6584
URL: https://issues.apache.org/jira/browse/HIVE-6584
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Attachments: HIVE-6584.0.patch, HIVE-6584.1.patch
HBASE-8369 provided mapreduce support for reading from HBase table snapsopts.
This allows a MR job to consume a stable, read-only view of an HBase table
directly off of HDFS. Bypassing the online region server API provides a nice
performance boost for the full scan. HBASE-10642 is backporting that feature
to 0.94/0.96 and also adding a {{mapred}} implementation. Once that's
available, we should add an input format. A follow-on patch could work out
how to integrate this functionality into the StorageHandler, similar to how
HIVE-6473 integrates the HFileOutputFormat into existing table definitions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-4561) Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0.0000 ,if all the column values larger than 0.0 (or if all column values smaller than 0.0)
[ https://issues.apache.org/jira/browse/HIVE-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-4561: Attachment: HIVE-4561.4.patch.txt Changed to allow null for min/max value of long/double/decimal stats. Let's see what will happen. Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0. ,if all the column values larger than 0.0 (or if all column values smaller than 0.0) Key: HIVE-4561 URL: https://issues.apache.org/jira/browse/HIVE-4561 Project: Hive Issue Type: Bug Components: Statistics Affects Versions: 0.12.0 Reporter: caofangkun Assignee: caofangkun Attachments: HIVE-4561.1.patch, HIVE-4561.2.patch, HIVE-4561.3.patch, HIVE-4561.4.patch.txt if all column values larger than 0.0 DOUBLE_LOW_VALUE always will be 0.0 or if all column values less than 0.0, DOUBLE_HIGH_VALUE will always be hive (default) create table src_test (price double); hive (default) load data local inpath './test.txt' into table src_test; hive (default) select * from src_test; OK 1.0 2.0 3.0 Time taken: 0.313 seconds, Fetched: 3 row(s) hive (default) analyze table src_test compute statistics for columns price; mysql select * from TAB_COL_STATS \G; CS_ID: 16 DB_NAME: default TABLE_NAME: src_test COLUMN_NAME: price COLUMN_TYPE: double TBL_ID: 2586 LONG_LOW_VALUE: 0 LONG_HIGH_VALUE: 0 DOUBLE_LOW_VALUE: 0. # Wrong Result ! Expected is 1. DOUBLE_HIGH_VALUE: 3. BIG_DECIMAL_LOW_VALUE: NULL BIG_DECIMAL_HIGH_VALUE: NULL NUM_NULLS: 0 NUM_DISTINCTS: 1 AVG_COL_LEN: 0. MAX_COL_LEN: 0 NUM_TRUES: 0 NUM_FALSES: 0 LAST_ANALYZED: 1368596151 2 rows in set (0.00 sec) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7024) Escape control characters for explain result
[ https://issues.apache.org/jira/browse/HIVE-7024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-7024: Description: Comments for columns are now delimited by 0x00, which is binary and make git refuse to make proper diff file. (was: Not a serious issue, but a little irritating. ) Escape control characters for explain result Key: HIVE-7024 URL: https://issues.apache.org/jira/browse/HIVE-7024 Project: Hive Issue Type: Bug Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-7024.1.patch.txt Comments for columns are now delimited by 0x00, which is binary and make git refuse to make proper diff file. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Is there a bug in hiveserver2 metastore caused hold huge objects
I post the heap bin file on http://jack-meng.qiniudn.com/heap.bin Thanks. 2014-05-21 15:41 GMT+08:00 Meng QingPing mqingp...@gmail.com: There are 18908 PM-s. I post it on http://jack-meng.qiniudn.com/hive-heap2.png Thanks. 2014-05-20 0:41 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: What is the count of PM-s that you are seeing when it crashes? From that screenshot I don't see what the problem is or I'm not reading it right. On Mon, May 19, 2014 at 1:17 AM, Meng QingPing mqingp...@gmail.com wrote: Thanks. I post it on http://jack-meng.qiniudn.com/hive-heap.png 2014-05-06 23:14 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: I don't think the dev list supports attachment. Can you post to some image sharing service? On Tue, May 6, 2014 at 3:20 AM, Meng QingPing mqingp...@gmail.com wrote: Weird, the picture attached in sent mail. Anyway send again. 2014-05-05 12:17 GMT+08:00 Chandra Reddy chandu...@gmail.com: seems you have missed attachment. -Chandra On Sun, May 4, 2014 at 6:57 PM, Meng QingPing mqingp...@gmail.com wrote: I run hiveserver2 with metastore in mysql. The hiveserver2 OOM and the heap dump show huge objects hold by org.datanucleus.api.jdo. JDOPersistenceManagerFactory as attached. It seems not release org.datanucleus.api.jdo.JDOPersistenceManager. Hive version is 0.13. Thanks, Jack -- Thanks, -Chandra. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- Thanks, Qingping
[jira] [Created] (HIVE-7105) Enable ReduceRecordProcessor to generate VectorizedRowBatches
Rajesh Balamohan created HIVE-7105: -- Summary: Enable ReduceRecordProcessor to generate VectorizedRowBatches Key: HIVE-7105 URL: https://issues.apache.org/jira/browse/HIVE-7105 Project: Hive Issue Type: Bug Components: Vectorization Reporter: Rajesh Balamohan Assignee: Jitendra Nath Pandey Currently, ReduceRecordProcessor sends one key,value pair at a time to its operator pipeline. It would be beneficial to send VectorizedRowBatch to downstream operators. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5764) Stopping Metastore and HiveServer2 from command line
[
https://issues.apache.org/jira/browse/HIVE-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004542#comment-14004542
]
Hive QA commented on HIVE-5764:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12645594/HIVE-5764.patch
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 5451 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.common.metrics.TestMetrics.testScopeConcurrency
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/252/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/252/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-252/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12645594
Stopping Metastore and HiveServer2 from command line
Key: HIVE-5764
URL: https://issues.apache.org/jira/browse/HIVE-5764
Project: Hive
Issue Type: Bug
Components: HiveServer2, Metastore
Reporter: Vaibhav Gumashta
Assignee: Xiaobing Zhou
Labels: patch
Fix For: 0.14.0
Attachments: HIVE-5764.patch
Currently a user needs to kill the process. Ideally there should be something
like:
hive --service metastore stop
hive --service hiveserver2 stop
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6584) Add HiveHBaseTableSnapshotInputFormat
[
https://issues.apache.org/jira/browse/HIVE-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004558#comment-14004558
]
Nick Dimiduk commented on HIVE-6584:
Thanks [~tenggyut]. Any thoughts regarding how to test this?
Add HiveHBaseTableSnapshotInputFormat
-
Key: HIVE-6584
URL: https://issues.apache.org/jira/browse/HIVE-6584
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Attachments: HIVE-6584.0.patch, HIVE-6584.1.patch
HBASE-8369 provided mapreduce support for reading from HBase table snapsopts.
This allows a MR job to consume a stable, read-only view of an HBase table
directly off of HDFS. Bypassing the online region server API provides a nice
performance boost for the full scan. HBASE-10642 is backporting that feature
to 0.94/0.96 and also adding a {{mapred}} implementation. Once that's
available, we should add an input format. A follow-on patch could work out
how to integrate this functionality into the StorageHandler, similar to how
HIVE-6473 integrates the HFileOutputFormat into existing table definitions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7105) Enable ReduceRecordProcessor to generate VectorizedRowBatches
[ https://issues.apache.org/jira/browse/HIVE-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004572#comment-14004572 ] Remus Rusanu commented on HIVE-7105: Extending the vectorized processing to the reduce side is a complex undertaking. None of the vector mode operators are implemented in reduce side. The thinking is that the bulk of the CPU intensive processing occurs on the map side and our goal was to provide maximum feature coverage (ie. implement as many operators as needed to cover the most queries) but atm vectorization only works for map side of first stage. I'm not sure whether at this stage we can call the map side effort stable/mature/complete enough to warrant a focus shift to reduce side. Enable ReduceRecordProcessor to generate VectorizedRowBatches - Key: HIVE-7105 URL: https://issues.apache.org/jira/browse/HIVE-7105 Project: Hive Issue Type: Bug Components: Vectorization Reporter: Rajesh Balamohan Assignee: Jitendra Nath Pandey Currently, ReduceRecordProcessor sends one key,value pair at a time to its operator pipeline. It would be beneficial to send VectorizedRowBatch to downstream operators. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7075) JsonSerde raises NullPointerException when object key is not lower case
[
https://issues.apache.org/jira/browse/HIVE-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004582#comment-14004582
]
Harsh J commented on HIVE-7075:
---
Can a test case be added as well, aside of just the fix, so this does not
regress in future?
bq. Cannot sure it's right to use lower cased field names.
Hive explicitly appears to make them lower case though?
JsonSerde raises NullPointerException when object key is not lower case
---
Key: HIVE-7075
URL: https://issues.apache.org/jira/browse/HIVE-7075
Project: Hive
Issue Type: Bug
Components: HCatalog, Serializers/Deserializers
Affects Versions: 0.12.0
Reporter: Yibing Shi
Attachments: HIVE-7075.1.patch.txt
We have noticed that the JsonSerde produces a NullPointerException if a JSON
object has a key value that is not lower case. For example. Assume we have
the file one.json:
{ empId : 123, name : John }
{ empId : 456, name : Jane }
hive CREATE TABLE emps (empId INT, name STRING)
ROW FORMAT SERDE org.apache.hive.hcatalog.data.JsonSerDe;
hive LOAD DATA LOCAL INPATH 'one.json' INTO TABLE emps;
hive SELECT * FROM emps;
Failed with exception java.io.IOException:java.lang.NullPointerException
Notice, it seems to work if the keys are lower case. Assume we have the file
'two.json':
{ empid : 123, name : John }
{ empid : 456, name : Jane }
hive DROP TABLE emps;
hive CREATE TABLE emps (empId INT, name STRING)
ROW FORMAT SERDE org.apache.hive.hcatalog.data.JsonSerDe;
hive LOAD DATA LOCAL INPATH 'two.json' INTO TABLE emps;
hive SELECT * FROM emps;
OK
123 John
456 Jane
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7105) Enable ReduceRecordProcessor to generate VectorizedRowBatches
[ https://issues.apache.org/jira/browse/HIVE-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-7105: --- Attachment: HIVE-7105.1.patch [~rusanu] Agreed that there are no vectorized operators on the reducer side atm. Purpose of this JIRA is to take the first step in enabling it on the reducer side for simple queries. Enable ReduceRecordProcessor to generate VectorizedRowBatches - Key: HIVE-7105 URL: https://issues.apache.org/jira/browse/HIVE-7105 Project: Hive Issue Type: Bug Components: Vectorization Reporter: Rajesh Balamohan Assignee: Jitendra Nath Pandey Attachments: HIVE-7105.1.patch Currently, ReduceRecordProcessor sends one key,value pair at a time to its operator pipeline. It would be beneficial to send VectorizedRowBatch to downstream operators. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7098) RecordUpdater should extend RecordWriter
[
https://issues.apache.org/jira/browse/HIVE-7098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004624#comment-14004624
]
Hive QA commented on HIVE-7098:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12645858/HIVE-7098.patch
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 5526 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/253/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/253/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-253/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12645858
RecordUpdater should extend RecordWriter
Key: HIVE-7098
URL: https://issues.apache.org/jira/browse/HIVE-7098
Project: Hive
Issue Type: Sub-task
Components: File Formats, Transactions
Reporter: Alan Gates
Assignee: Alan Gates
Attachments: HIVE-7098.patch
A new interface ql.io.RecordUpdater was added as part of the ACID work in
0.13. This interface should extend RecordWriter because:
# If it does not significant portions of FileSinkOperator will have to be
reworked to handle both RecordWriter and RecordUpdater
# Once a file format accepts transactions, it should not generally be
possible to write using RecordWriter.write as that will write old style
records without transaction information.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7092) Insert overwrite should not delete the original directory
[ https://issues.apache.org/jira/browse/HIVE-7092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004674#comment-14004674 ] Xuefu Zhang commented on HIVE-7092: --- +1 Insert overwrite should not delete the original directory - Key: HIVE-7092 URL: https://issues.apache.org/jira/browse/HIVE-7092 Project: Hive Issue Type: Bug Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-7092.patch Today the implementation of insert overwrite table or partition deletes the entire directory of the table/partition recursively using the HDFS shell (-rmr) and then re-creates it. This makes it get rid of certain user-set attributes of the directory, like permission, owner, group, and will become more important with introduction of HDFS extended-ACL's. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7048) CompositeKeyHBaseFactory should not use FamilyFilter
[ https://issues.apache.org/jira/browse/HIVE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004678#comment-14004678 ] Swarnim Kulkarni commented on HIVE-7048: I have also logged HIVE-7103 for the too-few-tests concern. CompositeKeyHBaseFactory should not use FamilyFilter Key: HIVE-7048 URL: https://issues.apache.org/jira/browse/HIVE-7048 Project: Hive Issue Type: Improvement Components: HBase Handler Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Priority: Blocker Attachments: HIVE-7048.1.patch.txt HIVE-6411 introduced a more generic way to provide composite key implementations via custom factory implementations. However it seems like the CompositeHBaseKeyFactory implementation uses a FamilyFilter for row key scans which doesn't seem appropriate. This should be investigated further and if possible replaced with a RowRangeScanFilter. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7048) CompositeKeyHBaseFactory should not use FamilyFilter
[ https://issues.apache.org/jira/browse/HIVE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004679#comment-14004679 ] Swarnim Kulkarni commented on HIVE-7048: [~xuefuz] If the above proposal sounds fine to you, I can work on getting a patch flushed out for this. CompositeKeyHBaseFactory should not use FamilyFilter Key: HIVE-7048 URL: https://issues.apache.org/jira/browse/HIVE-7048 Project: Hive Issue Type: Improvement Components: HBase Handler Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Priority: Blocker Attachments: HIVE-7048.1.patch.txt HIVE-6411 introduced a more generic way to provide composite key implementations via custom factory implementations. However it seems like the CompositeHBaseKeyFactory implementation uses a FamilyFilter for row key scans which doesn't seem appropriate. This should be investigated further and if possible replaced with a RowRangeScanFilter. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7048) CompositeKeyHBaseFactory should not use FamilyFilter
[ https://issues.apache.org/jira/browse/HIVE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004702#comment-14004702 ] Xuefu Zhang commented on HIVE-7048: --- We cannot make CompositeKeyHBaseFactory abstract as it's existing functionality: user provides a key class for HBase key via HBaseSerDe.HBASE_COMPOSITE_KEY_CLASS. However, without knowing how the key is serialized, CompositeKeyHBaseFactory is not able to do any pushdown. Thus, I think we should get rid of all pushdown/filter related code from CompositeKeyHBaseFactory. User desiring of this capability would have to provided their own key factory class. It's fine to extract the pushdown/filter related code into utility methods somewhere. CompositeKeyHBaseFactory should not use FamilyFilter Key: HIVE-7048 URL: https://issues.apache.org/jira/browse/HIVE-7048 Project: Hive Issue Type: Improvement Components: HBase Handler Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Priority: Blocker Attachments: HIVE-7048.1.patch.txt HIVE-6411 introduced a more generic way to provide composite key implementations via custom factory implementations. However it seems like the CompositeHBaseKeyFactory implementation uses a FamilyFilter for row key scans which doesn't seem appropriate. This should be investigated further and if possible replaced with a RowRangeScanFilter. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7048) CompositeKeyHBaseFactory should not use FamilyFilter
[
https://issues.apache.org/jira/browse/HIVE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004746#comment-14004746
]
Swarnim Kulkarni commented on HIVE-7048:
{quote}
User desiring of this capability would have to provided their own key factory
class.
{quote}
My only concern with that approach is that for a non-hive developer, it is
probably too much work to write custom implementations with pushdown logic
without much guidance. Also I cannot imagine a case where someone wouldn't
require the capability to set range scans or filters and do full-table scans
instead. So it might just cause unnecessary duplication of the same logic
across implementations which could otherwise be easily abstracted out in hive
itself. Creating hbase filters is a much more universally known concept. Hence
my proposal for asking them to implement only that as it would ease things out
for them. Also since the CompositeKeyHBaseFactory got merged just a few days
back, I don't think we need to worry too much about passivity at this point by
making the class abstract.
Anyways, just my 0.02 :). I am fine with letting consumers define the complete
logic. Just in my opinion(and experience), it's not very easy.
CompositeKeyHBaseFactory should not use FamilyFilter
Key: HIVE-7048
URL: https://issues.apache.org/jira/browse/HIVE-7048
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Reporter: Swarnim Kulkarni
Assignee: Swarnim Kulkarni
Priority: Blocker
Attachments: HIVE-7048.1.patch.txt
HIVE-6411 introduced a more generic way to provide composite key
implementations via custom factory implementations. However it seems like the
CompositeHBaseKeyFactory implementation uses a FamilyFilter for row key scans
which doesn't seem appropriate. This should be investigated further and if
possible replaced with a RowRangeScanFilter.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7061) sql std auth - insert queries without overwrite should not require delete privileges
[ https://issues.apache.org/jira/browse/HIVE-7061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-7061: --- Status: Patch Available (was: Open) sql std auth - insert queries without overwrite should not require delete privileges Key: HIVE-7061 URL: https://issues.apache.org/jira/browse/HIVE-7061 Project: Hive Issue Type: Bug Components: Authorization, SQLStandardAuthorization Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-7061.1.patch, HIVE-7061.2.patch Insert queries can do the equivalent of delete and insert of all rows of a table or partition, if the overwrite keyword is used. As a result DELETE privilege is applicable to such queries. However, SQL Standard auth requires DELETE privilege even for queries that don't have the overwrite keyword. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HIVE-4515) select count(*) from table query on hive-0.10.0, hbase-0.94.7 integration throws exceptions
[
https://issues.apache.org/jira/browse/HIVE-4515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan resolved HIVE-4515.
Resolution: Invalid
Resolving this as invalid, per [~swarnim] previous comment. Feel free to reopen
if you are able to repro this on trunk.
select count(*) from table query on hive-0.10.0, hbase-0.94.7 integration
throws exceptions
-
Key: HIVE-4515
URL: https://issues.apache.org/jira/browse/HIVE-4515
Project: Hive
Issue Type: Bug
Components: HBase Handler
Affects Versions: 0.10.0, 0.11.0
Environment: hive-0.10.0, hive-0.11.0
hbase-0.94.7, hbase-0.94.6.1
zookeeper-3.4.3
hadoop-1.0.4
centos-5.7
Reporter: Yanhui Ma
Assignee: Swarnim Kulkarni
Priority: Critical
After integration hive-0.10.0+hbase-0.94.7, these commands could be executed
successfully:
{noformat}
create table
insert overwrite table
select * from table
However, when execute select count(*) from table, throws exception:
hive select count(*) from test;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapred.reduce.tasks=number
Starting Job = job_201305061042_0028, Tracking URL =
http://master0:50030/jobdetails.jsp?jobid=job_201305061042_0028
Kill Command = /opt/modules/hadoop/hadoop-1.0.4/libexec/../bin/hadoop job
-kill job_201305061042_0028
Hadoop job information for Stage-1: number of mappers: 1; number of reducers:
1
2013-05-07 18:41:42,649 Stage-1 map = 0%, reduce = 0%
2013-05-07 18:42:14,789 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201305061042_0028 with errors
Error during job, obtaining debugging information...
Job Tracking URL:
http://master0:50030/jobdetails.jsp?jobid=job_201305061042_0028
Examining task ID: task_201305061042_0028_m_02 (and more) from job
job_201305061042_0028
Task with the most failures(4):
-
Task ID:
task_201305061042_0028_m_00
URL:
http://master0:50030/taskdetails.jsp?jobid=job_201305061042_0028tipid=task_201305061042_0028_m_00
-
Diagnostic Messages for this Task:
java.lang.NegativeArraySizeException: -1
at org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:148)
at
org.apache.hadoop.hbase.mapreduce.TableSplit.readFields(TableSplit.java:133)
at
org.apache.hadoop.hive.hbase.HBaseSplit.readFields(HBaseSplit.java:53)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat$HiveInputSplit.readFields(HiveInputFormat.java:150)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:396)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:412)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
==
The log of tasktracker:
stderr logs
13/05/07 18:43:20 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/05/07 18:43:20 INFO mapred.TaskRunner: Creating symlink:
/tmp/hadoop-hadoop/mapred/local/taskTracker/distcache/107328478296390_-1298160740_2123690974/master0/tmp/hive-hadoop/hive_2013-05-07_18-41-30_290_832140779606816147/-mr-10003/fd22448b-e923-498c-bc00-2164ca68447d
-
/tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201305061042_0028/attempt_201305061042_0028_m_00_0/work/HIVE_PLANfd22448b-e923-498c-bc00-2164ca68447d
13/05/07 18:43:20 INFO filecache.TrackerDistributedCacheManager: Creating
symlink:
/tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201305061042_0028/jars/javolution
-
[jira] [Comment Edited] (HIVE-7084) TestWebHCatE2e is failing on trunk
[ https://issues.apache.org/jira/browse/HIVE-7084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004811#comment-14004811 ] Ashutosh Chauhan edited comment on HIVE-7084 at 5/21/14 3:55 PM: - Patch for reverting HIVE-7000 was (Author: ashutoshc): Patch for reverting HIVE-7084 TestWebHCatE2e is failing on trunk -- Key: HIVE-7084 URL: https://issues.apache.org/jira/browse/HIVE-7084 Project: Hive Issue Type: Test Components: WebHCat Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-7084.patch I am able to repro it consistently on fresh checkout. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7084) TestWebHCatE2e is failing on trunk
[ https://issues.apache.org/jira/browse/HIVE-7084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-7084: --- Assignee: Ashutosh Chauhan (was: Harish Butani) Status: Patch Available (was: Open) TestWebHCatE2e is failing on trunk -- Key: HIVE-7084 URL: https://issues.apache.org/jira/browse/HIVE-7084 Project: Hive Issue Type: Test Components: WebHCat Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-7084.patch I am able to repro it consistently on fresh checkout. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7048) CompositeKeyHBaseFactory should not use FamilyFilter
[ https://issues.apache.org/jira/browse/HIVE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004838#comment-14004838 ] Xuefu Zhang commented on HIVE-7048: --- The might be some misunderstanding. While CompositeKeyHBaseFactory was just recently introduced, HBaseSerDe.HBASE_COMPOSITE_KEY_CLASS is old. CompositeKeyHBaseFactory was to replace the old implementation around HBaseSerDe.HBASE_COMPOSITE_KEY_CLASS. The intention of extending this functionality is good, but the obvious difficulty of doing so w/o knowing much about the key would make me think twice. If we make CompositeKeyHBaseFactory abstract, user isn't able to use HBaseSerDe.HBASE_COMPOSITE_KEY_CLASS to specify a key class any more. CompositeKeyHBaseFactory should not use FamilyFilter Key: HIVE-7048 URL: https://issues.apache.org/jira/browse/HIVE-7048 Project: Hive Issue Type: Improvement Components: HBase Handler Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Priority: Blocker Attachments: HIVE-7048.1.patch.txt HIVE-6411 introduced a more generic way to provide composite key implementations via custom factory implementations. However it seems like the CompositeHBaseKeyFactory implementation uses a FamilyFilter for row key scans which doesn't seem appropriate. This should be investigated further and if possible replaced with a RowRangeScanFilter. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7104) Unit tests are disabled
[
https://issues.apache.org/jira/browse/HIVE-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004874#comment-14004874
]
Hive QA commented on HIVE-7104:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12645915/HIVE-7104.1.patch
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 5451 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_reduce_deduplicate_exclude_gby
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/255/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/255/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-255/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12645915
Unit tests are disabled
---
Key: HIVE-7104
URL: https://issues.apache.org/jira/browse/HIVE-7104
Project: Hive
Issue Type: Bug
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7104.1.patch
When I run {{mvn clean test -Phadoop-1|2}}, none of the unit tests are run. I
did a binary search through the commit logs and found that the change that
caused the unit tests to be disabled was the the change to the root pom.xml
in the patch for HIVE-7067 (e77f38dc44de5a9b10bce8e0a2f1f5452f6921ed).
Removing that change allowed the unit tests to be run again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7105) Enable ReduceRecordProcessor to generate VectorizedRowBatches
[ https://issues.apache.org/jira/browse/HIVE-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004882#comment-14004882 ] Eric Hanson commented on HIVE-7105: --- I agree with Remus. If you do want to get good performance with vectorization on the reduce side, you'll need to think carefully about how you can efficiently create full VectorizedRowBatches. Single-row or small VectorizedRowBatches will not give performance gains. Also, if it is expensive to load rows into the batches on the reduce side, that could dominate total runtime. Enable ReduceRecordProcessor to generate VectorizedRowBatches - Key: HIVE-7105 URL: https://issues.apache.org/jira/browse/HIVE-7105 Project: Hive Issue Type: Bug Components: Vectorization Reporter: Rajesh Balamohan Assignee: Jitendra Nath Pandey Attachments: HIVE-7105.1.patch Currently, ReduceRecordProcessor sends one key,value pair at a time to its operator pipeline. It would be beneficial to send VectorizedRowBatch to downstream operators. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6684) Beeline does not accept comments that are preceded by spaces
[ https://issues.apache.org/jira/browse/HIVE-6684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Beard updated HIVE-6684: --- Attachment: HIVE-6684.2.patch Good point Xuefu. Added patch that only trims once. Beeline does not accept comments that are preceded by spaces Key: HIVE-6684 URL: https://issues.apache.org/jira/browse/HIVE-6684 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.10.0 Reporter: Jeremy Beard Attachments: HIVE-6684.1.patch, HIVE-6684.2.patch Beeline throws an error if single-line comments are indented with spaces. This works in the embedded Hive CLI. For example: SELECT -- this is the field we want field FROM table; Error: Error while processing statement: FAILED: ParseException line 1:71 cannot recognize input near 'EOF' 'EOF' 'EOF' in select clause (state=42000,code=4) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6684) Beeline does not accept comments that are preceded by spaces
[ https://issues.apache.org/jira/browse/HIVE-6684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Beard updated HIVE-6684: --- Attachment: HIVE-6684.2.patch Beeline does not accept comments that are preceded by spaces Key: HIVE-6684 URL: https://issues.apache.org/jira/browse/HIVE-6684 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.10.0 Reporter: Jeremy Beard Attachments: HIVE-6684.1.patch, HIVE-6684.2.patch Beeline throws an error if single-line comments are indented with spaces. This works in the embedded Hive CLI. For example: SELECT -- this is the field we want field FROM table; Error: Error while processing statement: FAILED: ParseException line 1:71 cannot recognize input near 'EOF' 'EOF' 'EOF' in select clause (state=42000,code=4) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6684) Beeline does not accept comments that are preceded by spaces
[ https://issues.apache.org/jira/browse/HIVE-6684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Beard updated HIVE-6684: --- Attachment: (was: HIVE-6684.2.patch) Beeline does not accept comments that are preceded by spaces Key: HIVE-6684 URL: https://issues.apache.org/jira/browse/HIVE-6684 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.10.0 Reporter: Jeremy Beard Attachments: HIVE-6684.1.patch, HIVE-6684.2.patch Beeline throws an error if single-line comments are indented with spaces. This works in the embedded Hive CLI. For example: SELECT -- this is the field we want field FROM table; Error: Error while processing statement: FAILED: ParseException line 1:71 cannot recognize input near 'EOF' 'EOF' 'EOF' in select clause (state=42000,code=4) -- This message was sent by Atlassian JIRA (v6.2#6252)
Replacement for 'nextColumnsBatch' method in RCFile.Reader
Hi, All, I noticed ‘nextColumnsBatch’ is marked as ‘Deprecated’ in RCFile.Reader class. What is the method that will replace ‘nextColumnsBatch’? And why? Thanks John
[jira] [Commented] (HIVE-6994) parquet-hive createArray strips null elements
[ https://issues.apache.org/jira/browse/HIVE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004924#comment-14004924 ] Justin Coffey commented on HIVE-6994: - hmmm... good catch. It didn't get picked up by my qtest regex of parquet* and now that I run it locally I see it failing. I'll debug. parquet-hive createArray strips null elements - Key: HIVE-6994 URL: https://issues.apache.org/jira/browse/HIVE-6994 Project: Hive Issue Type: Bug Affects Versions: 0.13.0, 0.14.0 Reporter: Justin Coffey Assignee: Justin Coffey Fix For: 0.14.0 Attachments: HIVE-6994-1.patch, HIVE-6994.2.patch, HIVE-6994.3.patch, HIVE-6994.3.patch, HIVE-6994.patch The createArray method in ParquetHiveSerDe strips null values from resultant ArrayWritables. tracked here as well: https://github.com/Parquet/parquet-mr/issues/377 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7105) Enable ReduceRecordProcessor to generate VectorizedRowBatches
[ https://issues.apache.org/jira/browse/HIVE-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-7105: -- Fix Version/s: 0.14.0 Enable ReduceRecordProcessor to generate VectorizedRowBatches - Key: HIVE-7105 URL: https://issues.apache.org/jira/browse/HIVE-7105 Project: Hive Issue Type: Bug Components: Tez, Vectorization Reporter: Rajesh Balamohan Assignee: Jitendra Nath Pandey Fix For: 0.14.0 Attachments: HIVE-7105.1.patch Currently, ReduceRecordProcessor sends one key,value pair at a time to its operator pipeline. It would be beneficial to send VectorizedRowBatch to downstream operators. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7085) TestOrcHCatPigStorer.testWriteDecimal tests are failing on trunk
[
https://issues.apache.org/jira/browse/HIVE-7085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004933#comment-14004933
]
Ashutosh Chauhan commented on HIVE-7085:
Additionally, {{TestHCatLoader.testReadDataPrimitiveTypes}} also seems to be
flaky.
TestOrcHCatPigStorer.testWriteDecimal tests are failing on trunk
Key: HIVE-7085
URL: https://issues.apache.org/jira/browse/HIVE-7085
Project: Hive
Issue Type: Test
Components: HCatalog
Affects Versions: 0.14.0
Reporter: Ashutosh Chauhan
TestOrcHCatPigStorer.testWriteDecimal,
TestOrcHCatPigStorer.testWriteDecimalX,
TestOrcHCatPigStorer.testWriteDecimalXY
are failing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7105) Enable ReduceRecordProcessor to generate VectorizedRowBatches
[ https://issues.apache.org/jira/browse/HIVE-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-7105: -- Component/s: Tez Enable ReduceRecordProcessor to generate VectorizedRowBatches - Key: HIVE-7105 URL: https://issues.apache.org/jira/browse/HIVE-7105 Project: Hive Issue Type: Bug Components: Tez, Vectorization Reporter: Rajesh Balamohan Assignee: Jitendra Nath Pandey Fix For: 0.14.0 Attachments: HIVE-7105.1.patch Currently, ReduceRecordProcessor sends one key,value pair at a time to its operator pipeline. It would be beneficial to send VectorizedRowBatch to downstream operators. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6684) Beeline does not accept comments that are preceded by spaces
[ https://issues.apache.org/jira/browse/HIVE-6684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004945#comment-14004945 ] Xuefu Zhang commented on HIVE-6684: --- +1 [~leftylev] Comments are applicable to script from -e, -f, jdbc, or beeline. The bug was probably there since day one. Beeline does not accept comments that are preceded by spaces Key: HIVE-6684 URL: https://issues.apache.org/jira/browse/HIVE-6684 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.10.0 Reporter: Jeremy Beard Attachments: HIVE-6684.1.patch, HIVE-6684.2.patch Beeline throws an error if single-line comments are indented with spaces. This works in the embedded Hive CLI. For example: SELECT -- this is the field we want field FROM table; Error: Error while processing statement: FAILED: ParseException line 1:71 cannot recognize input near 'EOF' 'EOF' 'EOF' in select clause (state=42000,code=4) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-7106) Predicates in where clause are not getting pushed beyond join for left outer joins
Ashutosh Chauhan created HIVE-7106: -- Summary: Predicates in where clause are not getting pushed beyond join for left outer joins Key: HIVE-7106 URL: https://issues.apache.org/jira/browse/HIVE-7106 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Inner joins they indeed are getting pushed up. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7106) Predicates in where clause are not getting pushed beyond join for left outer joins
[
https://issues.apache.org/jira/browse/HIVE-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004956#comment-14004956
]
Ashutosh Chauhan commented on HIVE-7106:
This is easier to demonstrate via strict mode config which requires predicates
on partitioning columns to be specified for partitioned tables.
{code}
hive create table t1 (a int, b string) partitioned by (c string);
hive create table t2 (a int, b string) partitioned by (c string);
hive set hive.mapred.mode=strict;
hive explain select * from t1 left outer join t2 on t1.a = t2.a where t2.c =
'abc';
FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: No
partition predicate found for Alias TS Table t2
{code}
Predicates in where clause are not getting pushed beyond join for left outer
joins
--
Key: HIVE-7106
URL: https://issues.apache.org/jira/browse/HIVE-7106
Project: Hive
Issue Type: Improvement
Reporter: Ashutosh Chauhan
Inner joins they indeed are getting pushed up.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7106) Predicates in where clause are not getting pushed beyond join for left outer joins
[ https://issues.apache.org/jira/browse/HIVE-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004959#comment-14004959 ] Ashutosh Chauhan commented on HIVE-7106: If someone wants to take this up, logic in o.a.h.h.ql.ppd.OpProcFactory::getQualifiedAliases() needs to be improved for it. Predicates in where clause are not getting pushed beyond join for left outer joins -- Key: HIVE-7106 URL: https://issues.apache.org/jira/browse/HIVE-7106 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Inner joins they indeed are getting pushed up. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-7107) Clean up HiveDrivers and fix an issue
Lars Francke created HIVE-7107:
--
Summary: Clean up HiveDrivers and fix an issue
Key: HIVE-7107
URL: https://issues.apache.org/jira/browse/HIVE-7107
Project: Hive
Issue Type: Bug
Reporter: Lars Francke
Priority: Minor
The HiveServer1 driver does not adhere to the JDBC standard by returning null
when it can't handle a connection URL. It instead throws an exception which
leads to subsequent exceptions (from other drivers, i.e. the HiveServer2 one)
being swallowed.
This is what you'd see:
{quote}
Error: Invalid URL:
jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com
(state=08S01,code=0)
java.sql.SQLException: Invalid URL:
jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com
at
org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:86)
{quote}
In addition this patch cleans up the drivers a bit.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7108) Cleanup HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swarnim Kulkarni updated HIVE-7108: --- Priority: Minor (was: Major) Cleanup HBaseStorageHandler --- Key: HIVE-7108 URL: https://issues.apache.org/jira/browse/HIVE-7108 Project: Hive Issue Type: Task Components: HBase Handler Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Priority: Minor HBaseStorageHandler has bunch of variables and unused imports that should be cleaned up. We should probably also revisit our findbugs and checkstyle plugins to see why they did not catch these issues in first place. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-7108) Cleanup HBaseStorageHandler
Swarnim Kulkarni created HIVE-7108: -- Summary: Cleanup HBaseStorageHandler Key: HIVE-7108 URL: https://issues.apache.org/jira/browse/HIVE-7108 Project: Hive Issue Type: Task Components: HBase Handler Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni HBaseStorageHandler has bunch of variables and unused imports that should be cleaned up. We should probably also revisit our findbugs and checkstyle plugins to see why they did not catch these issues in first place. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7107) Clean up HiveDrivers and fix an issue
[
https://issues.apache.org/jira/browse/HIVE-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Francke updated HIVE-7107:
---
Assignee: Lars Francke
Status: Patch Available (was: Open)
Clean up HiveDrivers and fix an issue
-
Key: HIVE-7107
URL: https://issues.apache.org/jira/browse/HIVE-7107
Project: Hive
Issue Type: Bug
Reporter: Lars Francke
Assignee: Lars Francke
Priority: Minor
Attachments: HIVE-7107.2.patch
The HiveServer1 driver does not adhere to the JDBC standard by returning null
when it can't handle a connection URL. It instead throws an exception which
leads to subsequent exceptions (from other drivers, i.e. the HiveServer2 one)
being swallowed.
This is what you'd see:
{quote}
Error: Invalid URL:
jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com
(state=08S01,code=0)
java.sql.SQLException: Invalid URL:
jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com
at
org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:86)
{quote}
In addition this patch cleans up the drivers a bit.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7107) Clean up HiveDrivers and fix an issue
[
https://issues.apache.org/jira/browse/HIVE-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Francke updated HIVE-7107:
---
Attachment: (was: HIVE-7107.1.patch)
Clean up HiveDrivers and fix an issue
-
Key: HIVE-7107
URL: https://issues.apache.org/jira/browse/HIVE-7107
Project: Hive
Issue Type: Bug
Reporter: Lars Francke
Priority: Minor
Attachments: HIVE-7107.2.patch
The HiveServer1 driver does not adhere to the JDBC standard by returning null
when it can't handle a connection URL. It instead throws an exception which
leads to subsequent exceptions (from other drivers, i.e. the HiveServer2 one)
being swallowed.
This is what you'd see:
{quote}
Error: Invalid URL:
jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com
(state=08S01,code=0)
java.sql.SQLException: Invalid URL:
jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com
at
org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:86)
{quote}
In addition this patch cleans up the drivers a bit.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-7109) Resource leak in HBaseStorageHandler
Swarnim Kulkarni created HIVE-7109:
--
Summary: Resource leak in HBaseStorageHandler
Key: HIVE-7109
URL: https://issues.apache.org/jira/browse/HIVE-7109
Project: Hive
Issue Type: Bug
Components: HBase Handler
Reporter: Swarnim Kulkarni
Assignee: Swarnim Kulkarni
The preCreateTable method in the HBaseStorageHandler checks that the HBase
table is still online by creating a new instance of HTable
{code}
// ensure the table is online
new HTable(hbaseConf, tableDesc.getName());
{code}
However this instance is never closed. So if this test succeeds, we would have
a resource leak in the code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7107) Clean up HiveDrivers and fix an issue
[
https://issues.apache.org/jira/browse/HIVE-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Francke updated HIVE-7107:
---
Attachment: HIVE-7107.2.patch
Clean up HiveDrivers and fix an issue
-
Key: HIVE-7107
URL: https://issues.apache.org/jira/browse/HIVE-7107
Project: Hive
Issue Type: Bug
Reporter: Lars Francke
Priority: Minor
Attachments: HIVE-7107.1.patch, HIVE-7107.2.patch
The HiveServer1 driver does not adhere to the JDBC standard by returning null
when it can't handle a connection URL. It instead throws an exception which
leads to subsequent exceptions (from other drivers, i.e. the HiveServer2 one)
being swallowed.
This is what you'd see:
{quote}
Error: Invalid URL:
jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com
(state=08S01,code=0)
java.sql.SQLException: Invalid URL:
jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com
at
org.apache.hadoop.hive.jdbc.HiveConnection.init(HiveConnection.java:86)
{quote}
In addition this patch cleans up the drivers a bit.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7108) Cleanup HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swarnim Kulkarni updated HIVE-7108: --- Priority: Major (was: Minor) Cleanup HBaseStorageHandler --- Key: HIVE-7108 URL: https://issues.apache.org/jira/browse/HIVE-7108 Project: Hive Issue Type: Task Components: HBase Handler Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni HBaseStorageHandler has bunch of variables and unused imports that should be cleaned up. We should probably also revisit our findbugs and checkstyle plugins to see why they did not catch these issues in first place. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7104) Unit tests are disabled
[
https://issues.apache.org/jira/browse/HIVE-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005083#comment-14005083
]
Szehon Ho commented on HIVE-7104:
-
NIce catch. Fortunately the Hive build is running all tests individually so
its not hitting this issue, but this affects devs trying to run unit-tests
outside /itests. [~prasanth_j] was that pom change committed by mistake ?
Unit tests are disabled
---
Key: HIVE-7104
URL: https://issues.apache.org/jira/browse/HIVE-7104
Project: Hive
Issue Type: Bug
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7104.1.patch
When I run {{mvn clean test -Phadoop-1|2}}, none of the unit tests are run. I
did a binary search through the commit logs and found that the change that
caused the unit tests to be disabled was the the change to the root pom.xml
in the patch for HIVE-7067 (e77f38dc44de5a9b10bce8e0a2f1f5452f6921ed).
Removing that change allowed the unit tests to be run again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7104) Unit tests are disabled
[
https://issues.apache.org/jira/browse/HIVE-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005096#comment-14005096
]
Prasanth J commented on HIVE-7104:
--
+1
Unit tests are disabled
---
Key: HIVE-7104
URL: https://issues.apache.org/jira/browse/HIVE-7104
Project: Hive
Issue Type: Bug
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7104.1.patch
When I run {{mvn clean test -Phadoop-1|2}}, none of the unit tests are run. I
did a binary search through the commit logs and found that the change that
caused the unit tests to be disabled was the the change to the root pom.xml
in the patch for HIVE-7067 (e77f38dc44de5a9b10bce8e0a2f1f5452f6921ed).
Removing that change allowed the unit tests to be run again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7104) Unit tests are disabled
[
https://issues.apache.org/jira/browse/HIVE-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005095#comment-14005095
]
Prasanth J commented on HIVE-7104:
--
Oops! My bad. That was a mistake. I added that to run unit test locally in my
environment. I forgot to revert that while committing. Sorry for the
inconvenience. [~davidzchen] thanks for catching it!
Unit tests are disabled
---
Key: HIVE-7104
URL: https://issues.apache.org/jira/browse/HIVE-7104
Project: Hive
Issue Type: Bug
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7104.1.patch
When I run {{mvn clean test -Phadoop-1|2}}, none of the unit tests are run. I
did a binary search through the commit logs and found that the change that
caused the unit tests to be disabled was the the change to the root pom.xml
in the patch for HIVE-7067 (e77f38dc44de5a9b10bce8e0a2f1f5452f6921ed).
Removing that change allowed the unit tests to be run again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7099) Add Decimal datatype support for Windowing
[ https://issues.apache.org/jira/browse/HIVE-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-7099: Status: Patch Available (was: Open) Add Decimal datatype support for Windowing -- Key: HIVE-7099 URL: https://issues.apache.org/jira/browse/HIVE-7099 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-7099.1.patch, HIVE-7099.2.patch Decimal datatype is not handled by Windowing -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7099) Add Decimal datatype support for Windowing
[ https://issues.apache.org/jira/browse/HIVE-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-7099: Attachment: HIVE-7099.2.patch Add Decimal datatype support for Windowing -- Key: HIVE-7099 URL: https://issues.apache.org/jira/browse/HIVE-7099 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-7099.1.patch, HIVE-7099.2.patch Decimal datatype is not handled by Windowing -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7099) Add Decimal datatype support for Windowing
[ https://issues.apache.org/jira/browse/HIVE-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-7099: Status: Open (was: Patch Available) Add Decimal datatype support for Windowing -- Key: HIVE-7099 URL: https://issues.apache.org/jira/browse/HIVE-7099 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-7099.1.patch, HIVE-7099.2.patch Decimal datatype is not handled by Windowing -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6147) Support avro data stored in HBase columns
[ https://issues.apache.org/jira/browse/HIVE-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swarnim Kulkarni updated HIVE-6147: --- Issue Type: Improvement (was: Bug) Support avro data stored in HBase columns - Key: HIVE-6147 URL: https://issues.apache.org/jira/browse/HIVE-6147 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.12.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-6147.1.patch.txt, HIVE-6147.2.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.4.patch.txt, HIVE-6147.5.patch.txt Presently, the HBase Hive integration supports querying only primitive data types in columns. It would be nice to be able to store and query Avro objects in HBase columns by making them visible as structs to Hive. This will allow Hive to perform ad hoc analysis of HBase data which can be deeply structured. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6928) Beeline should not chop off describe extended results by default
[ https://issues.apache.org/jira/browse/HIVE-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005125#comment-14005125 ] Szehon Ho commented on HIVE-6928: - Thanks [~chinnalalam] for the fix. One minor nit here, as there is no review-board, do you think it would be better to call BufferedRows constructor width argument to be 'consoleWidth' for clarity, to disambiguate from row width. I was also wondering do we need an upper-bound as now it will show entire result length even if its huge, but Hive CLI did not have that either. Beeline should not chop off describe extended results by default -- Key: HIVE-6928 URL: https://issues.apache.org/jira/browse/HIVE-6928 Project: Hive Issue Type: Bug Components: CLI Reporter: Szehon Ho Assignee: Chinna Rao Lalam Attachments: HIVE-6928.patch By default, beeline truncates long results based on the console width like: +-+--+ | col_name | | +-+--+ | pat_id | string | | score | float | | acutes | float | | | | | Detailed Table Information | Table(tableName:refills, dbName:default, owner:hdadmin, createTime:1393882396, lastAccessTime:0, retention:0, sd:Sto | +-+--+ 5 rows selected (0.4 seconds) This can be changed by !outputformat, but the default should behave better to give a better experience to the first-time beeline user. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7095) Fix test fails for both hadoop-1 and hadoop-2
[
https://issues.apache.org/jira/browse/HIVE-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005189#comment-14005189
]
Hive QA commented on HIVE-7095:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12645919/HIVE-7095.1.patch.txt
{color:red}ERROR:{color} -1 due to 331 failed/errored test(s), 5453 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_tblproperties
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unset_table_view_property
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_pushdown
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_genericFileFormat
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby2_map_skew_multi_distinct
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby2_multi_distinct
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby3_map_skew_multi_distinct
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby3_multi_distinct
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby_cube1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby_cube2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby_grouping_id1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby_grouping_sets1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby_grouping_sets2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby_grouping_sets3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby_grouping_sets4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby_grouping_sets5
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby_grouping_sets6
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby_grouping_sets7
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby_invalid_position
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby_key
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby_rollup1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_groupby_rollup2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_having1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_illegal_partition_type
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_illegal_partition_type2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_illegal_partition_type3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_illegal_partition_type4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_index_bitmap_no_map_aggr
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_index_compact_entry_limit
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_index_compact_size_limit
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_input1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_input2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_input4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_input41
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_input_part0_neg
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into5
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into6
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_view_failure
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insertexternal1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insertover_dynapart_ifnotexists
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_invalid_arithmetic_type
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_invalid_avg_syntax
[jira] [Commented] (HIVE-7025) Support retention on hive tables
[
https://issues.apache.org/jira/browse/HIVE-7025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005192#comment-14005192
]
Hive QA commented on HIVE-7025:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12645951/HIVE-7025.2.patch.txt
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/257/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/257/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-257/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-257/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted
'itests/util/src/main/java/org/apache/hadoop/hive/ql/hooks/VerifyNumReducersHook.java'
Reverted 'itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java'
Reverted 'ql/src/test/results/clientpositive/groupby_sort_1.q.out'
Reverted 'ql/src/test/results/clientpositive/show_create_table_alter.q.out'
Reverted 'ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_sort_skew_1.q.out'
Reverted 'ql/src/test/results/clientpositive/nullformatCTAS.q.out'
Reverted 'ql/src/test/queries/clientpositive/groupby_sort_1.q'
Reverted 'ql/src/test/queries/clientpositive/bucketizedhiveinputformat.q'
Reverted 'ql/src/test/queries/clientpositive/groupby_sort_skew_1.q'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java'
Reverted
'ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/Driver.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target
shims/0.20S/target shims/0.23/target shims/aggregator/target
shims/common/target shims/common-secure/target packaging/target
hbase-handler/target testutils/target jdbc/target metastore/target
itests/target itests/hcatalog-unit/target itests/test-serde/target
itests/qtest/target itests/hive-minikdc/target itests/hive-unit/target
itests/custom-serde/target itests/util/target
itests/util/src/main/java/org/apache/hadoop/hive/ql/hooks/ShowMapredStatsHook.java
hcatalog/target hcatalog/core/target hcatalog/streaming/target
hcatalog/server-extensions/target hcatalog/webhcat/svr/target
hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target
hwi/target common/target common/src/gen contrib/target service/target
serde/target beeline/target odbc/target cli/target
ql/dependency-reduced-pom.xml ql/target
ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out
ql/src/test/results/clientpositive/groupby_sort_1_23.q.out
ql/src/test/queries/clientpositive/groupby_sort_skew_1_23.q
ql/src/test/queries/clientpositive/groupby_sort_1_23.q
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1596672.
At revision 1596672.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
[jira] [Commented] (HIVE-7076) Plugin (exec hook) to log to application timeline data to Yarn
[ https://issues.apache.org/jira/browse/HIVE-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005218#comment-14005218 ] Thejas M Nair commented on HIVE-7076: - I have added some minor comments on review board. We also need to figure out how to make this change work with 1.x build. Plugin (exec hook) to log to application timeline data to Yarn -- Key: HIVE-7076 URL: https://issues.apache.org/jira/browse/HIVE-7076 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7076.1.patch See: https://issues.apache.org/jira/browse/YARN-1530 This is a simple pre/post exec hook to log query + plan information to yarn. This information can be used to build tools and UIs to monitor, track, debug and tune Hive queries. Off by default, but can be enabled via: hive.exec.pre.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.post.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.failure.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7104) Unit tests are disabled
[
https://issues.apache.org/jira/browse/HIVE-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-7104:
-
Resolution: Fixed
Fix Version/s: 0.14.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks David!
Unit tests are disabled
---
Key: HIVE-7104
URL: https://issues.apache.org/jira/browse/HIVE-7104
Project: Hive
Issue Type: Bug
Reporter: David Chen
Assignee: David Chen
Fix For: 0.14.0
Attachments: HIVE-7104.1.patch
When I run {{mvn clean test -Phadoop-1|2}}, none of the unit tests are run. I
did a binary search through the commit logs and found that the change that
caused the unit tests to be disabled was the the change to the root pom.xml
in the patch for HIVE-7067 (e77f38dc44de5a9b10bce8e0a2f1f5452f6921ed).
Removing that change allowed the unit tests to be run again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Plan: permanently move hive builds from bigtop
The pre-commit builds should be officially good now. The test logs are properly downloaded, and correctly displayed on the HiveQA JIRA comment as the link under TestReport, and all logs themselves in raw form are at Test logs, see HIVE-6937 and HIVE-7080 for details. Also as we had troubles with the patch-upload trigger job on Apache Jenkins having outages and not triggering the test, so we also moved that to the Hive Jenkins on the same server: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-Admin/, which now runs the hadoopQA script every 10 minutes. I'll continue to monitor this one, as its still new. Also as some might know already, the Pre-commit build is now using -Phadoop-2 and JDK1.7. Thanks Szehon http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-Admin/ On Tue, Apr 22, 2014 at 4:15 PM, Szehon Ho sze...@cloudera.com wrote: Patch upload auto-trigger is now enabled, thanks Ashutosh for helping me fix the Jenkins url from Apache-Infra side. Now there's no need to manually submit Pre-commit builds, unless it got missed somehow. And in this Jenkins, you can easily see all the JIRA's in the queue: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/ Sorry, please use this URL and not the first one, as job was renamed due to the auto-trigger expecting this format. And to repeat, Test Reporting and the 'Logs located at:' link is still broken and tracked in HIVE-6937. For now, to see the logs of a failed build, you have to manually navigate. Example, for PreCommit-Hive-BUILD#6, logs are at: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-6/http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-6/. The HiveQA comment on the JIRA should mention the build number. Thanks, Szehon On Sun, Apr 20, 2014 at 11:07 PM, Szehon Ho sze...@cloudera.com wrote: It looks great, thanks Lefty! On Sun, Apr 20, 2014 at 2:22 PM, Lefty Leverenz leftylever...@gmail.comwrote: Nice doc, Szehon. I did some minor editing so you might want to make sure I didn't introduce any errors. https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure -- Lefty On Sat, Apr 19, 2014 at 9:45 PM, Szehon Ho sze...@cloudera.com wrote: Migration is done, I updated the wiki to add all the details of the new setup: https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure New Jenkins URL to submit pre-commit jobs: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/precommit-hive/ Again, this has to be done manually for time being, by clicking on 'build with parameters', and entering the issue number as a parameter. I've submitted some already. I'll reach out to some committers to get the auto-trigger working. As I mentioned, there is some work to fix the test-reporting, due to the framework using old url scheme. I am tracking it at HIVE-6937https://issues.apache.org/jira/browse/HIVE-6937. For now I am hosting log directory separately, if you want to see test logs, you have to manually go the url corresponding to your build, like: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/precommit-hive-11/for run#11. Sorry about that. Let me know if you see other issues, thanks! Szehon On Fri, Apr 18, 2014 at 2:11 PM, Thejas Nair the...@hortonworks.com wrote: Sounds good. Thanks Szehon! On Fri, Apr 18, 2014 at 10:17 AM, Ashutosh Chauhan hashut...@apache.org wrote: +1 Thanks Szehon! On Fri, Apr 18, 2014 at 6:29 AM, Xuefu Zhang xzh...@cloudera.com wrote: +1. Thanks for taking care of this. On Thu, Apr 17, 2014 at 11:00 PM, Szehon Ho sze...@cloudera.com wrote: Hi, This week the machine running Hive builds at http://bigtop01.cloudera.org:8080/view/Hive/? ran out of space, so new jobs like Precommit tests stopped. Its still not resolved there, there was another email today on Bigtop list, but there's very few people with root access to that host, and they still haven't responded. I chatted with Brock, he has also seen various issues with Bigtop jenkins in the past, so I am thinking to move the Jenkins jobs to the PTest master itself, where some PMC already have access and can admin if needed. Currently I am hosting the pre-commit Jenkins job on my own EC2 instance as stop-gap. Other advantages of hosting our own Jenkins: 1. No need to wait for other Bigtop jobs to run. 2. Bigtop is using a version of Jenkins that doesnt show parameters like JIRA number for queued jobs, so impossible to tell whether a patch got picked up and where it is in queue. 3. Eliminate network hop from Bigtop box to our PTest master. The disadvantage is: 1. We don't have much
[jira] [Updated] (HIVE-7066) hive-exec jar is missing avro core
[
https://issues.apache.org/jira/browse/HIVE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-7066:
-
Component/s: Build Infrastructure
hive-exec jar is missing avro core
--
Key: HIVE-7066
URL: https://issues.apache.org/jira/browse/HIVE-7066
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.13.0, 0.13.1
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7066.1.patch
Running a simple query that reads an Avro table caused the following
exception to be thrown on the cluster side:
{code}
java.lang.RuntimeException:
org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.IllegalArgumentException: Unable to create serializer
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer for
class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
Serialization trace:
outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:365)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:276)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:445)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:438)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.init(MapTask.java:191)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:412)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.IllegalArgumentException: Unable to create serializer
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer for
class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
Serialization trace:
outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
at
org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:942)
at
org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:850)
at
org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:864)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:334)
... 13 more
Caused by: java.lang.IllegalArgumentException: Unable to create serializer
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer for
class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
at
org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
at
org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:26)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:343)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:336)
at
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:56)
at
[jira] [Updated] (HIVE-7066) hive-exec jar is missing avro core
[
https://issues.apache.org/jira/browse/HIVE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-7066:
-
Affects Version/s: 0.13.1
0.13.0
hive-exec jar is missing avro core
--
Key: HIVE-7066
URL: https://issues.apache.org/jira/browse/HIVE-7066
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.13.0, 0.13.1
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7066.1.patch
Running a simple query that reads an Avro table caused the following
exception to be thrown on the cluster side:
{code}
java.lang.RuntimeException:
org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.IllegalArgumentException: Unable to create serializer
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer for
class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
Serialization trace:
outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:365)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:276)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:445)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:438)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.init(MapTask.java:191)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:412)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.IllegalArgumentException: Unable to create serializer
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer for
class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
Serialization trace:
outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
at
org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:942)
at
org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:850)
at
org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:864)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:334)
... 13 more
Caused by: java.lang.IllegalArgumentException: Unable to create serializer
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer for
class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
at
org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
at
org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:26)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:343)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:336)
at
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:56)
at
[jira] [Updated] (HIVE-7066) hive-exec jar is missing avro core
[
https://issues.apache.org/jira/browse/HIVE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-7066:
-
Resolution: Fixed
Fix Version/s: 0.14.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks David!
hive-exec jar is missing avro core
--
Key: HIVE-7066
URL: https://issues.apache.org/jira/browse/HIVE-7066
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.13.0, 0.13.1
Reporter: David Chen
Assignee: David Chen
Fix For: 0.14.0
Attachments: HIVE-7066.1.patch
Running a simple query that reads an Avro table caused the following
exception to be thrown on the cluster side:
{code}
java.lang.RuntimeException:
org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.IllegalArgumentException: Unable to create serializer
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer for
class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
Serialization trace:
outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:365)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:276)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:445)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:438)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.init(MapTask.java:191)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:412)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.IllegalArgumentException: Unable to create serializer
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer for
class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
Serialization trace:
outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
at
org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:942)
at
org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:850)
at
org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:864)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:334)
... 13 more
Caused by: java.lang.IllegalArgumentException: Unable to create serializer
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer for
class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
at
org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
at
org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:26)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:343)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:336)
at
Re: Is there a bug in hiveserver2 metastore caused hold huge objects
Thanks a lot for the dump Meng! On Wed, May 21, 2014 at 1:41 AM, Meng QingPing mqingp...@gmail.com wrote: I post the heap bin file on http://jack-meng.qiniudn.com/heap.bin Thanks. 2014-05-21 15:41 GMT+08:00 Meng QingPing mqingp...@gmail.com: There are 18908 PM-s. I post it on http://jack-meng.qiniudn.com/hive-heap2.png Thanks. 2014-05-20 0:41 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: What is the count of PM-s that you are seeing when it crashes? From that screenshot I don't see what the problem is or I'm not reading it right. On Mon, May 19, 2014 at 1:17 AM, Meng QingPing mqingp...@gmail.com wrote: Thanks. I post it on http://jack-meng.qiniudn.com/hive-heap.png 2014-05-06 23:14 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: I don't think the dev list supports attachment. Can you post to some image sharing service? On Tue, May 6, 2014 at 3:20 AM, Meng QingPing mqingp...@gmail.com wrote: Weird, the picture attached in sent mail. Anyway send again. 2014-05-05 12:17 GMT+08:00 Chandra Reddy chandu...@gmail.com: seems you have missed attachment. -Chandra On Sun, May 4, 2014 at 6:57 PM, Meng QingPing mqingp...@gmail.com wrote: I run hiveserver2 with metastore in mysql. The hiveserver2 OOM and the heap dump show huge objects hold by org.datanucleus.api.jdo. JDOPersistenceManagerFactory as attached. It seems not release org.datanucleus.api.jdo.JDOPersistenceManager. Hive version is 0.13. Thanks, Jack -- Thanks, -Chandra. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Is there a bug in hiveserver2 metastore caused hold huge objects
Meng, are you running a remote metastore or an embedded one (which would be true if you added this config while starting HS2: -hiveconf hive.metastore.uris= )? Thanks, --Vaibhav On Wed, May 21, 2014 at 2:28 PM, Vaibhav Gumashta vgumas...@hortonworks.com wrote: Thanks a lot for the dump Meng! On Wed, May 21, 2014 at 1:41 AM, Meng QingPing mqingp...@gmail.comwrote: I post the heap bin file on http://jack-meng.qiniudn.com/heap.bin Thanks. 2014-05-21 15:41 GMT+08:00 Meng QingPing mqingp...@gmail.com: There are 18908 PM-s. I post it on http://jack-meng.qiniudn.com/hive-heap2.png Thanks. 2014-05-20 0:41 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: What is the count of PM-s that you are seeing when it crashes? From that screenshot I don't see what the problem is or I'm not reading it right. On Mon, May 19, 2014 at 1:17 AM, Meng QingPing mqingp...@gmail.com wrote: Thanks. I post it on http://jack-meng.qiniudn.com/hive-heap.png 2014-05-06 23:14 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com : I don't think the dev list supports attachment. Can you post to some image sharing service? On Tue, May 6, 2014 at 3:20 AM, Meng QingPing mqingp...@gmail.com wrote: Weird, the picture attached in sent mail. Anyway send again. 2014-05-05 12:17 GMT+08:00 Chandra Reddy chandu...@gmail.com: seems you have missed attachment. -Chandra On Sun, May 4, 2014 at 6:57 PM, Meng QingPing mqingp...@gmail.com wrote: I run hiveserver2 with metastore in mysql. The hiveserver2 OOM and the heap dump show huge objects hold by org.datanucleus.api.jdo. JDOPersistenceManagerFactory as attached. It seems not release org.datanucleus.api.jdo.JDOPersistenceManager. Hive version is 0.13. Thanks, Jack -- Thanks, -Chandra. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Review Request 21749: HIVE-7061 - sql std auth - insert queries without overwrite should not require delete privileges
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21749/#review43659 --- ql/src/java/org/apache/hadoop/hive/ql/Driver.java https://reviews.apache.org/r/21749/#comment77941 To keep Driver file size in control, shall we move this to AuthorizationUtils class ? ql/src/java/org/apache/hadoop/hive/ql/Driver.java https://reviews.apache.org/r/21749/#comment77943 This kind of null check may result in DEFAULT being returned which will be undesirable. I think instead of that we should make sure that writeType can never be null here and don't do null check here. ql/src/java/org/apache/hadoop/hive/ql/Driver.java https://reviews.apache.org/r/21749/#comment77944 What other writeType is legal here? This should throw unsupported writeType instead I think. ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HivePrivilegeObject.java https://reviews.apache.org/r/21749/#comment77948 Better name : HivePrivType? ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HivePrivilegeObject.java https://reviews.apache.org/r/21749/#comment77947 Better name : HivePrivActionType ? Object in there sounds redundant ? ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HivePrivilegeObject.java https://reviews.apache.org/r/21749/#comment77942 Why do we need DEFAULT here? Its good to document what actions it covers. Else, I think UNKNOWN is better name. ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java https://reviews.apache.org/r/21749/#comment77949 Better name : RequiredPrivs? ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java https://reviews.apache.org/r/21749/#comment77950 In what case, actionType could be null ? Good to document here. - Ashutosh Chauhan On May 21, 2014, 1:49 a.m., Thejas Nair wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21749/ --- (Updated May 21, 2014, 1:49 a.m.) Review request for hive, Ashutosh Chauhan and Thejas Nair. Bugs: HIVE-7061 https://issues.apache.org/jira/browse/HIVE-7061 Repository: hive-git Description --- See bug Diffs - ql/src/java/org/apache/hadoop/hive/ql/Driver.java 9040d9b ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HivePrivilegeObject.java a3a689d ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java b0a804c ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAuthorizationValidator.java 229c063 ql/src/test/queries/clientnegative/authorization_insertoverwrite_nodel.q PRE-CREATION ql/src/test/queries/clientpositive/authorization_insert.q PRE-CREATION ql/src/test/results/clientnegative/authorization_insert_noinspriv.q.out ee8d49e ql/src/test/results/clientnegative/authorization_insertoverwrite_nodel.q.out PRE-CREATION ql/src/test/results/clientpositive/authorization_insert.q.out PRE-CREATION Diff: https://reviews.apache.org/r/21749/diff/ Testing --- Test included. Thanks, Thejas Nair
[jira] [Updated] (HIVE-7095) Fix test fails for both hadoop-1 and hadoop-2
[ https://issues.apache.org/jira/browse/HIVE-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-7095: Attachment: HIVE-7095.2.patch.txt Fix test fails for both hadoop-1 and hadoop-2 - Key: HIVE-7095 URL: https://issues.apache.org/jira/browse/HIVE-7095 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Navis Attachments: HIVE-7095.1.patch.txt, HIVE-7095.2.patch.txt There are some try to fix test results for hadoop-2(HIVE-6838, HIVE-6979, etc.). But these remained problems for hadoop-1. Let's finish this here, not prolonging output ping-pongs between two versions. 1. Data size in explain result is different bucketizedhiveinputformat.q 2. forced bucketing(=2) does not make two files in hadoop-1. groupby_sort_1.q groupby_sort_skew_1.q 3. Column stats of srcpart, which has loaded a file rather than inserted, has COLUMN_STATS_ACCURATE=true, numRows=0, rawDataSize=0 for hadoop-2, which seemed wrong. sample8.q transform_ppr1.q transform_ppr2.q union_ppr.q 4. Iteration order of table properties for show create table is different. nullformatCTAS.q show_create_table_alter.q 5. Iteration order of show table properties is different unset_table_view_property.q -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 21748: Fix test fails for both hadoop-1 and hadoop-2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21748/ --- (Updated May 22, 2014, 1:07 a.m.) Review request for hive. Changes --- Fixed test fails Bugs: HIVE-7095 https://issues.apache.org/jira/browse/HIVE-7095 Repository: hive-git Description --- There are some try to fix test results for hadoop-2(HIVE-6838, HIVE-6979, etc.). But these remained problems for hadoop-1. Let's finish this here, not prolonging output ping-pongs between two versions. 1. Data size in explain result is different bucketizedhiveinputformat.q 2. forced bucketing(=2) does not make two files in hadoop-1. groupby_sort_1.q groupby_sort_skew_1.q 3. Column stats of srcpart, which has loaded a file rather than inserted, has COLUMN_STATS_ACCURATE=true, numRows=0, rawDataSize=0 for hadoop-2, which seemed wrong. sample8.q transform_ppr1.q transform_ppr2.q union_ppr.q 4. Iteration order of table properties for show create table is different. nullformatCTAS.q show_create_table_alter.q 5. Iteration order of show table properties is different unset_table_view_property.q Diffs (updated) - itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6fcc510 itests/util/src/main/java/org/apache/hadoop/hive/ql/hooks/ShowMapredStatsHook.java PRE-CREATION itests/util/src/main/java/org/apache/hadoop/hive/ql/hooks/VerifyNumReducersHook.java 4b2c184 ql/src/java/org/apache/hadoop/hive/ql/Driver.java 9040d9b ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 8ba2b49 ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 7c175aa ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 1f782db ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java dab8610 ql/src/test/queries/clientpositive/bucketizedhiveinputformat.q d2e12e8 ql/src/test/queries/clientpositive/groupby_sort_1.q 7401a9c ql/src/test/queries/clientpositive/groupby_sort_1_23.q PRE-CREATION ql/src/test/queries/clientpositive/groupby_sort_skew_1.q db0faa0 ql/src/test/queries/clientpositive/groupby_sort_skew_1_23.q PRE-CREATION ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 8428dbf ql/src/test/results/clientpositive/groupby_sort_1.q.out d684158 ql/src/test/results/clientpositive/groupby_sort_1_23.q.out PRE-CREATION ql/src/test/results/clientpositive/groupby_sort_skew_1.q.out 8d731b9 ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out PRE-CREATION ql/src/test/results/clientpositive/nullformatCTAS.q.out 85eaf37 ql/src/test/results/clientpositive/show_create_table_alter.q.out c4baa5b ql/src/test/results/clientpositive/show_tblproperties.q.out c60e710 ql/src/test/results/clientpositive/unset_table_view_property.q.out f4f1e91 Diff: https://reviews.apache.org/r/21748/diff/ Testing --- Thanks, Navis Ryu
[jira] [Updated] (HIVE-7095) Fix test fails for both hadoop-1 and hadoop-2
[ https://issues.apache.org/jira/browse/HIVE-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-7095: Attachment: HIVE-7095.2.patch.txt Fix test fails for both hadoop-1 and hadoop-2 - Key: HIVE-7095 URL: https://issues.apache.org/jira/browse/HIVE-7095 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Navis Attachments: HIVE-7095.1.patch.txt, HIVE-7095.2.patch.txt There are some try to fix test results for hadoop-2(HIVE-6838, HIVE-6979, etc.). But these remained problems for hadoop-1. Let's finish this here, not prolonging output ping-pongs between two versions. 1. Data size in explain result is different bucketizedhiveinputformat.q 2. forced bucketing(=2) does not make two files in hadoop-1. groupby_sort_1.q groupby_sort_skew_1.q 3. Column stats of srcpart, which has loaded a file rather than inserted, has COLUMN_STATS_ACCURATE=true, numRows=0, rawDataSize=0 for hadoop-2, which seemed wrong. sample8.q transform_ppr1.q transform_ppr2.q union_ppr.q 4. Iteration order of table properties for show create table is different. nullformatCTAS.q show_create_table_alter.q 5. Iteration order of show table properties is different unset_table_view_property.q -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7095) Fix test fails for both hadoop-1 and hadoop-2
[ https://issues.apache.org/jira/browse/HIVE-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-7095: Attachment: (was: HIVE-7095.2.patch.txt) Fix test fails for both hadoop-1 and hadoop-2 - Key: HIVE-7095 URL: https://issues.apache.org/jira/browse/HIVE-7095 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Navis Attachments: HIVE-7095.1.patch.txt, HIVE-7095.2.patch.txt There are some try to fix test results for hadoop-2(HIVE-6838, HIVE-6979, etc.). But these remained problems for hadoop-1. Let's finish this here, not prolonging output ping-pongs between two versions. 1. Data size in explain result is different bucketizedhiveinputformat.q 2. forced bucketing(=2) does not make two files in hadoop-1. groupby_sort_1.q groupby_sort_skew_1.q 3. Column stats of srcpart, which has loaded a file rather than inserted, has COLUMN_STATS_ACCURATE=true, numRows=0, rawDataSize=0 for hadoop-2, which seemed wrong. sample8.q transform_ppr1.q transform_ppr2.q union_ppr.q 4. Iteration order of table properties for show create table is different. nullformatCTAS.q show_create_table_alter.q 5. Iteration order of show table properties is different unset_table_view_property.q -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7075) JsonSerde raises NullPointerException when object key is not lower case
[
https://issues.apache.org/jira/browse/HIVE-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005498#comment-14005498
]
Navis commented on HIVE-7075:
-
Test fails seemed not related to this.
bq. Can a test case be added as well
Sure.
bq. Hive explicitly appears to make them lower case though
Yes, but this means all of the fields in json are regarded as lower case
literal. Highly not realistic, but for example,
{ empid : 123, empId : 234 }
'empId' will return 123, which was 234 before this patch.
JsonSerde raises NullPointerException when object key is not lower case
---
Key: HIVE-7075
URL: https://issues.apache.org/jira/browse/HIVE-7075
Project: Hive
Issue Type: Bug
Components: HCatalog, Serializers/Deserializers
Affects Versions: 0.12.0
Reporter: Yibing Shi
Attachments: HIVE-7075.1.patch.txt
We have noticed that the JsonSerde produces a NullPointerException if a JSON
object has a key value that is not lower case. For example. Assume we have
the file one.json:
{ empId : 123, name : John }
{ empId : 456, name : Jane }
hive CREATE TABLE emps (empId INT, name STRING)
ROW FORMAT SERDE org.apache.hive.hcatalog.data.JsonSerDe;
hive LOAD DATA LOCAL INPATH 'one.json' INTO TABLE emps;
hive SELECT * FROM emps;
Failed with exception java.io.IOException:java.lang.NullPointerException
Notice, it seems to work if the keys are lower case. Assume we have the file
'two.json':
{ empid : 123, name : John }
{ empid : 456, name : Jane }
hive DROP TABLE emps;
hive CREATE TABLE emps (empId INT, name STRING)
ROW FORMAT SERDE org.apache.hive.hcatalog.data.JsonSerDe;
hive LOAD DATA LOCAL INPATH 'two.json' INTO TABLE emps;
hive SELECT * FROM emps;
OK
123 John
456 Jane
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Is there a bug in hiveserver2 metastore caused hold huge objects
Vaibhav, thanks for reply. It's a remote metastore, stored in mysql. 2014-05-22 5:31 GMT+08:00 Vaibhav Gumashta vgumas...@hortonworks.com: Meng, are you running a remote metastore or an embedded one (which would be true if you added this config while starting HS2: -hiveconf hive.metastore.uris= )? Thanks, --Vaibhav On Wed, May 21, 2014 at 2:28 PM, Vaibhav Gumashta vgumas...@hortonworks.com wrote: Thanks a lot for the dump Meng! On Wed, May 21, 2014 at 1:41 AM, Meng QingPing mqingp...@gmail.com wrote: I post the heap bin file on http://jack-meng.qiniudn.com/heap.bin Thanks. 2014-05-21 15:41 GMT+08:00 Meng QingPing mqingp...@gmail.com: There are 18908 PM-s. I post it on http://jack-meng.qiniudn.com/hive-heap2.png Thanks. 2014-05-20 0:41 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: What is the count of PM-s that you are seeing when it crashes? From that screenshot I don't see what the problem is or I'm not reading it right. On Mon, May 19, 2014 at 1:17 AM, Meng QingPing mqingp...@gmail.com wrote: Thanks. I post it on http://jack-meng.qiniudn.com/hive-heap.png 2014-05-06 23:14 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com : I don't think the dev list supports attachment. Can you post to some image sharing service? On Tue, May 6, 2014 at 3:20 AM, Meng QingPing mqingp...@gmail.com wrote: Weird, the picture attached in sent mail. Anyway send again. 2014-05-05 12:17 GMT+08:00 Chandra Reddy chandu...@gmail.com : seems you have missed attachment. -Chandra On Sun, May 4, 2014 at 6:57 PM, Meng QingPing mqingp...@gmail.com wrote: I run hiveserver2 with metastore in mysql. The hiveserver2 OOM and the heap dump show huge objects hold by org.datanucleus.api.jdo. JDOPersistenceManagerFactory as attached. It seems not release org.datanucleus.api.jdo.JDOPersistenceManager. Hive version is 0.13. Thanks, Jack -- Thanks, -Chandra. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping
[jira] [Created] (HIVE-7110) TestHCatPartitionPublish test failure: No FileSystem or scheme: pfile
David Chen created HIVE-7110:
Summary: TestHCatPartitionPublish test failure: No FileSystem or
scheme: pfile
Key: HIVE-7110
URL: https://issues.apache.org/jira/browse/HIVE-7110
Project: Hive
Issue Type: Bug
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
I got the following TestHCatPartitionPublish test failure when running all unit
tests against Hadoop 1. This also appears when testing against Hadoop 2.
{code}
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.06 sec
FAILURE! - in org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish
testPartitionPublish(org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish)
Time elapsed: 1.361 sec ERROR!
org.apache.hive.hcatalog.common.HCatException:
org.apache.hive.hcatalog.common.HCatException : 2001 : Error setting output
information. Cause : java.io.IOException: No FileSystem for scheme: pfile
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1443)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at
org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:212)
at
org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:70)
at
org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish.runMRCreateFail(TestHCatPartitionPublish.java:191)
at
org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish.testPartitionPublish(TestHCatPartitionPublish.java:155)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-7111) Extend join transitivity PPD to non-column expressions
Navis created HIVE-7111: --- Summary: Extend join transitivity PPD to non-column expressions Key: HIVE-7111 URL: https://issues.apache.org/jira/browse/HIVE-7111 Project: Hive Issue Type: Task Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Join transitive in PPD only supports column expressions, but it's possible to extend this to generic expressions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-4867) Deduplicate columns appearing in both the key list and value list of ReduceSinkOperator
[
https://issues.apache.org/jira/browse/HIVE-4867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005560#comment-14005560
]
Navis commented on HIVE-4867:
-
Waiting on HIVE-7087 to be committed first.
Deduplicate columns appearing in both the key list and value list of
ReduceSinkOperator
---
Key: HIVE-4867
URL: https://issues.apache.org/jira/browse/HIVE-4867
Project: Hive
Issue Type: Improvement
Reporter: Yin Huai
Assignee: Navis
Attachments: HIVE-4867.1.patch.txt, source_only.txt
A ReduceSinkOperator emits data in the format of keys and values. Right now,
a column may appear in both the key list and value list, which result in
unnecessary overhead for shuffling.
Example:
We have a query shown below ...
{code:sql}
explain select ss_ticket_number from store_sales cluster by ss_ticket_number;
{\code}
The plan is ...
{code}
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Alias - Map Operator Tree:
store_sales
TableScan
alias: store_sales
Select Operator
expressions:
expr: ss_ticket_number
type: int
outputColumnNames: _col0
Reduce Output Operator
key expressions:
expr: _col0
type: int
sort order: +
Map-reduce partition columns:
expr: _col0
type: int
tag: -1
value expressions:
expr: _col0
type: int
Reduce Operator Tree:
Extract
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Stage: Stage-0
Fetch Operator
limit: -1
{\code}
The column 'ss_ticket_number' is in both the key list and value list of the
ReduceSinkOperator. The type of ss_ticket_number is int. For this case,
BinarySortableSerDe will introduce 1 byte more for every int in the key.
LazyBinarySerDe will also introduce overhead when recording the length of a
int. For every int, 10 bytes should be a rough estimation of the size of data
emitted from the Map phase.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-4561) Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0.0000 ,if all the column values larger than 0.0 (or if all column values smaller than 0.0)
[
https://issues.apache.org/jira/browse/HIVE-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005569#comment-14005569
]
Hive QA commented on HIVE-4561:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12645973/HIVE-4561.4.patch.txt
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 5451 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadata_only_queries
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.conf.TestHiveConf.testConfProperties
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/259/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/259/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-259/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12645973
Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0. ,if all the
column values larger than 0.0 (or if all column values smaller than 0.0)
Key: HIVE-4561
URL: https://issues.apache.org/jira/browse/HIVE-4561
Project: Hive
Issue Type: Bug
Components: Statistics
Affects Versions: 0.12.0
Reporter: caofangkun
Assignee: caofangkun
Attachments: HIVE-4561.1.patch, HIVE-4561.2.patch, HIVE-4561.3.patch,
HIVE-4561.4.patch.txt
if all column values larger than 0.0 DOUBLE_LOW_VALUE always will be 0.0
or if all column values less than 0.0, DOUBLE_HIGH_VALUE will always be
hive (default) create table src_test (price double);
hive (default) load data local inpath './test.txt' into table src_test;
hive (default) select * from src_test;
OK
1.0
2.0
3.0
Time taken: 0.313 seconds, Fetched: 3 row(s)
hive (default) analyze table src_test compute statistics for columns price;
mysql select * from TAB_COL_STATS \G;
CS_ID: 16
DB_NAME: default
TABLE_NAME: src_test
COLUMN_NAME: price
COLUMN_TYPE: double
TBL_ID: 2586
LONG_LOW_VALUE: 0
LONG_HIGH_VALUE: 0
DOUBLE_LOW_VALUE: 0. # Wrong Result ! Expected is 1.
DOUBLE_HIGH_VALUE: 3.
BIG_DECIMAL_LOW_VALUE: NULL
BIG_DECIMAL_HIGH_VALUE: NULL
NUM_NULLS: 0
NUM_DISTINCTS: 1
AVG_COL_LEN: 0.
MAX_COL_LEN: 0
NUM_TRUES: 0
NUM_FALSES: 0
LAST_ANALYZED: 1368596151
2 rows in set (0.00 sec)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-7112) Tez processor swallows errors
Gunther Hagleitner created HIVE-7112: Summary: Tez processor swallows errors Key: HIVE-7112 URL: https://issues.apache.org/jira/browse/HIVE-7112 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner If a OOM or other occurs during initializing a TezProcessor the original stack trace/throwable is swallowed and replaced by a misleading NPE caused by trying to close an uninitialized processor. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7112) Tez processor swallows errors
[ https://issues.apache.org/jira/browse/HIVE-7112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-7112: - Attachment: HIVE-7112.1.patch Tez processor swallows errors - Key: HIVE-7112 URL: https://issues.apache.org/jira/browse/HIVE-7112 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7112.1.patch If a OOM or other occurs during initializing a TezProcessor the original stack trace/throwable is swallowed and replaced by a misleading NPE caused by trying to close an uninitialized processor. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7112) Tez processor swallows errors
[ https://issues.apache.org/jira/browse/HIVE-7112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-7112: - Status: Patch Available (was: Open) Tez processor swallows errors - Key: HIVE-7112 URL: https://issues.apache.org/jira/browse/HIVE-7112 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7112.1.patch If a OOM or other occurs during initializing a TezProcessor the original stack trace/throwable is swallowed and replaced by a misleading NPE caused by trying to close an uninitialized processor. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-7113) Support fully managed partitioned table, which does not allow partitions to have different properties with table
Navis created HIVE-7113: --- Summary: Support fully managed partitioned table, which does not allow partitions to have different properties with table Key: HIVE-7113 URL: https://issues.apache.org/jira/browse/HIVE-7113 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Mostly, partitioned table has the same storage description with mother table just except data location. If partitions can be forced not to have different schema with table by configuration, the time for retrieving partition objects from metastore and for creating serde for each partition could be nullified. -- This message was sent by Atlassian JIRA (v6.2#6252)
