[jira] [Commented] (HIVE-7121) Use murmur hash to distribute HiveKey
[
https://issues.apache.org/jira/browse/HIVE-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008306#comment-14008306
]
Hive QA commented on HIVE-7121:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12646646/HIVE-7121.1.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 5462 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_decimal1
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_bucketed_table
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_quotedid_smb
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_truncate_column_buckets
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testDropTable
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testListPartitions
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testNameMethods
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/288/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/288/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-288/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12646646
Use murmur hash to distribute HiveKey
-
Key: HIVE-7121
URL: https://issues.apache.org/jira/browse/HIVE-7121
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Gopal V
Assignee: Gopal V
Attachments: HIVE-7121.1.patch, HIVE-7121.WIP.patch
The current hashCode implementation produces poor parallelism when dealing
with single integers or doubles.
And for partitioned inserts into a 1 bucket table, there is a significant
hotspot on Reducer #31.
Removing the magic number 31 and using a more normal hash algorithm would
help fix these hotspots.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6756) alter table set fileformat should set serde too
[ https://issues.apache.org/jira/browse/HIVE-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-6756: --- Status: Patch Available (was: Open) alter table set fileformat should set serde too --- Key: HIVE-6756 URL: https://issues.apache.org/jira/browse/HIVE-6756 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Owen O'Malley Assignee: Chinna Rao Lalam Attachments: HIVE-6756.1.patch, HIVE-6756.2.patch, HIVE-6756.3.patch, HIVE-6756.patch Currently doing alter table set fileformat doesn't change the serde. This is unexpected by customers because the serdes are largely file format specific. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6756) alter table set fileformat should set serde too
[ https://issues.apache.org/jira/browse/HIVE-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-6756: --- Attachment: HIVE-6756.3.patch alter table set fileformat should set serde too --- Key: HIVE-6756 URL: https://issues.apache.org/jira/browse/HIVE-6756 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Owen O'Malley Assignee: Chinna Rao Lalam Attachments: HIVE-6756.1.patch, HIVE-6756.2.patch, HIVE-6756.3.patch, HIVE-6756.patch Currently doing alter table set fileformat doesn't change the serde. This is unexpected by customers because the serdes are largely file format specific. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6756) alter table set fileformat should set serde too
[ https://issues.apache.org/jira/browse/HIVE-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008314#comment-14008314 ] Chinna Rao Lalam commented on HIVE-6756: Hi Ashutosh, Thanks for reviewing the patch. Reworked the patch. alter table set fileformat should set serde too --- Key: HIVE-6756 URL: https://issues.apache.org/jira/browse/HIVE-6756 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Owen O'Malley Assignee: Chinna Rao Lalam Attachments: HIVE-6756.1.patch, HIVE-6756.2.patch, HIVE-6756.3.patch, HIVE-6756.patch Currently doing alter table set fileformat doesn't change the serde. This is unexpected by customers because the serdes are largely file format specific. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6928) Beeline should not chop off describe extended results by default
[ https://issues.apache.org/jira/browse/HIVE-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-6928: --- Attachment: HIVE-6928.1.patch Beeline should not chop off describe extended results by default -- Key: HIVE-6928 URL: https://issues.apache.org/jira/browse/HIVE-6928 Project: Hive Issue Type: Bug Components: CLI Reporter: Szehon Ho Assignee: Chinna Rao Lalam Attachments: HIVE-6928.1.patch, HIVE-6928.patch By default, beeline truncates long results based on the console width like: +-+--+ | col_name | | +-+--+ | pat_id | string | | score | float | | acutes | float | | | | | Detailed Table Information | Table(tableName:refills, dbName:default, owner:hdadmin, createTime:1393882396, lastAccessTime:0, retention:0, sd:Sto | +-+--+ 5 rows selected (0.4 seconds) This can be changed by !outputformat, but the default should behave better to give a better experience to the first-time beeline user. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6928) Beeline should not chop off describe extended results by default
[ https://issues.apache.org/jira/browse/HIVE-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008317#comment-14008317 ] Chinna Rao Lalam commented on HIVE-6928: Hi Szehon Ho, Thanks for reviewing the patch. Reworked the patch. Beeline should not chop off describe extended results by default -- Key: HIVE-6928 URL: https://issues.apache.org/jira/browse/HIVE-6928 Project: Hive Issue Type: Bug Components: CLI Reporter: Szehon Ho Assignee: Chinna Rao Lalam Attachments: HIVE-6928.1.patch, HIVE-6928.patch By default, beeline truncates long results based on the console width like: +-+--+ | col_name | | +-+--+ | pat_id | string | | score | float | | acutes | float | | | | | Detailed Table Information | Table(tableName:refills, dbName:default, owner:hdadmin, createTime:1393882396, lastAccessTime:0, retention:0, sd:Sto | +-+--+ 5 rows selected (0.4 seconds) This can be changed by !outputformat, but the default should behave better to give a better experience to the first-time beeline user. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-4561) Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0.0000 ,if all the column values larger than 0.0 (or if all column values smaller than 0.0)
[
https://issues.apache.org/jira/browse/HIVE-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008354#comment-14008354
]
Hive QA commented on HIVE-4561:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12646662/HIVE-4561.5.patch.txt
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 5462 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_tbllvl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_compute_stats_empty_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadata_only_queries_with_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_decimal1
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/291/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/291/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-291/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12646662
Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0. ,if all the
column values larger than 0.0 (or if all column values smaller than 0.0)
Key: HIVE-4561
URL: https://issues.apache.org/jira/browse/HIVE-4561
Project: Hive
Issue Type: Bug
Components: Statistics
Affects Versions: 0.12.0
Reporter: caofangkun
Assignee: Navis
Attachments: HIVE-4561.1.patch, HIVE-4561.2.patch, HIVE-4561.3.patch,
HIVE-4561.4.patch.txt, HIVE-4561.5.patch.txt
if all column values larger than 0.0 DOUBLE_LOW_VALUE always will be 0.0
or if all column values less than 0.0, DOUBLE_HIGH_VALUE will always be
hive (default) create table src_test (price double);
hive (default) load data local inpath './test.txt' into table src_test;
hive (default) select * from src_test;
OK
1.0
2.0
3.0
Time taken: 0.313 seconds, Fetched: 3 row(s)
hive (default) analyze table src_test compute statistics for columns price;
mysql select * from TAB_COL_STATS \G;
CS_ID: 16
DB_NAME: default
TABLE_NAME: src_test
COLUMN_NAME: price
COLUMN_TYPE: double
TBL_ID: 2586
LONG_LOW_VALUE: 0
LONG_HIGH_VALUE: 0
DOUBLE_LOW_VALUE: 0. # Wrong Result ! Expected is 1.
DOUBLE_HIGH_VALUE: 3.
BIG_DECIMAL_LOW_VALUE: NULL
BIG_DECIMAL_HIGH_VALUE: NULL
NUM_NULLS: 0
NUM_DISTINCTS: 1
AVG_COL_LEN: 0.
MAX_COL_LEN: 0
NUM_TRUES: 0
NUM_FALSES: 0
LAST_ANALYZED: 1368596151
2 rows in set (0.00 sec)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008364#comment-14008364 ] Ted Xu commented on HIVE-5771: -- Hi [~ashutoshc], I may need more time to fix some bugs. Constant propagation optimizer for Hive --- Key: HIVE-5771 URL: https://issues.apache.org/jira/browse/HIVE-5771 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Ted Xu Assignee: Ted Xu Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.9.patch, HIVE-5771.patch, HIVE-5771.patch.javaonly Currently there is no constant folding/propagation optimizer, all expressions are evaluated at runtime. HIVE-2470 did a great job on evaluating constants on UDF initializing phase, however, it is still a runtime evaluation and it doesn't propagate constants from a subquery to outside. It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7121) Use murmur hash to distribute HiveKey
[ https://issues.apache.org/jira/browse/HIVE-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008377#comment-14008377 ] Edward Capriolo commented on HIVE-7121: --- Does this effect bucketed tables? I think it does and then we can not just change the hash code because that would break assumptions of what is in the bucket. IE i create a bucket in hive 12, and in hive 13 different data would be in the bucket. I think this is why: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_num_buckets org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_quotedid_smb These tests are failing. If this is the case we need a way or recording the hashcode in the metadata for the table. Use murmur hash to distribute HiveKey - Key: HIVE-7121 URL: https://issues.apache.org/jira/browse/HIVE-7121 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Gopal V Assignee: Gopal V Attachments: HIVE-7121.1.patch, HIVE-7121.WIP.patch The current hashCode implementation produces poor parallelism when dealing with single integers or doubles. And for partitioned inserts into a 1 bucket table, there is a significant hotspot on Reducer #31. Removing the magic number 31 and using a more normal hash algorithm would help fix these hotspots. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6756) alter table set fileformat should set serde too
[ https://issues.apache.org/jira/browse/HIVE-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008378#comment-14008378 ] Ashutosh Chauhan commented on HIVE-6756: +1 alter table set fileformat should set serde too --- Key: HIVE-6756 URL: https://issues.apache.org/jira/browse/HIVE-6756 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Owen O'Malley Assignee: Chinna Rao Lalam Attachments: HIVE-6756.1.patch, HIVE-6756.2.patch, HIVE-6756.3.patch, HIVE-6756.patch Currently doing alter table set fileformat doesn't change the serde. This is unexpected by customers because the serdes are largely file format specific. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7085) TestOrcHCatPigStorer.testWriteDecimal tests are failing on trunk
[
https://issues.apache.org/jira/browse/HIVE-7085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008379#comment-14008379
]
Ashutosh Chauhan commented on HIVE-7085:
Stack trace for TestHCatLoader.testReadDataPrimitiveTypes
{code}
java.lang.AssertionError: rowNum=0 colNum=0 Reference data=true actual=1;
types=(class java.lang.Boolean,class java.lang.Integer)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.hive.hcatalog.pig.TestHCatLoader$AllTypesTable.testReadDataPrimitiveTypes(TestHCatLoader.java:614)
at
org.apache.hive.hcatalog.pig.TestHCatLoader$AllTypesTable.access$200(TestHCatLoader.java:540)
at
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes(TestHCatLoader.java:247)
{code}
TestOrcHCatPigStorer.testWriteDecimal tests are failing on trunk
Key: HIVE-7085
URL: https://issues.apache.org/jira/browse/HIVE-7085
Project: Hive
Issue Type: Test
Components: HCatalog
Affects Versions: 0.14.0
Reporter: Ashutosh Chauhan
TestOrcHCatPigStorer.testWriteDecimal,
TestOrcHCatPigStorer.testWriteDecimalX,
TestOrcHCatPigStorer.testWriteDecimalXY
are failing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-4561) Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0.0000 ,if all the column values larger than 0.0 (or if all column values smaller than 0.0)
[ https://issues.apache.org/jira/browse/HIVE-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4561: --- Affects Version/s: 0.13.0 Status: Open (was: Patch Available) Failures need to be looked at Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0. ,if all the column values larger than 0.0 (or if all column values smaller than 0.0) Key: HIVE-4561 URL: https://issues.apache.org/jira/browse/HIVE-4561 Project: Hive Issue Type: Bug Components: Statistics Affects Versions: 0.13.0, 0.12.0 Reporter: caofangkun Assignee: Navis Attachments: HIVE-4561.1.patch, HIVE-4561.2.patch, HIVE-4561.3.patch, HIVE-4561.4.patch.txt, HIVE-4561.5.patch.txt if all column values larger than 0.0 DOUBLE_LOW_VALUE always will be 0.0 or if all column values less than 0.0, DOUBLE_HIGH_VALUE will always be hive (default) create table src_test (price double); hive (default) load data local inpath './test.txt' into table src_test; hive (default) select * from src_test; OK 1.0 2.0 3.0 Time taken: 0.313 seconds, Fetched: 3 row(s) hive (default) analyze table src_test compute statistics for columns price; mysql select * from TAB_COL_STATS \G; CS_ID: 16 DB_NAME: default TABLE_NAME: src_test COLUMN_NAME: price COLUMN_TYPE: double TBL_ID: 2586 LONG_LOW_VALUE: 0 LONG_HIGH_VALUE: 0 DOUBLE_LOW_VALUE: 0. # Wrong Result ! Expected is 1. DOUBLE_HIGH_VALUE: 3. BIG_DECIMAL_LOW_VALUE: NULL BIG_DECIMAL_HIGH_VALUE: NULL NUM_NULLS: 0 NUM_DISTINCTS: 1 AVG_COL_LEN: 0. MAX_COL_LEN: 0 NUM_TRUES: 0 NUM_FALSES: 0 LAST_ANALYZED: 1368596151 2 rows in set (0.00 sec) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive
[ https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008381#comment-14008381 ] Ashutosh Chauhan commented on HIVE-5771: No worries. Let me know, if I can be of any help. Constant propagation optimizer for Hive --- Key: HIVE-5771 URL: https://issues.apache.org/jira/browse/HIVE-5771 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Ted Xu Assignee: Ted Xu Attachments: HIVE-5771.1.patch, HIVE-5771.10.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, HIVE-5771.8.patch, HIVE-5771.9.patch, HIVE-5771.patch, HIVE-5771.patch.javaonly Currently there is no constant folding/propagation optimizer, all expressions are evaluated at runtime. HIVE-2470 did a great job on evaluating constants on UDF initializing phase, however, it is still a runtime evaluation and it doesn't propagate constants from a subquery to outside. It may reduce I/O and accelerate process if we introduce such an optimizer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6910) Invalid column access info for partitioned table
[ https://issues.apache.org/jira/browse/HIVE-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6910: --- Status: Open (was: Patch Available) Failures need to be looked at Invalid column access info for partitioned table Key: HIVE-6910 URL: https://issues.apache.org/jira/browse/HIVE-6910 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.13.0, 0.12.0, 0.11.0 Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-6910.1.patch.txt, HIVE-6910.2.patch.txt, HIVE-6910.3.patch.txt, HIVE-6910.4.patch.txt, HIVE-6910.5.patch.txt, HIVE-6910.6.patch.txt From http://www.mail-archive.com/user@hive.apache.org/msg11324.html neededColumnIDs in TS is only for non-partition columns. But ColumnAccessAnalyzer is calculating it on all columns. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5961) Add explain authorize for checking privileges
[
https://issues.apache.org/jira/browse/HIVE-5961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008401#comment-14008401
]
Hive QA commented on HIVE-5961:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12646664/HIVE-5961.6.patch.txt
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 5463 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_decimal1
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/292/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/292/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-292/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12646664
Add explain authorize for checking privileges
-
Key: HIVE-5961
URL: https://issues.apache.org/jira/browse/HIVE-5961
Project: Hive
Issue Type: Improvement
Components: Authorization
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-5961.1.patch.txt, HIVE-5961.2.patch.txt,
HIVE-5961.3.patch.txt, HIVE-5961.4.patch.txt, HIVE-5961.5.patch.txt,
HIVE-5961.6.patch.txt
For easy checking of need privileges for a query,
{noformat}
explain authorize select * from src join srcpart
INPUTS:
default@srcpart
default@srcpart@ds=2008-04-08/hr=11
default@srcpart@ds=2008-04-08/hr=12
default@srcpart@ds=2008-04-09/hr=11
default@srcpart@ds=2008-04-09/hr=12
default@src
OUTPUTS:
file:/home/navis/apache/oss-hive/itests/qtest/target/tmp/localscratchdir/hive_2013-12-04_21-57-53_748_5323811717799107868-1/-mr-1
CURRENT_USER:
hive_test_user
OPERATION:
QUERY
AUTHORIZATION_FAILURES:
No privilege 'Select' found for inputs { database:default, table:srcpart,
columnName:key}
No privilege 'Select' found for inputs { database:default, table:src,
columnName:key}
No privilege 'Select' found for inputs { database:default, table:src,
columnName:key}
{noformat}
Hopefully good for debugging of authorization, which is in progress on
HIVE-5837.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5961) Add explain authorize for checking privileges
[
https://issues.apache.org/jira/browse/HIVE-5961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-5961:
---
Resolution: Fixed
Fix Version/s: 0.14.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks, Navis!
Add explain authorize for checking privileges
-
Key: HIVE-5961
URL: https://issues.apache.org/jira/browse/HIVE-5961
Project: Hive
Issue Type: Improvement
Components: Authorization
Reporter: Navis
Assignee: Navis
Priority: Trivial
Fix For: 0.14.0
Attachments: HIVE-5961.1.patch.txt, HIVE-5961.2.patch.txt,
HIVE-5961.3.patch.txt, HIVE-5961.4.patch.txt, HIVE-5961.5.patch.txt,
HIVE-5961.6.patch.txt
For easy checking of need privileges for a query,
{noformat}
explain authorize select * from src join srcpart
INPUTS:
default@srcpart
default@srcpart@ds=2008-04-08/hr=11
default@srcpart@ds=2008-04-08/hr=12
default@srcpart@ds=2008-04-09/hr=11
default@srcpart@ds=2008-04-09/hr=12
default@src
OUTPUTS:
file:/home/navis/apache/oss-hive/itests/qtest/target/tmp/localscratchdir/hive_2013-12-04_21-57-53_748_5323811717799107868-1/-mr-1
CURRENT_USER:
hive_test_user
OPERATION:
QUERY
AUTHORIZATION_FAILURES:
No privilege 'Select' found for inputs { database:default, table:srcpart,
columnName:key}
No privilege 'Select' found for inputs { database:default, table:src,
columnName:key}
No privilege 'Select' found for inputs { database:default, table:src,
columnName:key}
{noformat}
Hopefully good for debugging of authorization, which is in progress on
HIVE-5837.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7048) CompositeKeyHBaseFactory should not use FamilyFilter
[
https://issues.apache.org/jira/browse/HIVE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008448#comment-14008448
]
Hive QA commented on HIVE-7048:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12646678/HIVE-7048.4.patch.txt
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 5538 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_decimal1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_load_dyn_part1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/294/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/294/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-294/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12646678
CompositeKeyHBaseFactory should not use FamilyFilter
Key: HIVE-7048
URL: https://issues.apache.org/jira/browse/HIVE-7048
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Affects Versions: 0.14.0
Reporter: Swarnim Kulkarni
Assignee: Swarnim Kulkarni
Priority: Blocker
Attachments: HIVE-7048.1.patch.txt, HIVE-7048.2.patch.txt,
HIVE-7048.3.patch.txt, HIVE-7048.4.patch.txt
HIVE-6411 introduced a more generic way to provide composite key
implementations via custom factory implementations. However it seems like the
CompositeHBaseKeyFactory implementation uses a FamilyFilter for row key scans
which doesn't seem appropriate. This should be investigated further and if
possible replaced with a RowRangeScanFilter.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-4790) MapredLocalTask task does not make virtual columns
[
https://issues.apache.org/jira/browse/HIVE-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-4790:
Attachment: HIVE-4790.9.patch.txt
MapredLocalTask task does not make virtual columns
--
Key: HIVE-4790
URL: https://issues.apache.org/jira/browse/HIVE-4790
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: D11511.3.patch, D11511.4.patch, HIVE-4790.5.patch.txt,
HIVE-4790.6.patch.txt, HIVE-4790.7.patch.txt, HIVE-4790.8.patch.txt,
HIVE-4790.9.patch.txt, HIVE-4790.D11511.1.patch, HIVE-4790.D11511.2.patch
From mailing list,
http://www.mail-archive.com/user@hive.apache.org/msg08264.html
{noformat}
SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON
b.rownumber = a.number;
fails with this error:
SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON b.rownumber =
a.number;
Automatically selecting local only mode for query
Total MapReduce jobs = 1
setting HADOOP_USER_NAMEpmarron
13/06/25 10:52:56 WARN conf.HiveConf: DEPRECATED: Configuration property
hive.metastore.local no longer has any effect. Make sure to provide a valid
value for hive.metastore.uris if you are connecting to a remote metastore.
Execution log at: /tmp/pmarron/.log
2013-06-25 10:52:56 Starting to launch local task to process map join;
maximum memory = 932118528
java.lang.RuntimeException: cannot find field block__offset__inside__file
from [0:rownumber, 1:offset]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:366)
at
org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldRef(LazySimpleStructObjectInspector.java:168)
at
org.apache.hadoop.hive.serde2.objectinspector.DelegatedStructObjectInspector.getStructFieldRef(DelegatedStructObjectInspector.java:74)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
at
org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:68)
at
org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:222)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:407)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:186)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at
org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:394)
at
org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:277)
at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:676)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Execution failed with exit status: 2
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7122) Storage format for create like table
[
https://issues.apache.org/jira/browse/HIVE-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008512#comment-14008512
]
Hive QA commented on HIVE-7122:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12646680/HIVE-7122.patch
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 5463 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_decimal1
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_stats_counter
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.common.metrics.TestMetrics.testScopeConcurrency
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/295/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/295/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-295/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12646680
Storage format for create like table
Key: HIVE-7122
URL: https://issues.apache.org/jira/browse/HIVE-7122
Project: Hive
Issue Type: New Feature
Components: Query Processor
Reporter: Vasanth kumar RJ
Assignee: Vasanth kumar RJ
Fix For: 0.14.0
Attachments: HIVE-7122.patch
Using create like table user can specify the table storage format.
Example:
create table table1 like table2 stored as ORC;
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6910) Invalid column access info for partitioned table
[ https://issues.apache.org/jira/browse/HIVE-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-6910: Attachment: HIVE-6910.7.patch.txt Invalid column access info for partitioned table Key: HIVE-6910 URL: https://issues.apache.org/jira/browse/HIVE-6910 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-6910.1.patch.txt, HIVE-6910.2.patch.txt, HIVE-6910.3.patch.txt, HIVE-6910.4.patch.txt, HIVE-6910.5.patch.txt, HIVE-6910.6.patch.txt, HIVE-6910.7.patch.txt From http://www.mail-archive.com/user@hive.apache.org/msg11324.html neededColumnIDs in TS is only for non-partition columns. But ColumnAccessAnalyzer is calculating it on all columns. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6910) Invalid column access info for partitioned table
[ https://issues.apache.org/jira/browse/HIVE-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-6910: Attachment: HIVE-6910.7.patch.txt Invalid column access info for partitioned table Key: HIVE-6910 URL: https://issues.apache.org/jira/browse/HIVE-6910 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-6910.1.patch.txt, HIVE-6910.2.patch.txt, HIVE-6910.3.patch.txt, HIVE-6910.4.patch.txt, HIVE-6910.5.patch.txt, HIVE-6910.6.patch.txt, HIVE-6910.7.patch.txt From http://www.mail-archive.com/user@hive.apache.org/msg11324.html neededColumnIDs in TS is only for non-partition columns. But ColumnAccessAnalyzer is calculating it on all columns. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6910) Invalid column access info for partitioned table
[ https://issues.apache.org/jira/browse/HIVE-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-6910: Attachment: (was: HIVE-6910.7.patch.txt) Invalid column access info for partitioned table Key: HIVE-6910 URL: https://issues.apache.org/jira/browse/HIVE-6910 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-6910.1.patch.txt, HIVE-6910.2.patch.txt, HIVE-6910.3.patch.txt, HIVE-6910.4.patch.txt, HIVE-6910.5.patch.txt, HIVE-6910.6.patch.txt, HIVE-6910.7.patch.txt From http://www.mail-archive.com/user@hive.apache.org/msg11324.html neededColumnIDs in TS is only for non-partition columns. But ColumnAccessAnalyzer is calculating it on all columns. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6910) Invalid column access info for partitioned table
[ https://issues.apache.org/jira/browse/HIVE-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-6910: Status: Patch Available (was: Open) Removed transient marks for neededColumns and neededColumnIDs. Invalid column access info for partitioned table Key: HIVE-6910 URL: https://issues.apache.org/jira/browse/HIVE-6910 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.13.0, 0.12.0, 0.11.0 Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-6910.1.patch.txt, HIVE-6910.2.patch.txt, HIVE-6910.3.patch.txt, HIVE-6910.4.patch.txt, HIVE-6910.5.patch.txt, HIVE-6910.6.patch.txt, HIVE-6910.7.patch.txt From http://www.mail-archive.com/user@hive.apache.org/msg11324.html neededColumnIDs in TS is only for non-partition columns. But ColumnAccessAnalyzer is calculating it on all columns. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-4561) Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0.0000 ,if all the column values larger than 0.0 (or if all column values smaller than 0.0)
[ https://issues.apache.org/jira/browse/HIVE-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-4561: Attachment: HIVE-4561.6.patch.txt Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0. ,if all the column values larger than 0.0 (or if all column values smaller than 0.0) Key: HIVE-4561 URL: https://issues.apache.org/jira/browse/HIVE-4561 Project: Hive Issue Type: Bug Components: Statistics Affects Versions: 0.12.0, 0.13.0 Reporter: caofangkun Assignee: Navis Attachments: HIVE-4561.1.patch, HIVE-4561.2.patch, HIVE-4561.3.patch, HIVE-4561.4.patch.txt, HIVE-4561.5.patch.txt, HIVE-4561.6.patch.txt if all column values larger than 0.0 DOUBLE_LOW_VALUE always will be 0.0 or if all column values less than 0.0, DOUBLE_HIGH_VALUE will always be hive (default) create table src_test (price double); hive (default) load data local inpath './test.txt' into table src_test; hive (default) select * from src_test; OK 1.0 2.0 3.0 Time taken: 0.313 seconds, Fetched: 3 row(s) hive (default) analyze table src_test compute statistics for columns price; mysql select * from TAB_COL_STATS \G; CS_ID: 16 DB_NAME: default TABLE_NAME: src_test COLUMN_NAME: price COLUMN_TYPE: double TBL_ID: 2586 LONG_LOW_VALUE: 0 LONG_HIGH_VALUE: 0 DOUBLE_LOW_VALUE: 0. # Wrong Result ! Expected is 1. DOUBLE_HIGH_VALUE: 3. BIG_DECIMAL_LOW_VALUE: NULL BIG_DECIMAL_HIGH_VALUE: NULL NUM_NULLS: 0 NUM_DISTINCTS: 1 AVG_COL_LEN: 0. MAX_COL_LEN: 0 NUM_TRUES: 0 NUM_FALSES: 0 LAST_ANALYZED: 1368596151 2 rows in set (0.00 sec) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-4561) Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0.0000 ,if all the column values larger than 0.0 (or if all column values smaller than 0.0)
[ https://issues.apache.org/jira/browse/HIVE-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-4561: Status: Patch Available (was: Open) Fixed result of metadata_only_queries_with_filters.q, but cannot reproduce other failures. Rerunning test. Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0. ,if all the column values larger than 0.0 (or if all column values smaller than 0.0) Key: HIVE-4561 URL: https://issues.apache.org/jira/browse/HIVE-4561 Project: Hive Issue Type: Bug Components: Statistics Affects Versions: 0.13.0, 0.12.0 Reporter: caofangkun Assignee: Navis Attachments: HIVE-4561.1.patch, HIVE-4561.2.patch, HIVE-4561.3.patch, HIVE-4561.4.patch.txt, HIVE-4561.5.patch.txt, HIVE-4561.6.patch.txt if all column values larger than 0.0 DOUBLE_LOW_VALUE always will be 0.0 or if all column values less than 0.0, DOUBLE_HIGH_VALUE will always be hive (default) create table src_test (price double); hive (default) load data local inpath './test.txt' into table src_test; hive (default) select * from src_test; OK 1.0 2.0 3.0 Time taken: 0.313 seconds, Fetched: 3 row(s) hive (default) analyze table src_test compute statistics for columns price; mysql select * from TAB_COL_STATS \G; CS_ID: 16 DB_NAME: default TABLE_NAME: src_test COLUMN_NAME: price COLUMN_TYPE: double TBL_ID: 2586 LONG_LOW_VALUE: 0 LONG_HIGH_VALUE: 0 DOUBLE_LOW_VALUE: 0. # Wrong Result ! Expected is 1. DOUBLE_HIGH_VALUE: 3. BIG_DECIMAL_LOW_VALUE: NULL BIG_DECIMAL_HIGH_VALUE: NULL NUM_NULLS: 0 NUM_DISTINCTS: 1 AVG_COL_LEN: 0. MAX_COL_LEN: 0 NUM_TRUES: 0 NUM_FALSES: 0 LAST_ANALYZED: 1368596151 2 rows in set (0.00 sec) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-4561) Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0.0000 ,if all the column values larger than 0.0 (or if all column values smaller than 0.0)
[
https://issues.apache.org/jira/browse/HIVE-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008534#comment-14008534
]
Ashutosh Chauhan commented on HIVE-4561:
Both testCliDriver_columnstats_tbllvl.q
testCliDriver_compute_stats_empty_table.q failed for me on latest patch with
following exception. which is same as Hive QA run reported on
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/291/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_columnstats_tbllvl/
{code}
java.lang.Exception: java.lang.RuntimeException: Hive Runtime Error while
closing operators
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing
operators
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:260)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1141)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
... 9 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getInt(PrimitiveObjectInspectorUtils.java:598)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats$GenericUDAFLongStatsEvaluator.iterate(GenericUDAFComputeStats.java:468)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:183)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1132)
{code}
Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0. ,if all the
column values larger than 0.0 (or if all column values smaller than 0.0)
Key: HIVE-4561
URL: https://issues.apache.org/jira/browse/HIVE-4561
Project: Hive
Issue Type: Bug
Components: Statistics
Affects Versions: 0.12.0, 0.13.0
Reporter: caofangkun
Assignee: Navis
Attachments: HIVE-4561.1.patch, HIVE-4561.2.patch, HIVE-4561.3.patch,
HIVE-4561.4.patch.txt, HIVE-4561.5.patch.txt, HIVE-4561.6.patch.txt
if all column values larger than 0.0 DOUBLE_LOW_VALUE always will be 0.0
or if all column values less than 0.0, DOUBLE_HIGH_VALUE will always be
hive (default) create table src_test (price double);
hive (default) load data local inpath './test.txt' into table src_test;
hive (default) select * from src_test;
OK
1.0
2.0
3.0
Time taken: 0.313 seconds, Fetched: 3 row(s)
hive (default) analyze table src_test compute statistics for columns price;
mysql select * from TAB_COL_STATS \G;
CS_ID: 16
DB_NAME: default
TABLE_NAME: src_test
COLUMN_NAME: price
COLUMN_TYPE: double
TBL_ID: 2586
LONG_LOW_VALUE: 0
LONG_HIGH_VALUE: 0
DOUBLE_LOW_VALUE: 0. # Wrong Result ! Expected is 1.
DOUBLE_HIGH_VALUE: 3.
BIG_DECIMAL_LOW_VALUE: NULL
BIG_DECIMAL_HIGH_VALUE: NULL
NUM_NULLS: 0
NUM_DISTINCTS: 1
AVG_COL_LEN: 0.
MAX_COL_LEN: 0
NUM_TRUES: 0
NUM_FALSES: 0
LAST_ANALYZED: 1368596151
2 rows in set (0.00 sec)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-4561) Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0.0000 ,if all the column values larger than 0.0 (or if all column values smaller than 0.0)
[ https://issues.apache.org/jira/browse/HIVE-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-4561: Status: Open (was: Patch Available) Ok, I'll take a look. Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0. ,if all the column values larger than 0.0 (or if all column values smaller than 0.0) Key: HIVE-4561 URL: https://issues.apache.org/jira/browse/HIVE-4561 Project: Hive Issue Type: Bug Components: Statistics Affects Versions: 0.13.0, 0.12.0 Reporter: caofangkun Assignee: Navis Attachments: HIVE-4561.1.patch, HIVE-4561.2.patch, HIVE-4561.3.patch, HIVE-4561.4.patch.txt, HIVE-4561.5.patch.txt, HIVE-4561.6.patch.txt if all column values larger than 0.0 DOUBLE_LOW_VALUE always will be 0.0 or if all column values less than 0.0, DOUBLE_HIGH_VALUE will always be hive (default) create table src_test (price double); hive (default) load data local inpath './test.txt' into table src_test; hive (default) select * from src_test; OK 1.0 2.0 3.0 Time taken: 0.313 seconds, Fetched: 3 row(s) hive (default) analyze table src_test compute statistics for columns price; mysql select * from TAB_COL_STATS \G; CS_ID: 16 DB_NAME: default TABLE_NAME: src_test COLUMN_NAME: price COLUMN_TYPE: double TBL_ID: 2586 LONG_LOW_VALUE: 0 LONG_HIGH_VALUE: 0 DOUBLE_LOW_VALUE: 0. # Wrong Result ! Expected is 1. DOUBLE_HIGH_VALUE: 3. BIG_DECIMAL_LOW_VALUE: NULL BIG_DECIMAL_HIGH_VALUE: NULL NUM_NULLS: 0 NUM_DISTINCTS: 1 AVG_COL_LEN: 0. MAX_COL_LEN: 0 NUM_TRUES: 0 NUM_FALSES: 0 LAST_ANALYZED: 1368596151 2 rows in set (0.00 sec) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-3925) dependencies of fetch task are not shown by explain
[ https://issues.apache.org/jira/browse/HIVE-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-3925: Attachment: HIVE-3925.7.patch.txt Rebased to trunk Updated diff files dependencies of fetch task are not shown by explain --- Key: HIVE-3925 URL: https://issues.apache.org/jira/browse/HIVE-3925 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: Navis Attachments: HIVE-3925.4.patch.txt, HIVE-3925.5.patch.txt, HIVE-3925.6.patch.txt, HIVE-3925.7.patch.txt, HIVE-3925.D8577.1.patch, HIVE-3925.D8577.2.patch, HIVE-3925.D8577.3.patch A simple query like: hive explain select * from src order by key; OK ABSTRACT SYNTAX TREE: (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME src))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR TOK_ALLCOLREF)) (TOK_ORDERBY (TOK_TABSORTCOLNAMEASC (TOK_TABLE_OR_COL key) STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 is a root stage Stage: Stage-0 Fetch Operator limit: -1 Stage-0 is not a root stage and depends on stage-1. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-3925) dependencies of fetch task are not shown by explain
[ https://issues.apache.org/jira/browse/HIVE-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-3925: Attachment: (was: HIVE-3925.7.patch.txt) dependencies of fetch task are not shown by explain --- Key: HIVE-3925 URL: https://issues.apache.org/jira/browse/HIVE-3925 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: Navis Attachments: HIVE-3925.4.patch.txt, HIVE-3925.5.patch.txt, HIVE-3925.6.patch.txt, HIVE-3925.D8577.1.patch, HIVE-3925.D8577.2.patch, HIVE-3925.D8577.3.patch A simple query like: hive explain select * from src order by key; OK ABSTRACT SYNTAX TREE: (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME src))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR TOK_ALLCOLREF)) (TOK_ORDERBY (TOK_TABSORTCOLNAMEASC (TOK_TABLE_OR_COL key) STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 is a root stage Stage: Stage-0 Fetch Operator limit: -1 Stage-0 is not a root stage and depends on stage-1. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-3925) dependencies of fetch task are not shown by explain
[ https://issues.apache.org/jira/browse/HIVE-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-3925: Attachment: HIVE-3925.7.patch.txt dependencies of fetch task are not shown by explain --- Key: HIVE-3925 URL: https://issues.apache.org/jira/browse/HIVE-3925 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: Navis Attachments: HIVE-3925.4.patch.txt, HIVE-3925.5.patch.txt, HIVE-3925.6.patch.txt, HIVE-3925.7.patch.txt, HIVE-3925.D8577.1.patch, HIVE-3925.D8577.2.patch, HIVE-3925.D8577.3.patch A simple query like: hive explain select * from src order by key; OK ABSTRACT SYNTAX TREE: (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME src))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR TOK_ALLCOLREF)) (TOK_ORDERBY (TOK_TABSORTCOLNAMEASC (TOK_TABLE_OR_COL key) STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 is a root stage Stage: Stage-0 Fetch Operator limit: -1 Stage-0 is not a root stage and depends on stage-1. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7087) Remove lineage information after query completion
[ https://issues.apache.org/jira/browse/HIVE-7087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008564#comment-14008564 ] Navis commented on HIVE-7087: - Ran test by manual. I've added 'a' for meaning that it contains a part of whole diff. But didn't know Hive QA accepts only numeric values, and I will take that in mind. Remove lineage information after query completion - Key: HIVE-7087 URL: https://issues.apache.org/jira/browse/HIVE-7087 Project: Hive Issue Type: Bug Components: Logging Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-7087.1.patch.txt, HIVE-7087.2.patch.txt, HIVE-7087.3a.patch.txt Lineage information is stacked in session and is not cleared before the session is closed. That also makes redundant lineage logs in q.out files for all of the queries after any inserts, which should be available only for insert queries. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-3907) Hive should support adding multiple resources at once
[ https://issues.apache.org/jira/browse/HIVE-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-3907: Attachment: HIVE-3907.3.patch.txt Updated diff file Hive should support adding multiple resources at once - Key: HIVE-3907 URL: https://issues.apache.org/jira/browse/HIVE-3907 Project: Hive Issue Type: Improvement Components: CLI Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-3907.2.patch.txt, HIVE-3907.3.patch.txt, HIVE-3907.D7971.1.patch Currently hive adds resources in one by one manner. And for JAR resources, one classloader is created for each jar file, which seemed not good idea. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7075) JsonSerde raises NullPointerException when object key is not lower case
[
https://issues.apache.org/jira/browse/HIVE-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-7075:
Attachment: HIVE-7075.3.patch.txt
JsonSerde raises NullPointerException when object key is not lower case
---
Key: HIVE-7075
URL: https://issues.apache.org/jira/browse/HIVE-7075
Project: Hive
Issue Type: Bug
Components: HCatalog, Serializers/Deserializers
Affects Versions: 0.12.0
Reporter: Yibing Shi
Attachments: HIVE-7075.1.patch.txt, HIVE-7075.2.patch.txt,
HIVE-7075.3.patch.txt
We have noticed that the JsonSerde produces a NullPointerException if a JSON
object has a key value that is not lower case. For example. Assume we have
the file one.json:
{ empId : 123, name : John }
{ empId : 456, name : Jane }
hive CREATE TABLE emps (empId INT, name STRING)
ROW FORMAT SERDE org.apache.hive.hcatalog.data.JsonSerDe;
hive LOAD DATA LOCAL INPATH 'one.json' INTO TABLE emps;
hive SELECT * FROM emps;
Failed with exception java.io.IOException:java.lang.NullPointerException
Notice, it seems to work if the keys are lower case. Assume we have the file
'two.json':
{ empid : 123, name : John }
{ empid : 456, name : Jane }
hive DROP TABLE emps;
hive CREATE TABLE emps (empId INT, name STRING)
ROW FORMAT SERDE org.apache.hive.hcatalog.data.JsonSerDe;
hive LOAD DATA LOCAL INPATH 'two.json' INTO TABLE emps;
hive SELECT * FROM emps;
OK
123 John
456 Jane
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7077) Hive contrib compilation maybe broken with removal of org.apache.hadoop.record
[
https://issues.apache.org/jira/browse/HIVE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008585#comment-14008585
]
Hive QA commented on HIVE-7077:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12646695/HIVE-7077.patch
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 5463 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_decimal1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/296/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/296/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-296/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12646695
Hive contrib compilation maybe broken with removal of org.apache.hadoop.record
--
Key: HIVE-7077
URL: https://issues.apache.org/jira/browse/HIVE-7077
Project: Hive
Issue Type: Bug
Components: Contrib
Affects Versions: 0.12.0, 0.13.0
Environment: Hadoop 2.4.0.5 and beyond
Reporter: Viraj Bhat
Assignee: Ashutosh Chauhan
Fix For: 0.13.0, 0.12.1
Attachments: HIVE-7077.patch
Hadoop decided to move record to hadoop-streaming so the compilation of the
contrib code will be broken if we do not include this jar.
{quote}
compile:
[echo] Project: contrib
[javac] Compiling 39 source files to
/home/y/var/builds/thread2/workspace/Cloud-Hive-branch-0.12-Hadoop2-Component-JDK7/build/contrib/classes
[javac]
/home/y/var/builds/thread2/workspace/Cloud-Hive-branch-0.12-Hadoop2-Component-JDK7/contrib/src/java/org/apache/hadoop/hive/contrib/util/typedbytes/TypedBytesWritableOutput.java:47:
error: package org.apache.hadoop.record does not exist
[javac] import org.apache.hadoop.record.Record;
[javac]^
[javac]
/home/y/var/builds/thread2/workspace/Cloud-Hive-branch-0.12-Hadoop2-Component-JDK7/contrib/src/java/org/apache/hadoop/hive/contrib/util/typedbytes/TypedBytesOutput.java:30:
error: package org.apache.hadoop.record does not exist
[javac] import org.apache.hadoop.record.Buffer;
[javac]^
[javac]
/home/y/var/builds/thread2/workspace/Cloud-Hive-branch-0.12-Hadoop2-Component-JDK7/contrib/src/java/org/apache/hadoop/hive/contrib/util/typedbytes/TypedBytesWritableOutput.java:224:
error: cannot find symbol
[javac] public void writeRecord(Record r) throws IOException {
[javac] ^
[javac] symbol: class Record
[javac] location: class TypedBytesWritableOutput
[javac]
/home/y/var/builds/thread2/workspace/Cloud-Hive-branch-0.12-Hadoop2-Component-JDK7/contrib/src/java/org/apache/hadoop/hive/contrib/util/typedbytes/TypedBytesInput.java:29:
error: package org.apache.hadoop.record does not exist
[javac] import org.apache.hadoop.record.Buffer;
[javac]^
[javac]
/home/y/var/builds/thread2/workspace/Cloud-Hive-branch-0.12-Hadoop2-Component-JDK7/contrib/src/java/org/apache/hadoop/hive/contrib/util/typedbytes/TypedBytesRecordInput.java:24:
error: package org.apache.hadoop.record does not exist
[javac] import org.apache.hadoop.record.Buffer;
[javac]^
{quote}
Besides this, https://issues.apache.org/jira/browse/HADOOP-10485 removes most
of these classes. This Jira is being created to track this.
Viraj
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5799) session/operation timeout for hiveserver2
[ https://issues.apache.org/jira/browse/HIVE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-5799: Attachment: HIVE-5799.9.patch.txt session/operation timeout for hiveserver2 - Key: HIVE-5799 URL: https://issues.apache.org/jira/browse/HIVE-5799 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-5799.1.patch.txt, HIVE-5799.2.patch.txt, HIVE-5799.3.patch.txt, HIVE-5799.4.patch.txt, HIVE-5799.5.patch.txt, HIVE-5799.6.patch.txt, HIVE-5799.7.patch.txt, HIVE-5799.8.patch.txt, HIVE-5799.9.patch.txt Need some timeout facility for preventing resource leakages from instable or bad clients. -- This message was sent by Atlassian JIRA (v6.2#6252)
