[jira] [Commented] (HIVE-3251) Hive doesn't remove scrach directories while killing running MR job
[ https://issues.apache.org/jira/browse/HIVE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418109#comment-13418109 ] Namit Jain commented on HIVE-3251: -- +1 Hive doesn't remove scrach directories while killing running MR job --- Key: HIVE-3251 URL: https://issues.apache.org/jira/browse/HIVE-3251 Project: Hive Issue Type: Bug Components: Server Infrastructure Reporter: Gang Tim Liu Assignee: Gang Tim Liu Attachments: HIVE-3151.patch While killing running MR job, hive doesn't clean up scratch directory (mapred.cache.files). So that, afterwards, scratch directory is left there in hdfs. HDFS name node doesn't know it and try to do lease recovery. while such instances happen more, it will eventually crash namenode. The fix is to leverage hdfs clean up functionality. While creating scratch dirs, hive registers it to hdfs cleanup hook. While killing happens, hdfs will clean them up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3221) HiveConf.getPositionFromInternalName does not support more than sinle digit column numbers
[ https://issues.apache.org/jira/browse/HIVE-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418119#comment-13418119 ] Ashutosh Chauhan commented on HIVE-3221: +1. Looks good. Will commit if tests pass. HiveConf.getPositionFromInternalName does not support more than sinle digit column numbers -- Key: HIVE-3221 URL: https://issues.apache.org/jira/browse/HIVE-3221 Project: Hive Issue Type: Bug Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: HIVE-3221.patch For positions above 9, HiveConf.getPositionFromInternalName only looks at the last digit, and thus, causes collisions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3205) Bucketed mapjoin on partitioned table which has no partition throws NPE
[
https://issues.apache.org/jira/browse/HIVE-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418133#comment-13418133
]
Namit Jain commented on HIVE-3205:
--
very minor comments - mostly looks good
Bucketed mapjoin on partitioned table which has no partition throws NPE
---
Key: HIVE-3205
URL: https://issues.apache.org/jira/browse/HIVE-3205
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.10.0
Environment: ubuntu 10.04
Reporter: Navis
Assignee: Navis
Priority: Minor
{code}
create table hive_test_smb_bucket1 (key int, value string) partitioned by (ds
string) clustered by (key) sorted by (key) into 2 buckets;
create table hive_test_smb_bucket2 (key int, value string) partitioned by (ds
string) clustered by (key) sorted by (key) into 2 buckets;
set hive.optimize.bucketmapjoin = true;
set hive.input.format =
org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
explain
SELECT /* + MAPJOIN(b) */ b.key as k1, b.value, b.ds, a.key as k2
FROM hive_test_smb_bucket1 a JOIN
hive_test_smb_bucket2 b
ON a.key = b.key WHERE a.ds = '2010-10-15' and b.ds='2010-10-15' and b.key
IS NOT NULL;
{code}
throws NPE
{noformat}
2012-06-28 08:59:13,459 ERROR ql.Driver (SessionState.java:printError(400)) -
FAILED: NullPointerException null
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.optimizer.BucketMapJoinOptimizer$BucketMapjoinOptProc.process(BucketMapJoinOptimizer.java:269)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:125)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102)
at
org.apache.hadoop.hive.ql.optimizer.BucketMapJoinOptimizer.transform(BucketMapJoinOptimizer.java:100)
at
org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:87)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7564)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:245)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:50)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:245)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:744)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:607)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3205) Bucketed mapjoin on partitioned table which has no partition throws NPE
[
https://issues.apache.org/jira/browse/HIVE-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Namit Jain updated HIVE-3205:
-
Status: Open (was: Patch Available)
Bucketed mapjoin on partitioned table which has no partition throws NPE
---
Key: HIVE-3205
URL: https://issues.apache.org/jira/browse/HIVE-3205
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.10.0
Environment: ubuntu 10.04
Reporter: Navis
Assignee: Navis
Priority: Minor
{code}
create table hive_test_smb_bucket1 (key int, value string) partitioned by (ds
string) clustered by (key) sorted by (key) into 2 buckets;
create table hive_test_smb_bucket2 (key int, value string) partitioned by (ds
string) clustered by (key) sorted by (key) into 2 buckets;
set hive.optimize.bucketmapjoin = true;
set hive.input.format =
org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
explain
SELECT /* + MAPJOIN(b) */ b.key as k1, b.value, b.ds, a.key as k2
FROM hive_test_smb_bucket1 a JOIN
hive_test_smb_bucket2 b
ON a.key = b.key WHERE a.ds = '2010-10-15' and b.ds='2010-10-15' and b.key
IS NOT NULL;
{code}
throws NPE
{noformat}
2012-06-28 08:59:13,459 ERROR ql.Driver (SessionState.java:printError(400)) -
FAILED: NullPointerException null
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.optimizer.BucketMapJoinOptimizer$BucketMapjoinOptProc.process(BucketMapJoinOptimizer.java:269)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:125)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102)
at
org.apache.hadoop.hive.ql.optimizer.BucketMapJoinOptimizer.transform(BucketMapJoinOptimizer.java:100)
at
org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:87)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7564)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:245)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:50)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:245)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:744)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:607)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3070) Filter on outer join condition removed while merging join tree
[
https://issues.apache.org/jira/browse/HIVE-3070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-3070:
Affects Version/s: 0.10.0
0.9.0
Assignee: Navis
Summary: Filter on outer join condition removed while merging
join tree (was: outer join with filter over multiple tables on the same column
have bug)
Filter on outer join condition removed while merging join tree
--
Key: HIVE-3070
URL: https://issues.apache.org/jira/browse/HIVE-3070
Project: Hive
Issue Type: Bug
Components: SQL
Affects Versions: 0.8.1, 0.9.0, 0.10.0
Reporter: Zhang Xinyu
Assignee: Navis
should the result of query A:
select s.aa, s.bb, c.key keyc from (select a.key aa, b.key bb from src a left
outer join src b on a.key=b.key) s left outer join src c on s.bb=c.key and
s.bb10 where s.aa20;
be the same as query B:
select a.key keya, b.key keyb, c.key keyc from src a left outer join src b on
a.key=b.key left outer join src c on b.key=c.key and b.key10 where a.key20;
?
Currently, the result is different, query B gets wrong result!
In SemanticAnalyzer.java, mergeJoins():
ArrayListArrayListASTNode filters = target.getFilters();
for (int i = 0; i nodeRightAliases.length; i++) {
filters.add(node.getFilters().get(i + 1));
}
filters in node.getFilters().get(0) are lost.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3070) Filter on outer join condition removed while merging join tree
[
https://issues.apache.org/jira/browse/HIVE-3070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-3070:
Status: Patch Available (was: Open)
https://reviews.facebook.net/D4239
Filter on outer join condition removed while merging join tree
--
Key: HIVE-3070
URL: https://issues.apache.org/jira/browse/HIVE-3070
Project: Hive
Issue Type: Bug
Components: SQL
Affects Versions: 0.9.0, 0.8.1, 0.10.0
Reporter: Zhang Xinyu
Assignee: Navis
should the result of query A:
select s.aa, s.bb, c.key keyc from (select a.key aa, b.key bb from src a left
outer join src b on a.key=b.key) s left outer join src c on s.bb=c.key and
s.bb10 where s.aa20;
be the same as query B:
select a.key keya, b.key keyb, c.key keyc from src a left outer join src b on
a.key=b.key left outer join src c on b.key=c.key and b.key10 where a.key20;
?
Currently, the result is different, query B gets wrong result!
In SemanticAnalyzer.java, mergeJoins():
ArrayListArrayListASTNode filters = target.getFilters();
for (int i = 0; i nodeRightAliases.length; i++) {
filters.add(node.getFilters().get(i + 1));
}
filters in node.getFilters().get(0) are lost.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3205) Bucketed mapjoin on partitioned table which has no partition throws NPE
[
https://issues.apache.org/jira/browse/HIVE-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-3205:
Status: Patch Available (was: Open)
Bucketed mapjoin on partitioned table which has no partition throws NPE
---
Key: HIVE-3205
URL: https://issues.apache.org/jira/browse/HIVE-3205
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.10.0
Environment: ubuntu 10.04
Reporter: Navis
Assignee: Navis
Priority: Minor
{code}
create table hive_test_smb_bucket1 (key int, value string) partitioned by (ds
string) clustered by (key) sorted by (key) into 2 buckets;
create table hive_test_smb_bucket2 (key int, value string) partitioned by (ds
string) clustered by (key) sorted by (key) into 2 buckets;
set hive.optimize.bucketmapjoin = true;
set hive.input.format =
org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
explain
SELECT /* + MAPJOIN(b) */ b.key as k1, b.value, b.ds, a.key as k2
FROM hive_test_smb_bucket1 a JOIN
hive_test_smb_bucket2 b
ON a.key = b.key WHERE a.ds = '2010-10-15' and b.ds='2010-10-15' and b.key
IS NOT NULL;
{code}
throws NPE
{noformat}
2012-06-28 08:59:13,459 ERROR ql.Driver (SessionState.java:printError(400)) -
FAILED: NullPointerException null
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.optimizer.BucketMapJoinOptimizer$BucketMapjoinOptProc.process(BucketMapJoinOptimizer.java:269)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:125)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102)
at
org.apache.hadoop.hive.ql.optimizer.BucketMapJoinOptimizer.transform(BucketMapJoinOptimizer.java:100)
at
org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:87)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7564)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:245)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:50)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:245)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:744)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:607)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3205) Bucketed mapjoin on partitioned table which has no partition throws NPE
[
https://issues.apache.org/jira/browse/HIVE-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418178#comment-13418178
]
Namit Jain commented on HIVE-3205:
--
+1
Bucketed mapjoin on partitioned table which has no partition throws NPE
---
Key: HIVE-3205
URL: https://issues.apache.org/jira/browse/HIVE-3205
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.10.0
Environment: ubuntu 10.04
Reporter: Navis
Assignee: Navis
Priority: Minor
{code}
create table hive_test_smb_bucket1 (key int, value string) partitioned by (ds
string) clustered by (key) sorted by (key) into 2 buckets;
create table hive_test_smb_bucket2 (key int, value string) partitioned by (ds
string) clustered by (key) sorted by (key) into 2 buckets;
set hive.optimize.bucketmapjoin = true;
set hive.input.format =
org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
explain
SELECT /* + MAPJOIN(b) */ b.key as k1, b.value, b.ds, a.key as k2
FROM hive_test_smb_bucket1 a JOIN
hive_test_smb_bucket2 b
ON a.key = b.key WHERE a.ds = '2010-10-15' and b.ds='2010-10-15' and b.key
IS NOT NULL;
{code}
throws NPE
{noformat}
2012-06-28 08:59:13,459 ERROR ql.Driver (SessionState.java:printError(400)) -
FAILED: NullPointerException null
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.optimizer.BucketMapJoinOptimizer$BucketMapjoinOptProc.process(BucketMapJoinOptimizer.java:269)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:125)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102)
at
org.apache.hadoop.hive.ql.optimizer.BucketMapJoinOptimizer.transform(BucketMapJoinOptimizer.java:100)
at
org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:87)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7564)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:245)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:50)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:245)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:744)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:607)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3271) Previledge can be granted by any user(not owner) to any user(even to the same user)
[ https://issues.apache.org/jira/browse/HIVE-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Unnikrishnan V T updated HIVE-3271: --- I have created two users user 'unni' and user 'sachin'. user unni created a table 'test3' so that user sachin cannot view that table. But user sachin is able to grant all permission to the table test3. I have set 1)hive.security.authorization.enabled=true(in hive-site.xml) 2)dfs.permissions=true(in hdfs-site.xml) 3)dfs.permissions.supergroup=supergroup(in hdfs-site.xml) user unni is in supergroup group. The user sachin is even able to revoke all permissions from the owner of the table user unni. User sachin is not a member of supergroup. Previledge can be granted by any user(not owner) to any user(even to the same user) --- Key: HIVE-3271 URL: https://issues.apache.org/jira/browse/HIVE-3271 Project: Hive Issue Type: Bug Components: Authorization Affects Versions: 0.8.1 Reporter: Unnikrishnan V T Priority: Critical Attachments: Screenshot.png I have created two users user 'unni' and user 'sachin'. user unni created a table 'test3' so that user sachin cannot view that table. But user sachin is able to grant all permission to the table test3. I have set 1)hive.security.authorization.enabled=true(in hive-site.xml) 2)dfs.permissions=true(in hdfs-site.xml) 3)dfs.permissions.supergroup=supergroup(in hdfs-site.xml) User sachin and user unni are in supergroup group. The user sachin is even able to revoke all permissions from the owner of the table user unni. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3246) java primitive type for binary datatype should be byte[]
[ https://issues.apache.org/jira/browse/HIVE-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418208#comment-13418208 ] Edward Capriolo commented on HIVE-3246: --- We have to do a release note it people who write binary UDFs are going to be effected by the change. https://github.com/edwardcapriolo/hive_cassandra_udfs It seems like if you wrote a UDF that returns binary you may be effected depending on how you wrote it. java primitive type for binary datatype should be byte[] Key: HIVE-3246 URL: https://issues.apache.org/jira/browse/HIVE-3246 Project: Hive Issue Type: Bug Affects Versions: 0.9.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-3246.1.patch, HIVE-3246.2.patch PrimitiveObjectInspector.getPrimitiveJavaObject is supposed to return a java object. But in case of binary datatype, it returns ByteArrayRef (not java standard type). The suitable java object for it would be byte[]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3273) Add avro jars into hive execution classpath
[ https://issues.apache.org/jira/browse/HIVE-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418210#comment-13418210 ] Edward Capriolo commented on HIVE-3273: --- I can see this happening but when exactly do we get this error? Add avro jars into hive execution classpath --- Key: HIVE-3273 URL: https://issues.apache.org/jira/browse/HIVE-3273 Project: Hive Issue Type: Bug Reporter: Zhenxiao Luo Assignee: Zhenxiao Luo Attachments: HIVE-3273.1.patch.txt avro*.jar should be added to hive execution classpath -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3098) Memory leak from large number of FileSystem instances in FileSystem.CACHE. (Must cache UGIs.)
[
https://issues.apache.org/jira/browse/HIVE-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418294#comment-13418294
]
Daryn Sharp commented on HIVE-3098:
---
It's probably safer to use the iterator's remove instead of a direct remove
from the map. Ie. change {{trashedUGIs.add(ugiCache.remove(entry.getKey())}}
to {{entry.remove(); trashedUGIs.add(entry.getKey)}}.
How does this protect against requests using trashed UGIs having their
filesystems closed before the request is complete? Is it assuming all requests
will complete within the time interval between which the ugi is trashed and
then purged? If true, it seems fragile esp. when considering rpc retries, HA
failover delays, etc.
Has anyone benchmarked cached ugis to ensure this isn't a premature
optimization?
Memory leak from large number of FileSystem instances in FileSystem.CACHE.
(Must cache UGIs.)
-
Key: HIVE-3098
URL: https://issues.apache.org/jira/browse/HIVE-3098
Project: Hive
Issue Type: Bug
Components: Shims
Affects Versions: 0.9.0
Environment: Running with Hadoop 20.205.0.3+ / 1.0.x with security
turned on.
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Attachments: Hive_3098.patch
The problem manifested from stress-testing HCatalog 0.4.1 (as part of testing
the Oracle backend).
The HCatalog server ran out of memory (-Xmx2048m) when pounded by 60-threads,
in under 24 hours. The heap-dump indicates that hadoop::FileSystem.CACHE had
100 instances of FileSystem, whose combined retained-mem consumed the
entire heap.
It boiled down to hadoop::UserGroupInformation::equals() being implemented
such that the Subject member is compared for equality (==), and not
equivalence (.equals()). This causes equivalent UGI instances to compare as
unequal, and causes a new FileSystem instance to be created and cached.
The UGI.equals() is so implemented, incidentally, as a fix for yet another
problem (HADOOP-6670); so it is unlikely that that implementation can be
modified.
The solution for this is to check for UGI equivalence in HCatalog (i.e. in
the Hive metastore), using an cache for UGI instances in the shims.
I have a patch to fix this. I'll upload it shortly. I just ran an overnight
test to confirm that the memory-leak has been arrested.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false #80
See https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/80/ -- [...truncated 10160 lines...] [echo] Project: odbc [copy] Warning: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/odbc/src/conf does not exist. ivy-resolve-test: [echo] Project: odbc ivy-retrieve-test: [echo] Project: odbc compile-test: [echo] Project: odbc create-dirs: [echo] Project: serde [copy] Warning: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/serde/src/test/resources does not exist. init: [echo] Project: serde ivy-init-settings: [echo] Project: serde ivy-resolve: [echo] Project: serde [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml [ivy:report] Processing https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/80/artifact/hive/build/ivy/resolution-cache/org.apache.hive-hive-serde-default.xml to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/80/artifact/hive/build/ivy/report/org.apache.hive-hive-serde-default.html ivy-retrieve: [echo] Project: serde dynamic-serde: compile: [echo] Project: serde ivy-resolve-test: [echo] Project: serde ivy-retrieve-test: [echo] Project: serde compile-test: [echo] Project: serde [javac] Compiling 26 source files to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/80/artifact/hive/build/serde/test/classes [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. create-dirs: [echo] Project: service [copy] Warning: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/service/src/test/resources does not exist. init: [echo] Project: service ivy-init-settings: [echo] Project: service ivy-resolve: [echo] Project: service [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml [ivy:report] Processing https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/80/artifact/hive/build/ivy/resolution-cache/org.apache.hive-hive-service-default.xml to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/80/artifact/hive/build/ivy/report/org.apache.hive-hive-service-default.html ivy-retrieve: [echo] Project: service compile: [echo] Project: service ivy-resolve-test: [echo] Project: service ivy-retrieve-test: [echo] Project: service compile-test: [echo] Project: service [javac] Compiling 2 source files to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/80/artifact/hive/build/service/test/classes test: [echo] Project: hive test-shims: [echo] Project: hive test-conditions: [echo] Project: shims gen-test: [echo] Project: shims create-dirs: [echo] Project: shims [copy] Warning: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/shims/src/test/resources does not exist. init: [echo] Project: shims ivy-init-settings: [echo] Project: shims ivy-resolve: [echo] Project: shims [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml [ivy:report] Processing https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/80/artifact/hive/build/ivy/resolution-cache/org.apache.hive-hive-shims-default.xml to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/80/artifact/hive/build/ivy/report/org.apache.hive-hive-shims-default.html ivy-retrieve: [echo] Project: shims compile: [echo] Project: shims [echo] Building shims 0.20 build_shims: [echo] Project: shims [echo] Compiling https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/shims/src/common/java;/home/hudson/hudson-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/shims/src/0.20/java against hadoop 0.20.2 (https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/80/artifact/hive/build/hadoopcore/hadoop-0.20.2) ivy-init-settings: [echo] Project: shims ivy-resolve-hadoop-shim: [echo] Project: shims [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml ivy-retrieve-hadoop-shim: [echo] Project: shims [echo] Building shims 0.20S build_shims: [echo] Project: shims [echo] Compiling
[jira] [Commented] (HIVE-3098) Memory leak from large number of FileSystem instances in FileSystem.CACHE. (Must cache UGIs.)
[ https://issues.apache.org/jira/browse/HIVE-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418441#comment-13418441 ] Mithun Radhakrishnan commented on HIVE-3098: Hey, All. First off, I'm considering a rewrite, essentially identical but implemented via Guava caching. This will save on lines-of-code. @Ashutosh: Sure. I'll put a new patch on Phabricator, as soon as I've incorporated Rohini's comments. The code could stand to be refactored. @Daryn: Thank you for reviewing. 1. If the concern is over ConcurrentModificationExceptions, that's not a problem since ugiCache is a ConcurrentHashMap. :] 2. Yes, that was the assumption, but you have a point. Perhaps a better way to deal with this is through WeakReferences (or some such). 3. I'm not sure I follow your concern about premature-optimization. We're caching to avoid the creation of multiple FS instances. We've seen from tests that this solves that problem. Would you please elaborate? Memory leak from large number of FileSystem instances in FileSystem.CACHE. (Must cache UGIs.) - Key: HIVE-3098 URL: https://issues.apache.org/jira/browse/HIVE-3098 Project: Hive Issue Type: Bug Components: Shims Affects Versions: 0.9.0 Environment: Running with Hadoop 20.205.0.3+ / 1.0.x with security turned on. Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: Hive_3098.patch The problem manifested from stress-testing HCatalog 0.4.1 (as part of testing the Oracle backend). The HCatalog server ran out of memory (-Xmx2048m) when pounded by 60-threads, in under 24 hours. The heap-dump indicates that hadoop::FileSystem.CACHE had 100 instances of FileSystem, whose combined retained-mem consumed the entire heap. It boiled down to hadoop::UserGroupInformation::equals() being implemented such that the Subject member is compared for equality (==), and not equivalence (.equals()). This causes equivalent UGI instances to compare as unequal, and causes a new FileSystem instance to be created and cached. The UGI.equals() is so implemented, incidentally, as a fix for yet another problem (HADOOP-6670); so it is unlikely that that implementation can be modified. The solution for this is to check for UGI equivalence in HCatalog (i.e. in the Hive metastore), using an cache for UGI instances in the shims. I have a patch to fix this. I'll upload it shortly. I just ran an overnight test to confirm that the memory-leak has been arrested. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-3251) Hive doesn't remove scrach directories while killing running MR job
[ https://issues.apache.org/jira/browse/HIVE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain resolved HIVE-3251. -- Resolution: Fixed Hadoop Flags: Reviewed Committed. Thanks Tim Hive doesn't remove scrach directories while killing running MR job --- Key: HIVE-3251 URL: https://issues.apache.org/jira/browse/HIVE-3251 Project: Hive Issue Type: Bug Components: Server Infrastructure Reporter: Gang Tim Liu Assignee: Gang Tim Liu Attachments: HIVE-3151.patch While killing running MR job, hive doesn't clean up scratch directory (mapred.cache.files). So that, afterwards, scratch directory is left there in hdfs. HDFS name node doesn't know it and try to do lease recovery. while such instances happen more, it will eventually crash namenode. The fix is to leverage hdfs clean up functionality. While creating scratch dirs, hive registers it to hdfs cleanup hook. While killing happens, hdfs will clean them up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3246) java primitive type for binary datatype should be byte[]
[ https://issues.apache.org/jira/browse/HIVE-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418447#comment-13418447 ] Ashutosh Chauhan commented on HIVE-3246: Yeah. I will add the release notes while resolving this ticket. java primitive type for binary datatype should be byte[] Key: HIVE-3246 URL: https://issues.apache.org/jira/browse/HIVE-3246 Project: Hive Issue Type: Bug Affects Versions: 0.9.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-3246.1.patch, HIVE-3246.2.patch PrimitiveObjectInspector.getPrimitiveJavaObject is supposed to return a java object. But in case of binary datatype, it returns ByteArrayRef (not java standard type). The suitable java object for it would be byte[]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3246) java primitive type for binary datatype should be byte[]
[ https://issues.apache.org/jira/browse/HIVE-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-3246: --- Resolution: Fixed Fix Version/s: 0.10.0 Release Note: This fix changes the internal representation of binary type within Hive. UDFs which were using binary type and making use of the fact that java representation of binary data in Hive is ByteArrayRef needs to be updated to reflect that now that representation is byte[] instead. Also note that this doesn't change the format for on-disk data. (was: Makes the java primitive type for binary datatype - byte[] ) Hadoop Flags: Incompatible change Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Thejas! java primitive type for binary datatype should be byte[] Key: HIVE-3246 URL: https://issues.apache.org/jira/browse/HIVE-3246 Project: Hive Issue Type: Bug Affects Versions: 0.9.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.10.0 Attachments: HIVE-3246.1.patch, HIVE-3246.2.patch PrimitiveObjectInspector.getPrimitiveJavaObject is supposed to return a java object. But in case of binary datatype, it returns ByteArrayRef (not java standard type). The suitable java object for it would be byte[]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3273) Add avro jars into hive execution classpath
[ https://issues.apache.org/jira/browse/HIVE-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418478#comment-13418478 ] Zhenxiao Luo commented on HIVE-3273: @Edward: I found this error when I am backporting HIVE-895 to an old version of Hive(CDH3), where I did not put avro jars into hive execution classpath, and kept having the following exception: java.lang.ClassNotFoundException: org.apache.avro.io.DatumReader We do not have this error on current trunk, since Hadoop classpath is not setting correctly in build-common.xml test target. I think this should be fixed in HIVE-3274. I think fixing this would emphasis that it is not enough to put avro jars in the hive client classpath. Avro jars should be put in hive execution(the mapreduce job) classpath. This would prevent future bugs. What do you think about it? Thanks, Zhenxiao Add avro jars into hive execution classpath --- Key: HIVE-3273 URL: https://issues.apache.org/jira/browse/HIVE-3273 Project: Hive Issue Type: Bug Reporter: Zhenxiao Luo Assignee: Zhenxiao Luo Attachments: HIVE-3273.1.patch.txt avro*.jar should be added to hive execution classpath -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3098) Memory leak from large number of FileSystem instances in FileSystem.CACHE. (Must cache UGIs.)
[
https://issues.apache.org/jira/browse/HIVE-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418485#comment-13418485
]
Daryn Sharp commented on HIVE-3098:
---
I think the large discussion has obscured the fact that there's nothing wrong
with the fs api. The leak is a result of hive misusing the fs api. Simply
calling {{FileSystem.closeAllForUGI(remoteUser)}} at the end of a request will
stop hive from leaking. My guess, and I could be wrong, is that trying to
cache UGIs will provide a negligible performance increase. Hence why I
question if this is premature optimization.
Memory leak from large number of FileSystem instances in FileSystem.CACHE.
(Must cache UGIs.)
-
Key: HIVE-3098
URL: https://issues.apache.org/jira/browse/HIVE-3098
Project: Hive
Issue Type: Bug
Components: Shims
Affects Versions: 0.9.0
Environment: Running with Hadoop 20.205.0.3+ / 1.0.x with security
turned on.
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Attachments: Hive_3098.patch
The problem manifested from stress-testing HCatalog 0.4.1 (as part of testing
the Oracle backend).
The HCatalog server ran out of memory (-Xmx2048m) when pounded by 60-threads,
in under 24 hours. The heap-dump indicates that hadoop::FileSystem.CACHE had
100 instances of FileSystem, whose combined retained-mem consumed the
entire heap.
It boiled down to hadoop::UserGroupInformation::equals() being implemented
such that the Subject member is compared for equality (==), and not
equivalence (.equals()). This causes equivalent UGI instances to compare as
unequal, and causes a new FileSystem instance to be created and cached.
The UGI.equals() is so implemented, incidentally, as a fix for yet another
problem (HADOOP-6670); so it is unlikely that that implementation can be
modified.
The solution for this is to check for UGI equivalence in HCatalog (i.e. in
the Hive metastore), using an cache for UGI instances in the shims.
I have a patch to fix this. I'll upload it shortly. I just ran an overnight
test to confirm that the memory-leak has been arrested.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3251) Hive doesn't remove scrach directories while killing running MR job
[ https://issues.apache.org/jira/browse/HIVE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418515#comment-13418515 ] Gang Tim Liu commented on HIVE-3251: Namit, thank you very much for review and commit. Hive doesn't remove scrach directories while killing running MR job --- Key: HIVE-3251 URL: https://issues.apache.org/jira/browse/HIVE-3251 Project: Hive Issue Type: Bug Components: Server Infrastructure Reporter: Gang Tim Liu Assignee: Gang Tim Liu Attachments: HIVE-3151.patch While killing running MR job, hive doesn't clean up scratch directory (mapred.cache.files). So that, afterwards, scratch directory is left there in hdfs. HDFS name node doesn't know it and try to do lease recovery. while such instances happen more, it will eventually crash namenode. The fix is to leverage hdfs clean up functionality. While creating scratch dirs, hive registers it to hdfs cleanup hook. While killing happens, hdfs will clean them up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1653) Ability to enforce correct stats
[ https://issues.apache.org/jira/browse/HIVE-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418545#comment-13418545 ] Kevin Wilfong commented on HIVE-1653: - Commented on the diff. Ability to enforce correct stats Key: HIVE-1653 URL: https://issues.apache.org/jira/browse/HIVE-1653 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain This is a follow-up for https://issues.apache.org/jira/browse/HIVE-1361. If one of the mappers/reducers cannot publish stats, it may lead to wrong aggregated stats. There should be a way to avoid this - at the least, a configuration variable which fails the task if stats cannot be published -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1653) Ability to enforce correct stats
[ https://issues.apache.org/jira/browse/HIVE-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-1653: Status: Open (was: Patch Available) Ability to enforce correct stats Key: HIVE-1653 URL: https://issues.apache.org/jira/browse/HIVE-1653 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain This is a follow-up for https://issues.apache.org/jira/browse/HIVE-1361. If one of the mappers/reducers cannot publish stats, it may lead to wrong aggregated stats. There should be a way to avoid this - at the least, a configuration variable which fails the task if stats cannot be published -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3261) alter the number of buckets for a non-empty partitioned table should not be allowed
[ https://issues.apache.org/jira/browse/HIVE-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418550#comment-13418550 ] Kevin Wilfong commented on HIVE-3261: - This seems ok for now, but could you file a follow up task to use partition metadata instead of table metadata wherever bucketing information is used, and mention that the error message added by this JIRA can be removed once that is done. One small comment on the diff. alter the number of buckets for a non-empty partitioned table should not be allowed --- Key: HIVE-3261 URL: https://issues.apache.org/jira/browse/HIVE-3261 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain This is dangerous since the code uses the table metadata everywhere to get the number of buckets -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3261) alter the number of buckets for a non-empty partitioned table should not be allowed
[ https://issues.apache.org/jira/browse/HIVE-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-3261: - Status: Open (was: Patch Available) @Namit: I also added a small comment on phabricator. Thanks. alter the number of buckets for a non-empty partitioned table should not be allowed --- Key: HIVE-3261 URL: https://issues.apache.org/jira/browse/HIVE-3261 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain This is dangerous since the code uses the table metadata everywhere to get the number of buckets -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3261) alter the number of buckets for a non-empty partitioned table should not be allowed
[ https://issues.apache.org/jira/browse/HIVE-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-3261: - Component/s: Query Processor Labels: bucketing (was: ) alter the number of buckets for a non-empty partitioned table should not be allowed --- Key: HIVE-3261 URL: https://issues.apache.org/jira/browse/HIVE-3261 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain Labels: bucketing This is dangerous since the code uses the table metadata everywhere to get the number of buckets -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3271) Previledge can be granted by any user(not owner) to any user(even to the same user)
[ https://issues.apache.org/jira/browse/HIVE-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-3271: - Component/s: Security Previledge can be granted by any user(not owner) to any user(even to the same user) --- Key: HIVE-3271 URL: https://issues.apache.org/jira/browse/HIVE-3271 Project: Hive Issue Type: Bug Components: Authorization, Security Affects Versions: 0.8.1 Reporter: Unnikrishnan V T Priority: Critical Attachments: Screenshot.png I have created two users user 'unni' and user 'sachin'. user unni created a table 'test3' so that user sachin cannot view that table. But user sachin is able to grant all permission to the table test3. I have set 1)hive.security.authorization.enabled=true(in hive-site.xml) 2)dfs.permissions=true(in hdfs-site.xml) 3)dfs.permissions.supergroup=supergroup(in hdfs-site.xml) User sachin and user unni are in supergroup group. The user sachin is even able to revoke all permissions from the owner of the table user unni. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3271) Privilege can be granted by any user(not owner) to any user(even to the same user)
[ https://issues.apache.org/jira/browse/HIVE-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-3271: - Summary: Privilege can be granted by any user(not owner) to any user(even to the same user) (was: Previledge can be granted by any user(not owner) to any user(even to the same user)) Privilege can be granted by any user(not owner) to any user(even to the same user) -- Key: HIVE-3271 URL: https://issues.apache.org/jira/browse/HIVE-3271 Project: Hive Issue Type: Bug Components: Authorization, Security Affects Versions: 0.8.1 Reporter: Unnikrishnan V T Priority: Critical Attachments: Screenshot.png I have created two users user 'unni' and user 'sachin'. user unni created a table 'test3' so that user sachin cannot view that table. But user sachin is able to grant all permission to the table test3. I have set 1)hive.security.authorization.enabled=true(in hive-site.xml) 2)dfs.permissions=true(in hdfs-site.xml) 3)dfs.permissions.supergroup=supergroup(in hdfs-site.xml) User sachin and user unni are in supergroup group. The user sachin is even able to revoke all permissions from the owner of the table user unni. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Jenkins build is back to normal : Hive-0.9.1-SNAPSHOT-h0.21 #80
See https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/80/
[jira] [Updated] (HIVE-1948) Have audit logging in the Metastore
[ https://issues.apache.org/jira/browse/HIVE-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-1948: - Component/s: Security Logging Have audit logging in the Metastore --- Key: HIVE-1948 URL: https://issues.apache.org/jira/browse/HIVE-1948 Project: Hive Issue Type: Improvement Components: Logging, Metastore, Security Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.7.0 Attachments: audit-log-2.patch, audit-log-3.patch, audit-log.1.patch, audit-log.patch It would be good to have audit logging in the metastore, similar to Hadoop's NameNode audit logging. This would allow administrators to dig into details about which user performed metadata operations (like create/drop tables/partitions) and from where (IP address). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3277) Enable Metastore audit logging for non-secure connections
Carl Steinbach created HIVE-3277: Summary: Enable Metastore audit logging for non-secure connections Key: HIVE-3277 URL: https://issues.apache.org/jira/browse/HIVE-3277 Project: Hive Issue Type: Improvement Components: Logging, Metastore, Security Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3277) Enable Metastore audit logging for non-secure connections
[ https://issues.apache.org/jira/browse/HIVE-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418664#comment-13418664 ] Carl Steinbach commented on HIVE-3277: -- Audit logging for the metastore was added in HIVE-1948, but it only gets invoked if the client connection is secure. We should also enable this for non-secure connections. Enable Metastore audit logging for non-secure connections - Key: HIVE-3277 URL: https://issues.apache.org/jira/browse/HIVE-3277 Project: Hive Issue Type: Improvement Components: Logging, Metastore, Security Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3278) Create wrappers around Thrift Metastore objects for improved readability
Carl Steinbach created HIVE-3278: Summary: Create wrappers around Thrift Metastore objects for improved readability Key: HIVE-3278 URL: https://issues.apache.org/jira/browse/HIVE-3278 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Carl Steinbach Assignee: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1552 - Still Failing
Changes for Build #1549 Changes for Build #1550 [namit] HIVE-3230 Make logging of plan progress in HadoopJobExecHelper configurable (Kevin Wilfong via namit) Changes for Build #1551 Changes for Build #1552 1 tests failed. REGRESSION: org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe(TestNegativeMinimrCliDriver.java:212) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:422) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:931) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:785) The Apache Jenkins build system has built Hive-trunk-h0.21 (build #$BUILD_NUMBER) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1552/ to view the results.
[jira] [Created] (HIVE-3279) Table schema not being copied to Partitions with no columns
Travis Crawford created HIVE-3279:
-
Summary: Table schema not being copied to Partitions with no
columns
Key: HIVE-3279
URL: https://issues.apache.org/jira/browse/HIVE-3279
Project: Hive
Issue Type: Bug
Reporter: Travis Crawford
Assignee: Travis Crawford
Hive has a feature where {{Partition}}'s without any defined columns use the
{{Table}} schema. This happens in
{{[Partition.initialize|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L167}}
{code}
// set default if columns are not set
if (tPartition.getSd().getCols() == null) {
if (table.getCols() != null) {
tPartition.getSd().setCols(table.getCols());
}
}
{code}
There's an issue though, because
{{[Table.getEmptyTable|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java#L121]}}
initializes cols to an empty array, which of course is not null, causing the
above feature to not work as expected.
I'm not sure of the fix - is there a case where cols can indeed be null? I
think the best thing to do here is:
{code}
-if (tPartition.getSd().getCols() == null) {
+if (tPartition.getSd().getCols() == null ||
tPartition.getSd().getCols().size() == 0) {
{code}
Thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3279) Table schema not being copied to Partitions with no columns
[
https://issues.apache.org/jira/browse/HIVE-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Travis Crawford updated HIVE-3279:
--
Description:
Hive has a feature where {{Partition}}'s without any defined columns use the
{{Table}} schema. This happens in
{{[Partition.initialize|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L167]}}
{code}
// set default if columns are not set
if (tPartition.getSd().getCols() == null) {
if (table.getCols() != null) {
tPartition.getSd().setCols(table.getCols());
}
}
{code}
There's an issue though, because
{{[Table.getEmptyTable|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java#L121]}}
initializes cols to an empty array, which of course is not null, causing the
above feature to not work as expected.
I'm not sure of the fix - is there a case where cols can indeed be null? I
think the best thing to do here is:
{code}
-if (tPartition.getSd().getCols() == null) {
+if (tPartition.getSd().getCols() == null ||
tPartition.getSd().getCols().size() == 0) {
{code}
Thoughts?
was:
Hive has a feature where {{Partition}}'s without any defined columns use the
{{Table}} schema. This happens in
{{[Partition.initialize|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L167}}
{code}
// set default if columns are not set
if (tPartition.getSd().getCols() == null) {
if (table.getCols() != null) {
tPartition.getSd().setCols(table.getCols());
}
}
{code}
There's an issue though, because
{{[Table.getEmptyTable|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java#L121]}}
initializes cols to an empty array, which of course is not null, causing the
above feature to not work as expected.
I'm not sure of the fix - is there a case where cols can indeed be null? I
think the best thing to do here is:
{code}
-if (tPartition.getSd().getCols() == null) {
+if (tPartition.getSd().getCols() == null ||
tPartition.getSd().getCols().size() == 0) {
{code}
Thoughts?
Table schema not being copied to Partitions with no columns
---
Key: HIVE-3279
URL: https://issues.apache.org/jira/browse/HIVE-3279
Project: Hive
Issue Type: Bug
Reporter: Travis Crawford
Assignee: Travis Crawford
Hive has a feature where {{Partition}}'s without any defined columns use the
{{Table}} schema. This happens in
{{[Partition.initialize|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L167]}}
{code}
// set default if columns are not set
if (tPartition.getSd().getCols() == null) {
if (table.getCols() != null) {
tPartition.getSd().setCols(table.getCols());
}
}
{code}
There's an issue though, because
{{[Table.getEmptyTable|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java#L121]}}
initializes cols to an empty array, which of course is not null, causing the
above feature to not work as expected.
I'm not sure of the fix - is there a case where cols can indeed be null? I
think the best thing to do here is:
{code}
-if (tPartition.getSd().getCols() == null) {
+if (tPartition.getSd().getCols() == null ||
tPartition.getSd().getCols().size() == 0) {
{code}
Thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3280) Make HiveMetaStoreClient a public API
Carl Steinbach created HIVE-3280: Summary: Make HiveMetaStoreClient a public API Key: HIVE-3280 URL: https://issues.apache.org/jira/browse/HIVE-3280 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3280) Make HiveMetaStoreClient a public API
[ https://issues.apache.org/jira/browse/HIVE-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418693#comment-13418693 ] Carl Steinbach commented on HIVE-3280: -- At the moment HiveMetaStoreClient is treated as a de-facto public API. We should make this official and provide better documentation. Make HiveMetaStoreClient a public API - Key: HIVE-3280 URL: https://issues.apache.org/jira/browse/HIVE-3280 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Carl Steinbach Labels: api-addition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Is HiveMetastoreClient a public interface?
Sorry for the delay in responding. The java API is an ask from one of our users for Hcatalog. Given that HiveMetaStoreClient is a thin wrapper around public apis, then change shouldn't be drastic? It seems the cleaner way to go about it is to create an hcat client wrapper around HiveMetaStoreClient and have HCatClient as public? The problem with this proposal is that until HiveMetaStoreClient is a public API, HCatalog has just as little right to integrate with it directly as any other third-party. I think the best solution is to invest the time required to make HiveMetaStoreClient a public API, or to wrap HiveMetaStoreClient with HiveMetaStoreClient2 and make that a public API (we'll probably need to take the latter approach since HiveMetastoreClient currently exposes raw Thrift types). I filed HIVE-3280 to track this task. Thanks. Carl
[jira] [Updated] (HIVE-3126) Generate build the velocity based Hive tests on windows by fixing the path issues
[ https://issues.apache.org/jira/browse/HIVE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-3126: - Status: Open (was: Patch Available) @Kanna: I responded to your comments. Thanks. Generate build the velocity based Hive tests on windows by fixing the path issues --- Key: HIVE-3126 URL: https://issues.apache.org/jira/browse/HIVE-3126 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.9.0, 0.10.0, 0.9.1 Reporter: Kanna Karanam Assignee: Kanna Karanam Labels: Windows, test Fix For: 0.10.0 Attachments: HIVE-3126.1.patch.txt, HIVE-3126.2.patch.txt, HIVE-3126.3.patch.txt, HIVE-3126.4.patch.txt, HIVE-3126.5.patch.txt, HIVE-3126.6.patch.txt, HIVE-3126.7.patch.txt, HIVE-3126.8.patch.txt 1)Escape the backward slash in Canonical Path if unit test runs on windows. 2)Diff comparison – a. Ignore the extra spacing on windows b. Ignore the different line endings on windows Unix c. Convert the file paths to windows specific. (Handle spaces etc..) 3)Set the right file scheme class path separators while invoking the junit task from -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3278) Create wrappers around Thrift Metastore objects for improved readability
[ https://issues.apache.org/jira/browse/HIVE-3278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418733#comment-13418733 ] Travis Crawford commented on HIVE-3278: --- Hey [~cwsteinbach] - can you clarify what you mean by wrappers? There are wrappers already (for example https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java) and I'm curious what the distinction will be. FWIW, having two sets of identically named classes (stuff under metastore.api ql.metadata) has already caused some confusion. Typically this confusion has been people using the metastore.api versions not realizing ql.metadata adds some business logic. Create wrappers around Thrift Metastore objects for improved readability Key: HIVE-3278 URL: https://issues.apache.org/jira/browse/HIVE-3278 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Carl Steinbach Assignee: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3281) Alter table ignores table/partition not existing in semantic analysis
Kevin Wilfong created HIVE-3281: --- Summary: Alter table ignores table/partition not existing in semantic analysis Key: HIVE-3281 URL: https://issues.apache.org/jira/browse/HIVE-3281 Project: Hive Issue Type: Bug Components: Logging, Query Processor Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Priority: Minor In the method addInputsOutputsAlterTable in the DDLSemanticAnalyzer we get the table and partition being altered from the metastore. Failures to get these are quietly ignored, even though they resurface during query execution. We can throw a semantic exception here instead of ignoring them and save some time/resources. Also, by doing it here we save on retrieving the table/partition from the metastore since we're doing it here already. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3282) Convert runtime exceptions to semantic exceptions for missing partitions/tables in show/describe statements
Sambavi Muthukrishnan created HIVE-3282: --- Summary: Convert runtime exceptions to semantic exceptions for missing partitions/tables in show/describe statements Key: HIVE-3282 URL: https://issues.apache.org/jira/browse/HIVE-3282 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.9.0 Reporter: Sambavi Muthukrishnan Assignee: Sambavi Muthukrishnan Priority: Minor Fix For: 0.9.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3282) Convert runtime exceptions to semantic exceptions for missing partitions/tables in show/describe statements
[ https://issues.apache.org/jira/browse/HIVE-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sambavi Muthukrishnan updated HIVE-3282: Description: The SHOW PARTITIONS command in Hive does not check for valid table and partition names during query compilation. Calling this command with non-existent table causes a run-time exception. The DESC command also does not check for this in semantic analysis. hive desc xxxyyy; OK Table xxxyyy does not exist Time taken: 1.403 seconds hive show partitions xxxyyy; Table xxxyyy does not exist FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask Remaining Estimate: 24h Original Estimate: 24h Convert runtime exceptions to semantic exceptions for missing partitions/tables in show/describe statements --- Key: HIVE-3282 URL: https://issues.apache.org/jira/browse/HIVE-3282 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.9.0 Reporter: Sambavi Muthukrishnan Assignee: Sambavi Muthukrishnan Priority: Minor Fix For: 0.9.1 Original Estimate: 24h Remaining Estimate: 24h The SHOW PARTITIONS command in Hive does not check for valid table and partition names during query compilation. Calling this command with non-existent table causes a run-time exception. The DESC command also does not check for this in semantic analysis. hive desc xxxyyy; OK Table xxxyyy does not exist Time taken: 1.403 seconds hive show partitions xxxyyy; Table xxxyyy does not exist FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3126) Generate build the velocity based Hive tests on windows by fixing the path issues
[ https://issues.apache.org/jira/browse/HIVE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418816#comment-13418816 ] Kanna Karanam commented on HIVE-3126: - @Carl – I added some follow up comments. can you please look into them? Generate build the velocity based Hive tests on windows by fixing the path issues --- Key: HIVE-3126 URL: https://issues.apache.org/jira/browse/HIVE-3126 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.9.0, 0.10.0, 0.9.1 Reporter: Kanna Karanam Assignee: Kanna Karanam Labels: Windows, test Fix For: 0.10.0 Attachments: HIVE-3126.1.patch.txt, HIVE-3126.2.patch.txt, HIVE-3126.3.patch.txt, HIVE-3126.4.patch.txt, HIVE-3126.5.patch.txt, HIVE-3126.6.patch.txt, HIVE-3126.7.patch.txt, HIVE-3126.8.patch.txt 1)Escape the backward slash in Canonical Path if unit test runs on windows. 2)Diff comparison – a. Ignore the extra spacing on windows b. Ignore the different line endings on windows Unix c. Convert the file paths to windows specific. (Handle spaces etc..) 3)Set the right file scheme class path separators while invoking the junit task from -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3246) java primitive type for binary datatype should be byte[]
[ https://issues.apache.org/jira/browse/HIVE-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418835#comment-13418835 ] Hudson commented on HIVE-3246: -- Integrated in Hive-trunk-h0.21 #1553 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1553/]) HIVE-3246 : java primitive type for binary datatype should be byte[] (Thejas Nair via Ashutosh Chauhan) (Revision 1363427) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1363427 Files : * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantBinaryObjectInspector.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java java primitive type for binary datatype should be byte[] Key: HIVE-3246 URL: https://issues.apache.org/jira/browse/HIVE-3246 Project: Hive Issue Type: Bug Affects Versions: 0.9.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.10.0 Attachments: HIVE-3246.1.patch, HIVE-3246.2.patch PrimitiveObjectInspector.getPrimitiveJavaObject is supposed to return a java object. But in case of binary datatype, it returns ByteArrayRef (not java standard type). The suitable java object for it would be byte[]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3251) Hive doesn't remove scrach directories while killing running MR job
[ https://issues.apache.org/jira/browse/HIVE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418836#comment-13418836 ] Hudson commented on HIVE-3251: -- Integrated in Hive-trunk-h0.21 #1553 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1553/]) HIVE-3251 Hive doesn't remove scrach directories while killing running MR job (Gang Tim Liu via namit) (Revision 1363424) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1363424 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java Hive doesn't remove scrach directories while killing running MR job --- Key: HIVE-3251 URL: https://issues.apache.org/jira/browse/HIVE-3251 Project: Hive Issue Type: Bug Components: Server Infrastructure Reporter: Gang Tim Liu Assignee: Gang Tim Liu Attachments: HIVE-3151.patch While killing running MR job, hive doesn't clean up scratch directory (mapred.cache.files). So that, afterwards, scratch directory is left there in hdfs. HDFS name node doesn't know it and try to do lease recovery. while such instances happen more, it will eventually crash namenode. The fix is to leverage hdfs clean up functionality. While creating scratch dirs, hive registers it to hdfs cleanup hook. While killing happens, hdfs will clean them up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3267) escaped columns in cluster/distribute/order/sort by are not working
[ https://issues.apache.org/jira/browse/HIVE-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-3267: Component/s: Query Processor Affects Version/s: 0.10.0 Fix Version/s: 0.10.0 escaped columns in cluster/distribute/order/sort by are not working --- Key: HIVE-3267 URL: https://issues.apache.org/jira/browse/HIVE-3267 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Namit Jain Assignee: Namit Jain Fix For: 0.10.0 The following query: select `key`, value from src cluster by `key`, value; fails -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3267) escaped columns in cluster/distribute/order/sort by are not working
[ https://issues.apache.org/jira/browse/HIVE-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-3267: Resolution: Fixed Status: Resolved (was: Patch Available) Committed, thanks Namit. escaped columns in cluster/distribute/order/sort by are not working --- Key: HIVE-3267 URL: https://issues.apache.org/jira/browse/HIVE-3267 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Reporter: Namit Jain Assignee: Namit Jain Fix For: 0.10.0 The following query: select `key`, value from src cluster by `key`, value; fails -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3205) Bucketed mapjoin on partitioned table which has no partition throws NPE
[
https://issues.apache.org/jira/browse/HIVE-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Namit Jain updated HIVE-3205:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed. Thanks Navis
Bucketed mapjoin on partitioned table which has no partition throws NPE
---
Key: HIVE-3205
URL: https://issues.apache.org/jira/browse/HIVE-3205
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.10.0
Environment: ubuntu 10.04
Reporter: Navis
Assignee: Navis
Priority: Minor
{code}
create table hive_test_smb_bucket1 (key int, value string) partitioned by (ds
string) clustered by (key) sorted by (key) into 2 buckets;
create table hive_test_smb_bucket2 (key int, value string) partitioned by (ds
string) clustered by (key) sorted by (key) into 2 buckets;
set hive.optimize.bucketmapjoin = true;
set hive.input.format =
org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
explain
SELECT /* + MAPJOIN(b) */ b.key as k1, b.value, b.ds, a.key as k2
FROM hive_test_smb_bucket1 a JOIN
hive_test_smb_bucket2 b
ON a.key = b.key WHERE a.ds = '2010-10-15' and b.ds='2010-10-15' and b.key
IS NOT NULL;
{code}
throws NPE
{noformat}
2012-06-28 08:59:13,459 ERROR ql.Driver (SessionState.java:printError(400)) -
FAILED: NullPointerException null
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.optimizer.BucketMapJoinOptimizer$BucketMapjoinOptProc.process(BucketMapJoinOptimizer.java:269)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:125)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102)
at
org.apache.hadoop.hive.ql.optimizer.BucketMapJoinOptimizer.transform(BucketMapJoinOptimizer.java:100)
at
org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:87)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7564)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:245)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:50)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:245)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:744)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:607)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3283) bucket information should be used from the partition instead of the table
Namit Jain created HIVE-3283: Summary: bucket information should be used from the partition instead of the table Key: HIVE-3283 URL: https://issues.apache.org/jira/browse/HIVE-3283 Project: Hive Issue Type: Bug Reporter: Namit Jain Current Hive uses the number of buckets from the table object. Ideally, the number of buckets from the partition should be used -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3283) bucket information should be used from the partition instead of the table
[ https://issues.apache.org/jira/browse/HIVE-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3283: - Description: Currently Hive uses the number of buckets from the table object. Ideally, the number of buckets from the partition should be used was: Current Hive uses the number of buckets from the table object. Ideally, the number of buckets from the partition should be used bucket information should be used from the partition instead of the table - Key: HIVE-3283 URL: https://issues.apache.org/jira/browse/HIVE-3283 Project: Hive Issue Type: Bug Reporter: Namit Jain Currently Hive uses the number of buckets from the table object. Ideally, the number of buckets from the partition should be used -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3261) alter the number of buckets for a non-empty partitioned table should not be allowed
[ https://issues.apache.org/jira/browse/HIVE-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418914#comment-13418914 ] Namit Jain commented on HIVE-3261: -- comments addressed alter the number of buckets for a non-empty partitioned table should not be allowed --- Key: HIVE-3261 URL: https://issues.apache.org/jira/browse/HIVE-3261 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain Labels: bucketing This is dangerous since the code uses the table metadata everywhere to get the number of buckets -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3261) alter the number of buckets for a non-empty partitioned table should not be allowed
[ https://issues.apache.org/jira/browse/HIVE-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3261: - Status: Patch Available (was: Open) alter the number of buckets for a non-empty partitioned table should not be allowed --- Key: HIVE-3261 URL: https://issues.apache.org/jira/browse/HIVE-3261 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain Labels: bucketing This is dangerous since the code uses the table metadata everywhere to get the number of buckets -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3284) create ExecutionException(s)
Namit Jain created HIVE-3284: Summary: create ExecutionException(s) Key: HIVE-3284 URL: https://issues.apache.org/jira/browse/HIVE-3284 Project: Hive Issue Type: Bug Reporter: Namit Jain Hive currently contains SemanticExceptions for all compile time exceptions. It might be a good idea to create ExecutionExceptions to get the list of all execution exceptions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1653) Ability to enforce correct stats
[ https://issues.apache.org/jira/browse/HIVE-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-1653: - Status: Patch Available (was: Open) comments Ability to enforce correct stats Key: HIVE-1653 URL: https://issues.apache.org/jira/browse/HIVE-1653 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain This is a follow-up for https://issues.apache.org/jira/browse/HIVE-1361. If one of the mappers/reducers cannot publish stats, it may lead to wrong aggregated stats. There should be a way to avoid this - at the least, a configuration variable which fails the task if stats cannot be published -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
