[jira] [Commented] (HIVE-4975) Reading orc file throws exception after adding new column
[ https://issues.apache.org/jira/browse/HIVE-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13917817#comment-13917817 ] Kevin Wilfong commented on HIVE-4975: - The goal of this is just feature parity with other file formats, e.g. RC file. AFAIK, no formats in Hive handle reordering of columns, or swapping the names of columns (I'm assuming that's what you're worried about with regards to changing the name of a column). Reading orc file throws exception after adding new column - Key: HIVE-4975 URL: https://issues.apache.org/jira/browse/HIVE-4975 Project: Hive Issue Type: Bug Components: File Formats Affects Versions: 0.11.0 Environment: hive 0.11.0 hadoop 1.0.0 Reporter: cyril liao Assignee: Kevin Wilfong Priority: Critical Labels: orcfile Fix For: 0.13.0 Attachments: HIVE-4975.1.patch.txt ORC file read failure after add table column. create a table which have three column .(a string,b string,c string). add a new column after c by executing ALTER TABLE table ADD COLUMNS (d string). execute hiveql select d from table,the following exception goes: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row [Error getting row data with exception java.lang.ArrayIndexOutOfBoundsException: 4 at org.apache.hadoop.hive.ql.io.orc.OrcStruct$OrcStructInspector.getStructFieldData(OrcStruct.java:206) at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldData(UnionStructObjectInspector.java:128) at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371) at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:236) at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:222) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:665) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249) ] at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row [Error getting row data with exception java.lang.ArrayIndexOutOfBoundsException: 4 at org.apache.hadoop.hive.ql.io.orc.OrcStruct$OrcStructInspector.getStructFieldData(OrcStruct.java:206) at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldData(UnionStructObjectInspector.java:128) at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371) at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:236) at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:222) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:665) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249) ] at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:671) at
[jira] [Comment Edited] (HIVE-4975) Reading orc file throws exception after adding new column
[ https://issues.apache.org/jira/browse/HIVE-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13917817#comment-13917817 ] Kevin Wilfong edited comment on HIVE-4975 at 3/3/14 8:03 AM: - The goal of this patch is just feature parity with other file formats, e.g. RC file. AFAIK, no formats in Hive handle reordering of columns, or swapping the names of columns (I'm assuming that's what you're worried about with regards to changing the name of a column). was (Author: kevinwilfong): The goal of this is just feature parity with other file formats, e.g. RC file. AFAIK, no formats in Hive handle reordering of columns, or swapping the names of columns (I'm assuming that's what you're worried about with regards to changing the name of a column). Reading orc file throws exception after adding new column - Key: HIVE-4975 URL: https://issues.apache.org/jira/browse/HIVE-4975 Project: Hive Issue Type: Bug Components: File Formats Affects Versions: 0.11.0 Environment: hive 0.11.0 hadoop 1.0.0 Reporter: cyril liao Assignee: Kevin Wilfong Priority: Critical Labels: orcfile Fix For: 0.13.0 Attachments: HIVE-4975.1.patch.txt ORC file read failure after add table column. create a table which have three column .(a string,b string,c string). add a new column after c by executing ALTER TABLE table ADD COLUMNS (d string). execute hiveql select d from table,the following exception goes: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row [Error getting row data with exception java.lang.ArrayIndexOutOfBoundsException: 4 at org.apache.hadoop.hive.ql.io.orc.OrcStruct$OrcStructInspector.getStructFieldData(OrcStruct.java:206) at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldData(UnionStructObjectInspector.java:128) at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371) at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:236) at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:222) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:665) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249) ] at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row [Error getting row data with exception java.lang.ArrayIndexOutOfBoundsException: 4 at org.apache.hadoop.hive.ql.io.orc.OrcStruct$OrcStructInspector.getStructFieldData(OrcStruct.java:206) at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldData(UnionStructObjectInspector.java:128) at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371) at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:236) at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:222) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:665) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at
[jira] [Commented] (HIVE-3938) Hive MetaStore should send a single AddPartitionEvent for atomically added partition-set.
[
https://issues.apache.org/jira/browse/HIVE-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13917841#comment-13917841
]
Hive QA commented on HIVE-3938:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12632169/HIVE-3938.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5208 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1598/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1598/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12632169
Hive MetaStore should send a single AddPartitionEvent for atomically added
partition-set.
-
Key: HIVE-3938
URL: https://issues.apache.org/jira/browse/HIVE-3938
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.10.0, 0.11.0, 0.12.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Attachments: HIVE-3938.patch,
Hive-3938-Support_for_Multi-table-insert.patch
HiveMetaStore::add_partitions() currently adds all partitions specified in
one call using a single meta-store transaction. This acts correctly. However,
there's one AddPartitionEvent created per partition specified.
Ideally, the set of partitions added atomically can be communicated using a
single AddPartitionEvent, such that they are consumed together.
I'll post a patch that does this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5926) Load Data OverWrite Into Table Throw org.apache.hadoop.hive.ql.metadata.HiveException
[ https://issues.apache.org/jira/browse/HIVE-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Tian updated HIVE-5926: -- Resolution: Duplicate Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Load Data OverWrite Into Table Throw org.apache.hadoop.hive.ql.metadata.HiveException - Key: HIVE-5926 URL: https://issues.apache.org/jira/browse/HIVE-5926 Project: Hive Issue Type: Bug Components: Database/Schema Affects Versions: 0.12.0 Environment: OS: Red Hat Enterprise Linux Server release 6.2 HDFS: CDH-4.2.1 MAPRED: CDH-4.2.1-mr1 Reporter: Yi Tian Assignee: Yi Tian Fix For: 0.13.0 Attachments: HIVE-5926.patch step1: create table step2: load data load data inpath '/tianyi/usys_etl_map_total.del' overwrite into table tianyi_test3 step3: copy file back hadoop fs -cp /user/hive/warehouse/tianyi_test3/usys_etl_map_total.del /tianyi step4: load data again load data inpath '/tianyi/usys_etl_map_total.del' overwrite into table tianyi_test3 here we can see the error in console: Failed with exception Error moving: hdfs://ocdccluster/tianyi/usys_etl_map_total.del into: /user/hive/warehouse/tianyi_test3/usys_etl_map_total.del FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask we can find error detail in hive.log: 2013-12-03 17:26:41,717 ERROR exec.Task (SessionState.java:printError(419)) - Failed with exception Error moving: hdfs://ocdccluster/tianyi/usys_etl_map_total.del into: /user/hive/warehouse/tianyi_test3/usys_etl_map_total.del org.apache.hadoop.hive.ql.metadata.HiveException: Error moving: hdfs://ocdccluster/tianyi/usys_etl_map_total.del into: /user/hive/warehouse/tianyi_test3/usys_etl_map_total.del at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:2323) at org.apache.hadoop.hive.ql.metadata.Table.replaceFiles(Table.java:639) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1441) at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:283) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) Caused by: java.io.IOException: Error moving: hdfs://ocdccluster/tianyi/usys_etl_map_total.del into: /user/hive/warehouse/tianyi_test3/usys_etl_map_total.del at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:2317) ... 20 more 2013-12-03 17:26:41,718 ERROR ql.Driver (SessionState.java:printError(419)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6455) Scalable dynamic partitioning and bucketing optimization
[ https://issues.apache.org/jira/browse/HIVE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-6455: - Attachment: HIVE-6455.10.patch To kick-off precommit tests earlier I reuploaded .9 version instead of .10 version. Reuploading .10 version again to kick off another precommit test run. Scalable dynamic partitioning and bucketing optimization Key: HIVE-6455 URL: https://issues.apache.org/jira/browse/HIVE-6455 Project: Hive Issue Type: New Feature Components: Query Processor Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: optimization Attachments: HIVE-6455.1.patch, HIVE-6455.1.patch, HIVE-6455.10.patch, HIVE-6455.10.patch, HIVE-6455.2.patch, HIVE-6455.3.patch, HIVE-6455.4.patch, HIVE-6455.4.patch, HIVE-6455.5.patch, HIVE-6455.6.patch, HIVE-6455.7.patch, HIVE-6455.8.patch, HIVE-6455.9.patch, HIVE-6455.9.patch The current implementation of dynamic partition works by keeping at least one record writer open per dynamic partition directory. In case of bucketing there can be multispray file writers which further adds up to the number of open record writers. The record writers of column oriented file format (like ORC, RCFile etc.) keeps some sort of in-memory buffers (value buffer or compression buffers) open all the time to buffer up the rows and compress them before flushing it to disk. Since these buffers are maintained per column basis the amount of constant memory that will required at runtime increases as the number of partitions and number of columns per partition increases. This often leads to OutOfMemory (OOM) exception in mappers or reducers depending on the number of open record writers. Users often tune the JVM heapsize (runtime memory) to get over such OOM issues. With this optimization, the dynamic partition columns and bucketing columns (in case of bucketed tables) are sorted before being fed to the reducers. Since the partitioning and bucketing columns are sorted, each reducers can keep only one record writer open at any time thereby reducing the memory pressure on the reducers. This optimization is highly scalable as the number of partition and number of columns per partition increases at the cost of sorting the columns. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6530) JDK 7 trunk build fails after HIVE-6418 patch
[
https://issues.apache.org/jira/browse/HIVE-6530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13917912#comment-13917912
]
Hive QA commented on HIVE-6530:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12632172/HIVE-6530.2.patch.txt
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5208 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1599/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1599/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12632172
JDK 7 trunk build fails after HIVE-6418 patch
-
Key: HIVE-6530
URL: https://issues.apache.org/jira/browse/HIVE-6530
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.0
Reporter: Prasad Mujumdar
Assignee: Navis
Priority: Blocker
Attachments: HIVE-6530.1.patch.txt, HIVE-6530.2.patch.txt
JDK7 build fails with following error
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
on project hive-exec: Compilation failure
[ERROR]
/home/prasadm/repos/apache/hive-trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/LazyFlatRowContainer.java:[118,15]
name clash: add(java.util.Listjava.lang.Object) in
org.apache.hadoop.hive.ql.exec.persistence.LazyFlatRowContainer overrides a
method whose erasure is the same as another method, yet neither overrides the
other
[ERROR] first method: add(E) in java.util.AbstractCollection
[ERROR] second method: add(ROW) in
org.apache.hadoop.hive.ql.exec.persistence.AbstractRowContainer
[ERROR] - [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the
command
[ERROR] mvn goals -rf :hive-exec
{noformat}
This LazyFlatRowContainer.java is a new file added as part of HIVE-6418
patch. It's extending AbstractCollection and implements AbstractRowContainer.
Looks like the both these have a add() method that's conflicting.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-6535) jdbc async wait should happen during fetch for results
Thejas M Nair created HIVE-6535: --- Summary: jdbc async wait should happen during fetch for results Key: HIVE-6535 URL: https://issues.apache.org/jira/browse/HIVE-6535 Project: Hive Issue Type: Bug Reporter: Thejas M Nair The hive jdbc client waits query completion during execute() call. It would be better to block in the jdbc for completion when the results are being fetched. This way the application using hive jdbc driver can do other tasks while asynchronous query execution is happening, until it needs to fetch the result set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6535) jdbc async wait should happen during fetch for results
[ https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13917939#comment-13917939 ] Thejas M Nair commented on HIVE-6535: - See discussion in review of https://reviews.apache.org/r/14950/ jdbc async wait should happen during fetch for results -- Key: HIVE-6535 URL: https://issues.apache.org/jira/browse/HIVE-6535 Project: Hive Issue Type: Bug Reporter: Thejas M Nair The hive jdbc client waits query completion during execute() call. It would be better to block in the jdbc for completion when the results are being fetched. This way the application using hive jdbc driver can do other tasks while asynchronous query execution is happening, until it needs to fetch the result set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5232) Make JDBC use the new HiveServer2 async execution API by default
[ https://issues.apache.org/jira/browse/HIVE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-5232: Resolution: Fixed Status: Resolved (was: Patch Available) Created jira for moving the blocking from jdbc from execute to fetch-results - HIVE-6535 Patch committed to trunk. Thanks for the contribution Vaibhav! Make JDBC use the new HiveServer2 async execution API by default Key: HIVE-5232 URL: https://issues.apache.org/jira/browse/HIVE-5232 Project: Hive Issue Type: Sub-task Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Fix For: 0.13.0 Attachments: HIVE-5232.1.patch, HIVE-5232.2.patch, HIVE-5232.3.patch HIVE-4617 provides support for async execution in HS2. There are some proposed improvements in followup JIRAs: HIVE-5217 HIVE-5229 HIVE-5230 HIVE-5441 There is also [HIVE-5060] which assumes that execute to be asynchronous by default. Once they are in, we can think of using the async API as the default for JDBC. This can enable the server to report back error sooner to the client. It can also be useful in cases where a statement.cancel is done in a different thread - the original thread will now be able to detect the cancel, as opposed to the use of the blocking execute calls, in which statement.cancel will be a no-op. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6440) sql std auth - add command to change owner of database
[
https://issues.apache.org/jira/browse/HIVE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13917988#comment-13917988
]
Hive QA commented on HIVE-6440:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12631983/HIVE-6440.3.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5190 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_input16_cc
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1601/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1601/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12631983
sql std auth - add command to change owner of database
--
Key: HIVE-6440
URL: https://issues.apache.org/jira/browse/HIVE-6440
Project: Hive
Issue Type: Sub-task
Components: Authorization
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Attachments: HIVE-6440.1.patch, HIVE-6440.2.patch, HIVE-6440.3.patch
It should be possible to change the owner of a database once it is created.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-4598) Incorrect results when using subquery in multi table insert
[ https://issues.apache.org/jira/browse/HIVE-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13917990#comment-13917990 ] pandeeswaran commented on HIVE-4598: Is anyone working on this issue? I would like to fix this ,if no-one started already? Incorrect results when using subquery in multi table insert --- Key: HIVE-4598 URL: https://issues.apache.org/jira/browse/HIVE-4598 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0, 0.11.0 Reporter: Sebastian I'm using a multi table insert like this: FROM x INSERT INTO TABLE t PARTITION (type='x') SELECT * WHERE type='x' INSERT INTO TABLE t PARTITION (type='y') SELECT * WHERE type='y'; Now when x is the name of a table, everything works as expected. However if I use a subquery as x, the query runs but it inserts all results from the subquery into each partition, as if there were no WHERE clauses in the selects. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6440) sql std auth - add command to change owner of database
[ https://issues.apache.org/jira/browse/HIVE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6440: Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) The 2 test failures are unrelated. I verified that the tests pass locally. Patch committed to trunk. Thanks for the review Ashutosh! sql std auth - add command to change owner of database -- Key: HIVE-6440 URL: https://issues.apache.org/jira/browse/HIVE-6440 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.13.0 Attachments: HIVE-6440.1.patch, HIVE-6440.2.patch, HIVE-6440.3.patch It should be possible to change the owner of a database once it is created. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6455) Scalable dynamic partitioning and bucketing optimization
[
https://issues.apache.org/jira/browse/HIVE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918047#comment-13918047
]
Hive QA commented on HIVE-6455:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12632210/HIVE-6455.10.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 5209 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1603/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1603/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12632210
Scalable dynamic partitioning and bucketing optimization
Key: HIVE-6455
URL: https://issues.apache.org/jira/browse/HIVE-6455
Project: Hive
Issue Type: New Feature
Components: Query Processor
Affects Versions: 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
Labels: optimization
Attachments: HIVE-6455.1.patch, HIVE-6455.1.patch,
HIVE-6455.10.patch, HIVE-6455.10.patch, HIVE-6455.2.patch, HIVE-6455.3.patch,
HIVE-6455.4.patch, HIVE-6455.4.patch, HIVE-6455.5.patch, HIVE-6455.6.patch,
HIVE-6455.7.patch, HIVE-6455.8.patch, HIVE-6455.9.patch, HIVE-6455.9.patch
The current implementation of dynamic partition works by keeping at least one
record writer open per dynamic partition directory. In case of bucketing
there can be multispray file writers which further adds up to the number of
open record writers. The record writers of column oriented file format (like
ORC, RCFile etc.) keeps some sort of in-memory buffers (value buffer or
compression buffers) open all the time to buffer up the rows and compress
them before flushing it to disk. Since these buffers are maintained per
column basis the amount of constant memory that will required at runtime
increases as the number of partitions and number of columns per partition
increases. This often leads to OutOfMemory (OOM) exception in mappers or
reducers depending on the number of open record writers. Users often tune the
JVM heapsize (runtime memory) to get over such OOM issues.
With this optimization, the dynamic partition columns and bucketing columns
(in case of bucketed tables) are sorted before being fed to the reducers.
Since the partitioning and bucketing columns are sorted, each reducers can
keep only one record writer open at any time thereby reducing the memory
pressure on the reducers. This optimization is highly scalable as the number
of partition and number of columns per partition increases at the cost of
sorting the columns.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6427) Hive Server2 should reopen Metastore client in case of any Thrift exceptions
[ https://issues.apache.org/jira/browse/HIVE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HIVE-6427: --- Attachment: 0001-HDPDEV-53-HiveServer-should-reconnect-automatically-.patch Patch was updated. Hive Server2 should reopen Metastore client in case of any Thrift exceptions Key: HIVE-6427 URL: https://issues.apache.org/jira/browse/HIVE-6427 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.12.0 Environment: cloudera cdh5 beta2 Reporter: Andrey Stepachev Priority: Critical Attachments: 0001-HDPDEV-53-HiveServer-should-reconnect-automatically-.patch, 0001-HIVE-6427-Hive-Server2-should-reopen-Metastore-clien.patch In case of metastore restart hive server doesn't reopen connection to metastore. Any command gives broken pipe or similar exceptions. http://paste.ubuntu.com/6926215/ Any subsequent command doesn't reestablish connection and tries to use stale (closed) connection. Looks like we shouldn't blindly convert any MetaException to HiveSQLException, but should distinguish between fatal exceptions and logical exceptions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6427) Hive Server2 should reopen Metastore client in case of any Thrift exceptions
[ https://issues.apache.org/jira/browse/HIVE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HIVE-6427: --- Attachment: (was: 0001-HDPDEV-53-HiveServer-should-reconnect-automatically-.patch) Hive Server2 should reopen Metastore client in case of any Thrift exceptions Key: HIVE-6427 URL: https://issues.apache.org/jira/browse/HIVE-6427 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.12.0 Environment: cloudera cdh5 beta2 Reporter: Andrey Stepachev Priority: Critical In case of metastore restart hive server doesn't reopen connection to metastore. Any command gives broken pipe or similar exceptions. http://paste.ubuntu.com/6926215/ Any subsequent command doesn't reestablish connection and tries to use stale (closed) connection. Looks like we shouldn't blindly convert any MetaException to HiveSQLException, but should distinguish between fatal exceptions and logical exceptions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6427) Hive Server2 should reopen Metastore client in case of any Thrift exceptions
[ https://issues.apache.org/jira/browse/HIVE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HIVE-6427: --- Attachment: (was: 0001-HIVE-6427-Hive-Server2-should-reopen-Metastore-clien.patch) Hive Server2 should reopen Metastore client in case of any Thrift exceptions Key: HIVE-6427 URL: https://issues.apache.org/jira/browse/HIVE-6427 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.12.0 Environment: cloudera cdh5 beta2 Reporter: Andrey Stepachev Priority: Critical In case of metastore restart hive server doesn't reopen connection to metastore. Any command gives broken pipe or similar exceptions. http://paste.ubuntu.com/6926215/ Any subsequent command doesn't reestablish connection and tries to use stale (closed) connection. Looks like we shouldn't blindly convert any MetaException to HiveSQLException, but should distinguish between fatal exceptions and logical exceptions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6427) Hive Server2 should reopen Metastore client in case of any Thrift exceptions
[ https://issues.apache.org/jira/browse/HIVE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HIVE-6427: --- Attachment: HIVE-6427-2.patch Hive Server2 should reopen Metastore client in case of any Thrift exceptions Key: HIVE-6427 URL: https://issues.apache.org/jira/browse/HIVE-6427 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.12.0 Environment: cloudera cdh5 beta2 Reporter: Andrey Stepachev Priority: Critical Attachments: HIVE-6427-2.patch In case of metastore restart hive server doesn't reopen connection to metastore. Any command gives broken pipe or similar exceptions. http://paste.ubuntu.com/6926215/ Any subsequent command doesn't reestablish connection and tries to use stale (closed) connection. Looks like we shouldn't blindly convert any MetaException to HiveSQLException, but should distinguish between fatal exceptions and logical exceptions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-3938) Hive MetaStore should send a single AddPartitionEvent for atomically added partition-set.
[ https://issues.apache.org/jira/browse/HIVE-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918140#comment-13918140 ] Mithun Radhakrishnan commented on HIVE-3938: With the updated patch, TestNegativeMinimrCliDriver has a failure. This looks unrelated, but would someone please confirm? Hive MetaStore should send a single AddPartitionEvent for atomically added partition-set. - Key: HIVE-3938 URL: https://issues.apache.org/jira/browse/HIVE-3938 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0, 0.11.0, 0.12.0 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: HIVE-3938.patch, Hive-3938-Support_for_Multi-table-insert.patch HiveMetaStore::add_partitions() currently adds all partitions specified in one call using a single meta-store transaction. This acts correctly. However, there's one AddPartitionEvent created per partition specified. Ideally, the set of partitions added atomically can be communicated using a single AddPartitionEvent, such that they are consumed together. I'll post a patch that does this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6389) LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in null-maps.
[
https://issues.apache.org/jira/browse/HIVE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-6389:
---
Status: Open (was: Patch Available)
Pulling patch. Will capitalize and resubmit.
LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in
null-maps.
Key: HIVE-6389
URL: https://issues.apache.org/jira/browse/HIVE-6389
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.12.0, 0.11.0, 0.10.0, 0.13.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
RCFile tables that use the LazyBinaryColumnarSerDe don't seem to handle
look-ups into map-columns when the value of the column is null.
When an RCFile table is created with LazyBinaryColumnarSerDe (as is default
in 0.12), and queried as follows:
{code}
select mymap['1024'] from mytable;
{code}
and if the mymap column has nulls, then one is treated to the following
guttural utterance:
{code}
2014-02-05 21:50:25,050 FATAL mr.ExecMapper (ExecMapper.java:map(194)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {id:null,mymap:null,isnull:null}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to
org.apache.hadoop.io.Text
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveWritableObject(WritableStringObjectInspector.java:41)
at
org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:226)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:486)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:439)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:423)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:560)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
... 10 more
{code}
A patch is on the way, but the short of it is that the LazyBinaryMapOI needs
to return nulls if either the map or the lookup-key is null.
This is handled correctly for Text data, and for RCFiles using ColumnarSerDe.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918146#comment-13918146
]
Hive QA commented on HIVE-5504:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12631095/HIVE-5504.2.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5218 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1604/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1604/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12631095
OrcOutputFormat honors compression properties only from within hive
-
Key: HIVE-5504
URL: https://issues.apache.org/jira/browse/HIVE-5504
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.11.0, 0.12.0, 0.13.0
Reporter: Venkat Ranganathan
Assignee: Sushanth Sowmyan
Attachments: HIVE-5504.2.patch, HIVE-5504.patch
When we import data into a HCatalog table created with the following storage
description
.. stored as orc tblproperties (orc.compress=SNAPPY)
the resultant orc file still uses the default zlib compression
It looks like HCatOutputFormat is ignoring the tblproperties specified.
show tblproperties shows that the table indeed has the properties properly
saved.
An insert/select into the table has the resulting orc file honor the tbl
property.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6389) LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in null-maps.
[
https://issues.apache.org/jira/browse/HIVE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-6389:
---
Attachment: (was: Hive-6389.patch)
LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in
null-maps.
Key: HIVE-6389
URL: https://issues.apache.org/jira/browse/HIVE-6389
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.10.0, 0.11.0, 0.12.0, 0.13.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
RCFile tables that use the LazyBinaryColumnarSerDe don't seem to handle
look-ups into map-columns when the value of the column is null.
When an RCFile table is created with LazyBinaryColumnarSerDe (as is default
in 0.12), and queried as follows:
{code}
select mymap['1024'] from mytable;
{code}
and if the mymap column has nulls, then one is treated to the following
guttural utterance:
{code}
2014-02-05 21:50:25,050 FATAL mr.ExecMapper (ExecMapper.java:map(194)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {id:null,mymap:null,isnull:null}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to
org.apache.hadoop.io.Text
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveWritableObject(WritableStringObjectInspector.java:41)
at
org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:226)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:486)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:439)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:423)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:560)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
... 10 more
{code}
A patch is on the way, but the short of it is that the LazyBinaryMapOI needs
to return nulls if either the map or the lookup-key is null.
This is handled correctly for Text data, and for RCFiles using ColumnarSerDe.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6389) LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in null-maps.
[
https://issues.apache.org/jira/browse/HIVE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-6389:
---
Attachment: HIVE-6389.patch
Renamed.
LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in
null-maps.
Key: HIVE-6389
URL: https://issues.apache.org/jira/browse/HIVE-6389
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.10.0, 0.11.0, 0.12.0, 0.13.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Attachments: HIVE-6389.patch
RCFile tables that use the LazyBinaryColumnarSerDe don't seem to handle
look-ups into map-columns when the value of the column is null.
When an RCFile table is created with LazyBinaryColumnarSerDe (as is default
in 0.12), and queried as follows:
{code}
select mymap['1024'] from mytable;
{code}
and if the mymap column has nulls, then one is treated to the following
guttural utterance:
{code}
2014-02-05 21:50:25,050 FATAL mr.ExecMapper (ExecMapper.java:map(194)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {id:null,mymap:null,isnull:null}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to
org.apache.hadoop.io.Text
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveWritableObject(WritableStringObjectInspector.java:41)
at
org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:226)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:486)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:439)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:423)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:560)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
... 10 more
{code}
A patch is on the way, but the short of it is that the LazyBinaryMapOI needs
to return nulls if either the map or the lookup-key is null.
This is handled correctly for Text data, and for RCFiles using ColumnarSerDe.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6389) LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in null-maps.
[
https://issues.apache.org/jira/browse/HIVE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-6389:
---
Status: Patch Available (was: Open)
LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in
null-maps.
Key: HIVE-6389
URL: https://issues.apache.org/jira/browse/HIVE-6389
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.12.0, 0.11.0, 0.10.0, 0.13.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Attachments: HIVE-6389.patch
RCFile tables that use the LazyBinaryColumnarSerDe don't seem to handle
look-ups into map-columns when the value of the column is null.
When an RCFile table is created with LazyBinaryColumnarSerDe (as is default
in 0.12), and queried as follows:
{code}
select mymap['1024'] from mytable;
{code}
and if the mymap column has nulls, then one is treated to the following
guttural utterance:
{code}
2014-02-05 21:50:25,050 FATAL mr.ExecMapper (ExecMapper.java:map(194)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {id:null,mymap:null,isnull:null}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to
org.apache.hadoop.io.Text
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveWritableObject(WritableStringObjectInspector.java:41)
at
org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:226)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:486)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:439)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:423)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:560)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
... 10 more
{code}
A patch is on the way, but the short of it is that the LazyBinaryMapOI needs
to return nulls if either the map or the lookup-key is null.
This is handled correctly for Text data, and for RCFiles using ColumnarSerDe.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-4598) Incorrect results when using subquery in multi table insert
[ https://issues.apache.org/jira/browse/HIVE-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918168#comment-13918168 ] Ashutosh Chauhan commented on HIVE-4598: Go ahead [~pandeesh] Looks like no one is working on it. Incorrect results when using subquery in multi table insert --- Key: HIVE-4598 URL: https://issues.apache.org/jira/browse/HIVE-4598 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0, 0.11.0 Reporter: Sebastian I'm using a multi table insert like this: FROM x INSERT INTO TABLE t PARTITION (type='x') SELECT * WHERE type='x' INSERT INTO TABLE t PARTITION (type='y') SELECT * WHERE type='y'; Now when x is the name of a table, everything works as expected. However if I use a subquery as x, the query runs but it inserts all results from the subquery into each partition, as if there were no WHERE clauses in the selects. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-3938) Hive MetaStore should send a single AddPartitionEvent for atomically added partition-set.
[ https://issues.apache.org/jira/browse/HIVE-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-3938: --- Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Ya, that test case is flaky. Committed to trunk. Thanks, Mithun! Hive MetaStore should send a single AddPartitionEvent for atomically added partition-set. - Key: HIVE-3938 URL: https://issues.apache.org/jira/browse/HIVE-3938 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0, 0.11.0, 0.12.0 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Fix For: 0.13.0 Attachments: HIVE-3938.patch, Hive-3938-Support_for_Multi-table-insert.patch HiveMetaStore::add_partitions() currently adds all partitions specified in one call using a single meta-store transaction. This acts correctly. However, there's one AddPartitionEvent created per partition specified. Ideally, the set of partitions added atomically can be communicated using a single AddPartitionEvent, such that they are consumed together. I'll post a patch that does this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6505) Make stats optimizer more robust in presence of distinct clause
[ https://issues.apache.org/jira/browse/HIVE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6505: --- Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Reran the tests. They passed locally. Committed to trunk. Make stats optimizer more robust in presence of distinct clause --- Key: HIVE-6505 URL: https://issues.apache.org/jira/browse/HIVE-6505 Project: Hive Issue Type: Bug Components: Statistics Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.13.0 Attachments: HIVE-6505.1.patch, HIVE-6505.patch Currently it throws exceptions in few cases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6536) Reduce dependencies of org.apache.hive:hive-jdbc maven module
Kevin Minder created HIVE-6536: -- Summary: Reduce dependencies of org.apache.hive:hive-jdbc maven module Key: HIVE-6536 URL: https://issues.apache.org/jira/browse/HIVE-6536 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.12.0 Environment: org.apache.hive:hive-jdbc:jar:0.12.0 Reporter: Kevin Minder The Hive JDBC driver maven module requires a significant number of dependencies that are likely unnecessary and will result in bloating of consumers. Most of this is a result of the dependency on org.apache.hive:hive-cli. I have attached a portion of the output from mvn depedency:tree output for a client that depends on the org.apache.hive:hive-jdbc module. Note the extra 2.0.6.1-102 in the output is the result of our local build and publish to a local nexus repo. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6536) Reduce dependencies of org.apache.hive:hive-jdbc maven module
[ https://issues.apache.org/jira/browse/HIVE-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Minder updated HIVE-6536: --- Attachment: hive-jdbc-maven-dependencies.log Reduce dependencies of org.apache.hive:hive-jdbc maven module - Key: HIVE-6536 URL: https://issues.apache.org/jira/browse/HIVE-6536 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.12.0 Environment: org.apache.hive:hive-jdbc:jar:0.12.0 Reporter: Kevin Minder Attachments: hive-jdbc-maven-dependencies.log The Hive JDBC driver maven module requires a significant number of dependencies that are likely unnecessary and will result in bloating of consumers. Most of this is a result of the dependency on org.apache.hive:hive-cli. I have attached a portion of the output from mvn depedency:tree output for a client that depends on the org.apache.hive:hive-jdbc module. Note the extra 2.0.6.1-102 in the output is the result of our local build and publish to a local nexus repo. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6389) LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in null-maps.
[
https://issues.apache.org/jira/browse/HIVE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918277#comment-13918277
]
Hive QA commented on HIVE-6389:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12632273/HIVE-6389.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5218 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hive.hcatalog.mapreduce.TestHCatMutablePartitioned.testHCatPartitionedTable
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1605/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1605/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12632273
LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in
null-maps.
Key: HIVE-6389
URL: https://issues.apache.org/jira/browse/HIVE-6389
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.10.0, 0.11.0, 0.12.0, 0.13.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Attachments: HIVE-6389.patch
RCFile tables that use the LazyBinaryColumnarSerDe don't seem to handle
look-ups into map-columns when the value of the column is null.
When an RCFile table is created with LazyBinaryColumnarSerDe (as is default
in 0.12), and queried as follows:
{code}
select mymap['1024'] from mytable;
{code}
and if the mymap column has nulls, then one is treated to the following
guttural utterance:
{code}
2014-02-05 21:50:25,050 FATAL mr.ExecMapper (ExecMapper.java:map(194)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {id:null,mymap:null,isnull:null}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to
org.apache.hadoop.io.Text
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveWritableObject(WritableStringObjectInspector.java:41)
at
org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:226)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:486)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:439)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:423)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:560)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
... 10 more
{code}
A patch is on the way, but the short of it is that the LazyBinaryMapOI needs
to return nulls if either the map or the lookup-key is null.
This is handled correctly for Text data, and for RCFiles using ColumnarSerDe.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6511) casting from decimal to tinyint,smallint, int and bigint generates different result when vectorization is on
[
https://issues.apache.org/jira/browse/HIVE-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918290#comment-13918290
]
Eric Hanson commented on HIVE-6511:
---
Can you put this up on ReviewBoard?
casting from decimal to tinyint,smallint, int and bigint generates different
result when vectorization is on
Key: HIVE-6511
URL: https://issues.apache.org/jira/browse/HIVE-6511
Project: Hive
Issue Type: Bug
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
Attachments: HIVE-6511.1.patch
select dc,cast(dc as int), cast(dc as smallint),cast(dc as tinyint) from
vectortab10korc limit 20 generates following result when vectorization is
enabled:
{code}
4619756289662.078125 -1628520834 -16770 126
1553532646710.316406 -1245514442 -2762 54
3367942487288.360352 688127224 -776-8
4386447830839.337891 1286221623 12087 55
-3234165331139.458008 -54957251 27453 61
-488378613475.326172 1247658269 -16099 29
-493942492598.691406 -21253559 -19895 73
3101852523586.039062 886135874 23618 66
2544105595941.381836 1484956709 -23515 37
-3997512403067.0625 1102149509 30597 -123
-1183754978977.589355 1655994718 31070 94
1408783849655.676758 34576568-26440 -72
-2993175106993.426758 417098319 27215 79
3004723551798.100586 -1753555402 -8650 54
1103792083527.786133 -14511544 -28088 72
469767055288.485352 1615620024 26552 -72
-1263700791098.294434 -980406074 12486 -58
-4244889766496.484375 -1462078048 30112 -96
-3962729491139.782715 1525323068 -27332 60
NULL NULLNULLNULL
{code}
When vectorization is disabled, result looks like this:
{code}
4619756289662.078125 -1628520834 -16770 126
1553532646710.316406 -1245514442 -2762 54
3367942487288.360352 688127224 -776-8
4386447830839.337891 1286221623 12087 55
-3234165331139.458008 -54957251 27453 61
-488378613475.326172 1247658269 -16099 29
-493942492598.691406 -21253558 -19894 74
3101852523586.039062 886135874 23618 66
2544105595941.381836 1484956709 -23515 37
-3997512403067.0625 1102149509 30597 -123
-1183754978977.589355 1655994719 31071 95
1408783849655.676758 34576567-26441 -73
-2993175106993.426758 417098319 27215 79
3004723551798.100586 -1753555402 -8650 54
1103792083527.786133 -14511545 -28089 71
469767055288.485352 1615620024 26552 -72
-1263700791098.294434 -980406074 12486 -58
-4244889766496.484375 -1462078048 30112 -96
-3962729491139.782715 1525323069 -27331 61
NULL NULLNULLNULL
{code}
This issue is visible only for certain decimal values. In above example, row
7,11,12, and 15 generates different results.
vectortab10korc table schema:
{code}
t tinyint from deserializer
sismallintfrom deserializer
i int from deserializer
b bigint from deserializer
f float from deserializer
d double from deserializer
dcdecimal(38,18) from deserializer
boboolean from deserializer
s string from deserializer
s2string from deserializer
tstimestamp from deserializer
# Detailed Table Information
Database: default
Owner:xyz
CreateTime: Tue Feb 25 21:54:28 UTC 2014
LastAccessTime: UNKNOWN
Protect Mode: None
Retention:0
Location:
hdfs://host1.domain.com:8020/apps/hive/warehouse/vectortab10korc
Table Type: MANAGED_TABLE
Table Parameters:
COLUMN_STATS_ACCURATE true
numFiles1
numRows 1
rawDataSize 0
totalSize 344748
transient_lastDdlTime 1393365281
# Storage Information
SerDe Library:org.apache.hadoop.hive.ql.io.orc.OrcSerde
InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
OutputFormat:
[jira] [Commented] (HIVE-5950) ORC SARG creation fails with NPE for predicate conditions with decimal/date/char/varchar datatypes
[ https://issues.apache.org/jira/browse/HIVE-5950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918320#comment-13918320 ] Gunther Hagleitner commented on HIVE-5950: -- comments on rb ORC SARG creation fails with NPE for predicate conditions with decimal/date/char/varchar datatypes -- Key: HIVE-5950 URL: https://issues.apache.org/jira/browse/HIVE-5950 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile Attachments: HIVE-5950.1.patch, HIVE-5950.2.patch, HIVE-5950.3.patch, HIVE-5950.4.patch When decimal or date column is used, the type field in PredicateLeafImpl will be set to null. This will result in NPE during predicate leaf generation because of null dereferencing in hashcode computation. SARG creation should be extended to support/handle decimal and date data types. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5933) SQL std auth - add support to metastore api to list all privileges for a user
[ https://issues.apache.org/jira/browse/HIVE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-5933: Attachment: HIVE-5933.thriftapi.patch HIVE-5933.thriftapi.patch - thrift api that I plan to introduce. I also think it will be cleaner to use an api that returns RoleGrant for the show-role-grant instead of adding the RoleGrant information to Role (which becomes confusing in the context of create-role). I am thinking of making that change in a separate follow up patch for 0.13 . I am hoping we can include that during the stabilization phase of 0.13 (ie, post branching). cc [~navis] SQL std auth - add support to metastore api to list all privileges for a user - Key: HIVE-5933 URL: https://issues.apache.org/jira/browse/HIVE-5933 Project: Hive Issue Type: Sub-task Components: Authorization Reporter: Thejas M Nair Attachments: HIVE-5933.thriftapi.patch Original Estimate: 24h Remaining Estimate: 24h This is for supporting SHOW GRANTS statements - SHOW GRANTS; SHOW GRANTS FOR user; SHOW GRANTS FOR role; -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6389) LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in null-maps.
[
https://issues.apache.org/jira/browse/HIVE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-6389:
---
Resolution: Fixed
Fix Version/s: 0.13.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks, Mithun!
LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in
null-maps.
Key: HIVE-6389
URL: https://issues.apache.org/jira/browse/HIVE-6389
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.10.0, 0.11.0, 0.12.0, 0.13.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Fix For: 0.13.0
Attachments: HIVE-6389.patch
RCFile tables that use the LazyBinaryColumnarSerDe don't seem to handle
look-ups into map-columns when the value of the column is null.
When an RCFile table is created with LazyBinaryColumnarSerDe (as is default
in 0.12), and queried as follows:
{code}
select mymap['1024'] from mytable;
{code}
and if the mymap column has nulls, then one is treated to the following
guttural utterance:
{code}
2014-02-05 21:50:25,050 FATAL mr.ExecMapper (ExecMapper.java:map(194)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {id:null,mymap:null,isnull:null}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to
org.apache.hadoop.io.Text
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveWritableObject(WritableStringObjectInspector.java:41)
at
org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:226)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:486)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:439)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:423)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:560)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
... 10 more
{code}
A patch is on the way, but the short of it is that the LazyBinaryMapOI needs
to return nulls if either the map or the lookup-key is null.
This is handled correctly for Text data, and for RCFiles using ColumnarSerDe.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 13845: HIVE-5155: Support secure proxy user access to HiveServer2
On Feb. 23, 2014, 9:47 a.m., Lefty Leverenz wrote: conf/hive-default.xml.template, line 2111 https://reviews.apache.org/r/13845/diff/4/?file=394500#file394500line2111 Agreed on both points. But as typos go, requestion is such a lovely word that it ought to be added to the English language. ;) Lefty Leverenz wrote: Oops, thought this would appear with Thejas's comment (the points are altername - alternate and requestion - request). :) My linguistic creativity is inversely proportional to my blood caffeine level ... sorry about the typos. - Prasad --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13845/#review35241 --- On Dec. 5, 2013, 8:08 p.m., Prasad Mujumdar wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13845/ --- (Updated Dec. 5, 2013, 8:08 p.m.) Review request for hive, Brock Noland, Carl Steinbach, and Thejas Nair. Bugs: HIVE-5155 https://issues.apache.org/jira/browse/HIVE-5155 Repository: hive-git Description --- Delegation token support - Enable delegation token connection for HiveServer2 Enhance the TCLIService interface to support delegation token requests Support passing the delegation token connection type via JDBC URL and Beeline option Direct proxy access - Define new proxy user property Shim interfaces to validate proxy access for a given user Note that the diff doesn't include thrift generated code. Diffs - beeline/src/java/org/apache/hive/beeline/BeeLine.java c5e36a5 beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java c3abba3 beeline/src/java/org/apache/hive/beeline/Commands.java d2d7fd3 beeline/src/java/org/apache/hive/beeline/DatabaseConnection.java 1de5829 common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 36503fa conf/hive-default.xml.template c61a0bb itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 7b1c9da jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java d08e05b jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java ef39573 jdbc/src/java/org/apache/hive/jdbc/Utils.java 4d75d98 service/if/TCLIService.thrift 62a9730 service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java d80649f service/src/java/org/apache/hive/service/auth/KerberosSaslHelper.java 519556c service/src/java/org/apache/hive/service/auth/PlainSaslHelper.java 15b1675 service/src/java/org/apache/hive/service/cli/CLIService.java 8c85386 service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 14ef54f service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 9dca874 service/src/java/org/apache/hive/service/cli/ICLIService.java f647ce6 service/src/java/org/apache/hive/service/cli/session/HiveSession.java 00058cc service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java cfda752 service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java 708f4e4 service/src/java/org/apache/hive/service/cli/session/SessionManager.java e262b72 service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 9df110e service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 9bb2a0f service/src/test/org/apache/hive/service/auth/TestPlainSaslHelper.java 8fa4afd service/src/test/org/apache/hive/service/cli/session/TestSessionHooks.java 2fac800 shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 6ff1a84 shims/common-secure/src/main/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java 84f3ddc shims/common-secure/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java dc89de1 shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 0d5615c shims/common/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java 03f4e51 Diff: https://reviews.apache.org/r/13845/diff/ Testing --- Since this requires kerberos setup, its tested by a standalone test program that runs various existing and new secure connection scenarios. The test code is attached to the ticket at https://issues.apache.org/jira/secure/attachment/12600119/ProxyAuth.java Thanks, Prasad Mujumdar
[jira] [Commented] (HIVE-5155) Support secure proxy user access to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918341#comment-13918341 ] Prasad Mujumdar commented on HIVE-5155: --- [~thejas] I responded to your comments on the review board. Please let me your feedback. [~vaibhavgumashta] Thanks for the offer. I am working on rebase and testing the updated patch. Support secure proxy user access to HiveServer2 --- Key: HIVE-5155 URL: https://issues.apache.org/jira/browse/HIVE-5155 Project: Hive Issue Type: Improvement Components: Authentication, HiveServer2, JDBC Affects Versions: 0.12.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-5155-1-nothrift.patch, HIVE-5155-noThrift.2.patch, HIVE-5155-noThrift.4.patch, HIVE-5155-noThrift.5.patch, HIVE-5155-noThrift.6.patch, HIVE-5155.1.patch, HIVE-5155.2.patch, HIVE-5155.3.patch, ProxyAuth.java, ProxyAuth.out, TestKERBEROS_Hive_JDBC.java The HiveServer2 can authenticate a client using via Kerberos and impersonate the connecting user with underlying secure hadoop. This becomes a gateway for a remote client to access secure hadoop cluster. Now this works fine for when the client obtains Kerberos ticket and directly connects to HiveServer2. There's another big use case for middleware tools where the end user wants to access Hive via another server. For example Oozie action or Hue submitting queries or a BI tool server accessing to HiveServer2. In these cases, the third party server doesn't have end user's Kerberos credentials and hence it can't submit queries to HiveServer2 on behalf of the end user. This ticket is for enabling proxy access to HiveServer2 for third party tools on behalf of end users. There are two parts of the solution proposed in this ticket: 1) Delegation token based connection for Oozie (OOZIE-1457) This is the common mechanism for Hadoop ecosystem components. Hive Remote Metastore and HCatalog already support this. This is suitable for tool like Oozie that submits the MR jobs as actions on behalf of its client. Oozie already uses similar mechanism for Metastore/HCatalog access. 2) Direct proxy access for privileged hadoop users The delegation token implementation can be a challenge for non-hadoop (especially non-java) components. This second part enables a privileged user to directly specify an alternate session user during the connection. If the connecting user has hadoop level privilege to impersonate the requested userid, then HiveServer2 will run the session as that requested user. For example, user Hue is allowed to impersonate user Bob (via core-site.xml proxy user configuration). Then user Hue can connect to HiveServer2 and specify Bob as session user via a session property. HiveServer2 will verify Hue's proxy user privilege and then impersonate user Bob instead of Hue. This will enable any third party tool to impersonate alternate userid without having to implement delegation token connection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6499) Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe
[ https://issues.apache.org/jira/browse/HIVE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-6499: --- Status: Patch Available (was: Open) Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe -- Key: HIVE-6499 URL: https://issues.apache.org/jira/browse/HIVE-6499 Project: Hive Issue Type: Bug Components: Metastore, Security Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: HIVE-6499.patch In cases where a user needs to use a custom IF/OF/SerDe that is not accessible from the metastore, calls like msc.createTable and msc.dropTable should still work without being able to load the class. This is possible as long as one does not enable MetaStore-side authorization, at which point this becomes impossible, erroring out with a ClassNotFoundException. The reason this happens is that since the AuthorizationProvider interface is defined against a ql.metadata.Table, we wind up needing to instantiate a ql.metadata.Table object, which, in its constructor tries to instantiate IF/OF/SerDe elements in an attempt to pre-load those fields. And if we do not have access to those classes in the metastore, this is when that fails. The constructor/initialize methods of Table and Partition do not really need to pre-initialize these fields, since the fields are accessed only through the accessor, and will be instantiated on first-use. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6499) Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe
[ https://issues.apache.org/jira/browse/HIVE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-6499: --- Status: Open (was: Patch Available) Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe -- Key: HIVE-6499 URL: https://issues.apache.org/jira/browse/HIVE-6499 Project: Hive Issue Type: Bug Components: Metastore, Security Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: HIVE-6499.patch In cases where a user needs to use a custom IF/OF/SerDe that is not accessible from the metastore, calls like msc.createTable and msc.dropTable should still work without being able to load the class. This is possible as long as one does not enable MetaStore-side authorization, at which point this becomes impossible, erroring out with a ClassNotFoundException. The reason this happens is that since the AuthorizationProvider interface is defined against a ql.metadata.Table, we wind up needing to instantiate a ql.metadata.Table object, which, in its constructor tries to instantiate IF/OF/SerDe elements in an attempt to pre-load those fields. And if we do not have access to those classes in the metastore, this is when that fails. The constructor/initialize methods of Table and Partition do not really need to pre-initialize these fields, since the fields are accessed only through the accessor, and will be instantiated on first-use. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[ https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-5504: --- Status: Patch Available (was: Open) OrcOutputFormat honors compression properties only from within hive - Key: HIVE-5504 URL: https://issues.apache.org/jira/browse/HIVE-5504 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.12.0, 0.11.0, 0.13.0 Reporter: Venkat Ranganathan Assignee: Sushanth Sowmyan Attachments: HIVE-5504.2.patch, HIVE-5504.patch When we import data into a HCatalog table created with the following storage description .. stored as orc tblproperties (orc.compress=SNAPPY) the resultant orc file still uses the default zlib compression It looks like HCatOutputFormat is ignoring the tblproperties specified. show tblproperties shows that the table indeed has the properties properly saved. An insert/select into the table has the resulting orc file honor the tbl property. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[ https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-5504: --- Status: Open (was: Patch Available) OrcOutputFormat honors compression properties only from within hive - Key: HIVE-5504 URL: https://issues.apache.org/jira/browse/HIVE-5504 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.12.0, 0.11.0, 0.13.0 Reporter: Venkat Ranganathan Assignee: Sushanth Sowmyan Attachments: HIVE-5504.2.patch, HIVE-5504.patch When we import data into a HCatalog table created with the following storage description .. stored as orc tblproperties (orc.compress=SNAPPY) the resultant orc file still uses the default zlib compression It looks like HCatOutputFormat is ignoring the tblproperties specified. show tblproperties shows that the table indeed has the properties properly saved. An insert/select into the table has the resulting orc file honor the tbl property. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6530) JDK 7 trunk build fails after HIVE-6418 patch
[
https://issues.apache.org/jira/browse/HIVE-6530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918389#comment-13918389
]
Prasad Mujumdar commented on HIVE-6530:
---
+1
[~navis] Thanks for taking care of it! I guess test failures are unrelated ..
JDK 7 trunk build fails after HIVE-6418 patch
-
Key: HIVE-6530
URL: https://issues.apache.org/jira/browse/HIVE-6530
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.0
Reporter: Prasad Mujumdar
Assignee: Navis
Priority: Blocker
Attachments: HIVE-6530.1.patch.txt, HIVE-6530.2.patch.txt
JDK7 build fails with following error
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
on project hive-exec: Compilation failure
[ERROR]
/home/prasadm/repos/apache/hive-trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/LazyFlatRowContainer.java:[118,15]
name clash: add(java.util.Listjava.lang.Object) in
org.apache.hadoop.hive.ql.exec.persistence.LazyFlatRowContainer overrides a
method whose erasure is the same as another method, yet neither overrides the
other
[ERROR] first method: add(E) in java.util.AbstractCollection
[ERROR] second method: add(ROW) in
org.apache.hadoop.hive.ql.exec.persistence.AbstractRowContainer
[ERROR] - [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the
command
[ERROR] mvn goals -rf :hive-exec
{noformat}
This LazyFlatRowContainer.java is a new file added as part of HIVE-6418
patch. It's extending AbstractCollection and implements AbstractRowContainer.
Looks like the both these have a add() method that's conflicting.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6486) Support secure Subject.doAs() in HiveServer2 JDBC client.
[ https://issues.apache.org/jira/browse/HIVE-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaraju Gowda updated HIVE-6486: -- Component/s: HiveServer2 Authentication Support secure Subject.doAs() in HiveServer2 JDBC client. - Key: HIVE-6486 URL: https://issues.apache.org/jira/browse/HIVE-6486 Project: Hive Issue Type: Improvement Components: Authentication, HiveServer2, JDBC Affects Versions: 0.11.0, 0.12.0 Reporter: Shivaraju Gowda Fix For: 0.13.0 Attachments: HIVE-6486.1.patch, Hive_011_Support-Subject_doAS.patch, TestHive_SujectDoAs.java HIVE-5155 addresses the problem of kerberos authentication in multi-user middleware server using proxy user. In this mode the principal used by the middle ware server has privileges to impersonate selected users in Hive/Hadoop. This enhancement is to support Subject.doAs() authentication in Hive JDBC layer so that the end users Kerberos Subject is passed through in the middle ware server. With this improvement there won't be any additional setup in the server to grant proxy privileges to some users and there won't be need to specify a proxy user in the JDBC client. This version should also be more secure since it won't require principals with the privileges to impersonate other users in Hive/Hadoop setup. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[ https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918392#comment-13918392 ] Sushanth Sowmyan commented on HIVE-5504: Looks like the tests finally ran! :) And the error reported by the pre-commit tests do not seem related to this patch - the table in question that's failing on that test is not even created as an orc table. OrcOutputFormat honors compression properties only from within hive - Key: HIVE-5504 URL: https://issues.apache.org/jira/browse/HIVE-5504 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Venkat Ranganathan Assignee: Sushanth Sowmyan Attachments: HIVE-5504.2.patch, HIVE-5504.patch When we import data into a HCatalog table created with the following storage description .. stored as orc tblproperties (orc.compress=SNAPPY) the resultant orc file still uses the default zlib compression It looks like HCatOutputFormat is ignoring the tblproperties specified. show tblproperties shows that the table indeed has the properties properly saved. An insert/select into the table has the resulting orc file honor the tbl property. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 13845: HIVE-5155: Support secure proxy user access to HiveServer2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13845/#review36000 --- shims/common-secure/src/main/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java https://reviews.apache.org/r/13845/#comment66789 Good point about adding this setting only in hive-site.xml, that way this privilege will be specific only to hive. On other hand, all hdfs/mr level proxy users will end up getting privileges in hive as well. But I think that is OK, as they already would be privileged users at webhdfs level etc. - Thejas Nair On Dec. 5, 2013, 8:08 p.m., Prasad Mujumdar wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13845/ --- (Updated Dec. 5, 2013, 8:08 p.m.) Review request for hive, Brock Noland, Carl Steinbach, and Thejas Nair. Bugs: HIVE-5155 https://issues.apache.org/jira/browse/HIVE-5155 Repository: hive-git Description --- Delegation token support - Enable delegation token connection for HiveServer2 Enhance the TCLIService interface to support delegation token requests Support passing the delegation token connection type via JDBC URL and Beeline option Direct proxy access - Define new proxy user property Shim interfaces to validate proxy access for a given user Note that the diff doesn't include thrift generated code. Diffs - beeline/src/java/org/apache/hive/beeline/BeeLine.java c5e36a5 beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java c3abba3 beeline/src/java/org/apache/hive/beeline/Commands.java d2d7fd3 beeline/src/java/org/apache/hive/beeline/DatabaseConnection.java 1de5829 common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 36503fa conf/hive-default.xml.template c61a0bb itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 7b1c9da jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java d08e05b jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java ef39573 jdbc/src/java/org/apache/hive/jdbc/Utils.java 4d75d98 service/if/TCLIService.thrift 62a9730 service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java d80649f service/src/java/org/apache/hive/service/auth/KerberosSaslHelper.java 519556c service/src/java/org/apache/hive/service/auth/PlainSaslHelper.java 15b1675 service/src/java/org/apache/hive/service/cli/CLIService.java 8c85386 service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 14ef54f service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 9dca874 service/src/java/org/apache/hive/service/cli/ICLIService.java f647ce6 service/src/java/org/apache/hive/service/cli/session/HiveSession.java 00058cc service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java cfda752 service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java 708f4e4 service/src/java/org/apache/hive/service/cli/session/SessionManager.java e262b72 service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 9df110e service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 9bb2a0f service/src/test/org/apache/hive/service/auth/TestPlainSaslHelper.java 8fa4afd service/src/test/org/apache/hive/service/cli/session/TestSessionHooks.java 2fac800 shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 6ff1a84 shims/common-secure/src/main/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java 84f3ddc shims/common-secure/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java dc89de1 shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 0d5615c shims/common/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java 03f4e51 Diff: https://reviews.apache.org/r/13845/diff/ Testing --- Since this requires kerberos setup, its tested by a standalone test program that runs various existing and new secure connection scenarios. The test code is attached to the ticket at https://issues.apache.org/jira/secure/attachment/12600119/ProxyAuth.java Thanks, Prasad Mujumdar
[jira] [Commented] (HIVE-5155) Support secure proxy user access to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918410#comment-13918410 ] Thejas M Nair commented on HIVE-5155: - [~prasadm] Yes, I think putting this in hive-site.xml is a reasonable way to restrict proxy user privileges to hs2. [~owen.omalley] Please refer to the conversation in reviewboard about the config settings. Support secure proxy user access to HiveServer2 --- Key: HIVE-5155 URL: https://issues.apache.org/jira/browse/HIVE-5155 Project: Hive Issue Type: Improvement Components: Authentication, HiveServer2, JDBC Affects Versions: 0.12.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-5155-1-nothrift.patch, HIVE-5155-noThrift.2.patch, HIVE-5155-noThrift.4.patch, HIVE-5155-noThrift.5.patch, HIVE-5155-noThrift.6.patch, HIVE-5155.1.patch, HIVE-5155.2.patch, HIVE-5155.3.patch, ProxyAuth.java, ProxyAuth.out, TestKERBEROS_Hive_JDBC.java The HiveServer2 can authenticate a client using via Kerberos and impersonate the connecting user with underlying secure hadoop. This becomes a gateway for a remote client to access secure hadoop cluster. Now this works fine for when the client obtains Kerberos ticket and directly connects to HiveServer2. There's another big use case for middleware tools where the end user wants to access Hive via another server. For example Oozie action or Hue submitting queries or a BI tool server accessing to HiveServer2. In these cases, the third party server doesn't have end user's Kerberos credentials and hence it can't submit queries to HiveServer2 on behalf of the end user. This ticket is for enabling proxy access to HiveServer2 for third party tools on behalf of end users. There are two parts of the solution proposed in this ticket: 1) Delegation token based connection for Oozie (OOZIE-1457) This is the common mechanism for Hadoop ecosystem components. Hive Remote Metastore and HCatalog already support this. This is suitable for tool like Oozie that submits the MR jobs as actions on behalf of its client. Oozie already uses similar mechanism for Metastore/HCatalog access. 2) Direct proxy access for privileged hadoop users The delegation token implementation can be a challenge for non-hadoop (especially non-java) components. This second part enables a privileged user to directly specify an alternate session user during the connection. If the connecting user has hadoop level privilege to impersonate the requested userid, then HiveServer2 will run the session as that requested user. For example, user Hue is allowed to impersonate user Bob (via core-site.xml proxy user configuration). Then user Hue can connect to HiveServer2 and specify Bob as session user via a session property. HiveServer2 will verify Hue's proxy user privilege and then impersonate user Bob instead of Hue. This will enable any third party tool to impersonate alternate userid without having to implement delegation token connection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5193) Columnar Pushdown for RC/ORC File not happening in HCatLoader
[ https://issues.apache.org/jira/browse/HIVE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918416#comment-13918416 ] Sushanth Sowmyan commented on HIVE-5193: Thanks for the update, viraj. I'm still reviewing this patch, and will be done by EOD today. Since the precommit tests picked it up, and the errors being reported do not seem related, if I have no further feedback request for this patch, I will commit it once I'm done. Columnar Pushdown for RC/ORC File not happening in HCatLoader -- Key: HIVE-5193 URL: https://issues.apache.org/jira/browse/HIVE-5193 Project: Hive Issue Type: Improvement Components: HCatalog Affects Versions: 0.10.0, 0.11.0, 0.12.0 Reporter: Viraj Bhat Assignee: Viraj Bhat Labels: hcatalog Fix For: 0.13.0 Attachments: HIVE-5193.2.patch, HIVE-5193.patch Currently the HCatLoader is not taking advantage of the ColumnProjectionUtils. where it could skip columns during read. The information is available in Pig it just needs to get to the Readers. Viraj -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[ https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-5504: Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Patch committed to trunk. Thanks for the contribution [~sushanth] OrcOutputFormat honors compression properties only from within hive - Key: HIVE-5504 URL: https://issues.apache.org/jira/browse/HIVE-5504 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Venkat Ranganathan Assignee: Sushanth Sowmyan Fix For: 0.13.0 Attachments: HIVE-5504.2.patch, HIVE-5504.patch When we import data into a HCatalog table created with the following storage description .. stored as orc tblproperties (orc.compress=SNAPPY) the resultant orc file still uses the default zlib compression It looks like HCatOutputFormat is ignoring the tblproperties specified. show tblproperties shows that the table indeed has the properties properly saved. An insert/select into the table has the resulting orc file honor the tbl property. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6537) NullPointerException when loading hashtable for MapJoin directly
Sergey Shelukhin created HIVE-6537:
--
Summary: NullPointerException when loading hashtable for MapJoin
directly
Key: HIVE-6537
URL: https://issues.apache.org/jira/browse/HIVE-6537
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
We see the following error:
{noformat}
2014-02-20 23:33:15,743 FATAL [main]
org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:103)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:149)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:164)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1026)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.NullPointerException
at java.util.Arrays.fill(Arrays.java:2685)
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.loadDirectly(HashTableLoader.java:155)
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:81)
... 15 more
{noformat}
It appears that the tables in Arrays.fill call is nulls. I don't really have
full understanding of this path, but what I gleaned so far is this...
From what I see, tables would be set unconditionally in initializeOp of the
sink, and in no other place, so I assume for this code to ever work that
startForward calls it at least some time.
Here, it doesn't call it, so it's null.
Previous loop also uses tables, and should have NPE-d before fill was ever
called; it didn't, so I'd assume it never executed.
There's a little bit of inconsistency in the above code where directWorks are
added to parents unconditionally but sink is only added as child conditionally.
I think it may be that some of the direct works are not table scans; in fact
given that loop never executes they may be null (which is rather strange).
Regardless, it seems that the logic should be fixed, it may be the root cause
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6409) FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.
[
https://issues.apache.org/jira/browse/HIVE-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918424#comment-13918424
]
Sushanth Sowmyan commented on HIVE-6409:
Committed (with some minor indenting fixes).
Thanks, Mithun!
FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.
--
Key: HIVE-6409
URL: https://issues.apache.org/jira/browse/HIVE-6409
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.12.0, 0.13.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Attachments: HIVE-6409.patch
When HCatalog's FileOutputCommitterContainer::commitJob() is run, it calls
the underlying OutputCommitter and then attempts to register partitions in
HCatalog.
If the commit fails (for example, because of HIVE-4996), commitJob() cancels
delegation tokens retrieved from HCatalog before the exception is rethrown.
{code}
java.io.IOException: java.lang.reflect.InvocationTargetException
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:183)
... 5 more
Caused by: org.apache.hcatalog.common.HCatException : 2006 : Error adding
partition to metastore. Cause :
MetaException(message:java.lang.RuntimeException: commitTransaction was called
but openTransactionCalls = 0. This probably indicates that there are
unbalanced
calls to openTransaction/commitTransaction)
at
org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:712)
{code}
The problem is that this happens before abortJob() has had a chance to run,
thus yielding the following error:
{code}
MetaException(message:Could not connect to meta store using any of the URIs
provided. Most recent failure:
org.apache.thrift.transport.TTransportException: Peer indicated failure:
DIGEST-MD5: IO error acquiring password
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
...
{code}
I'll have a patch out that only cancels delegation tokens if the commitJob()
has succeeded.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6409) FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.
[
https://issues.apache.org/jira/browse/HIVE-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918414#comment-13918414
]
Sushanth Sowmyan commented on HIVE-6409:
I'm +1 on this new patch, I'll go ahead and commit it.
As to your other concerns, this is per HIVE-5274, and per discussions that I
brought up in the next hive-dev meetup(after that patch) as well.
As to visibly breaking code, we will do that, as of 0.14, which will be trunk
in a couple more days as soon as we branch out for 0.13 - we have HIVE-6432
open for just that. The idea is that org.apache.hcatalog.* works predictably in
the way it has for all of those that have already been using it and find it
usable in the state it is in, whereas to all new users who may be evaluating,
they must be moving to org.apache.hive.hcatalog.
FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.
--
Key: HIVE-6409
URL: https://issues.apache.org/jira/browse/HIVE-6409
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.12.0, 0.13.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Attachments: HIVE-6409.patch
When HCatalog's FileOutputCommitterContainer::commitJob() is run, it calls
the underlying OutputCommitter and then attempts to register partitions in
HCatalog.
If the commit fails (for example, because of HIVE-4996), commitJob() cancels
delegation tokens retrieved from HCatalog before the exception is rethrown.
{code}
java.io.IOException: java.lang.reflect.InvocationTargetException
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:183)
... 5 more
Caused by: org.apache.hcatalog.common.HCatException : 2006 : Error adding
partition to metastore. Cause :
MetaException(message:java.lang.RuntimeException: commitTransaction was called
but openTransactionCalls = 0. This probably indicates that there are
unbalanced
calls to openTransaction/commitTransaction)
at
org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:712)
{code}
The problem is that this happens before abortJob() has had a chance to run,
thus yielding the following error:
{code}
MetaException(message:Could not connect to meta store using any of the URIs
provided. Most recent failure:
org.apache.thrift.transport.TTransportException: Peer indicated failure:
DIGEST-MD5: IO error acquiring password
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
...
{code}
I'll have a patch out that only cancels delegation tokens if the commitJob()
has succeeded.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6537) NullPointerException when loading hashtable for MapJoin directly
[
https://issues.apache.org/jira/browse/HIVE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-6537:
---
Attachment: HIVE-6537.patch
NullPointerException when loading hashtable for MapJoin directly
Key: HIVE-6537
URL: https://issues.apache.org/jira/browse/HIVE-6537
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
We see the following error:
{noformat}
2014-02-20 23:33:15,743 FATAL [main]
org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:103)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:149)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:164)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1026)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.NullPointerException
at java.util.Arrays.fill(Arrays.java:2685)
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.loadDirectly(HashTableLoader.java:155)
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:81)
... 15 more
{noformat}
It appears that the tables in Arrays.fill call is nulls. I don't really have
full understanding of this path, but what I gleaned so far is this...
From what I see, tables would be set unconditionally in initializeOp of the
sink, and in no other place, so I assume for this code to ever work that
startForward calls it at least some time.
Here, it doesn't call it, so it's null.
Previous loop also uses tables, and should have NPE-d before fill was ever
called; it didn't, so I'd assume it never executed.
There's a little bit of inconsistency in the above code where directWorks are
added to parents unconditionally but sink is only added as child
conditionally. I think it may be that some of the direct works are not table
scans; in fact given that loop never executes they may be null (which is
rather strange).
Regardless, it seems that the logic should be fixed, it may be the root cause
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6537) NullPointerException when loading hashtable for MapJoin directly
[
https://issues.apache.org/jira/browse/HIVE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-6537:
---
Attachment: (was: HIVE-6537.patch)
NullPointerException when loading hashtable for MapJoin directly
Key: HIVE-6537
URL: https://issues.apache.org/jira/browse/HIVE-6537
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
We see the following error:
{noformat}
2014-02-20 23:33:15,743 FATAL [main]
org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:103)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:149)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:164)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1026)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.NullPointerException
at java.util.Arrays.fill(Arrays.java:2685)
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.loadDirectly(HashTableLoader.java:155)
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:81)
... 15 more
{noformat}
It appears that the tables in Arrays.fill call is nulls. I don't really have
full understanding of this path, but what I gleaned so far is this...
From what I see, tables would be set unconditionally in initializeOp of the
sink, and in no other place, so I assume for this code to ever work that
startForward calls it at least some time.
Here, it doesn't call it, so it's null.
Previous loop also uses tables, and should have NPE-d before fill was ever
called; it didn't, so I'd assume it never executed.
There's a little bit of inconsistency in the above code where directWorks are
added to parents unconditionally but sink is only added as child
conditionally. I think it may be that some of the direct works are not table
scans; in fact given that loop never executes they may be null (which is
rather strange).
Regardless, it seems that the logic should be fixed, it may be the root cause
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5155) Support secure proxy user access to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918428#comment-13918428 ] Vaibhav Gumashta commented on HIVE-5155: [~prasadm] Thanks a lot for the heads up Prasad! Support secure proxy user access to HiveServer2 --- Key: HIVE-5155 URL: https://issues.apache.org/jira/browse/HIVE-5155 Project: Hive Issue Type: Improvement Components: Authentication, HiveServer2, JDBC Affects Versions: 0.12.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-5155-1-nothrift.patch, HIVE-5155-noThrift.2.patch, HIVE-5155-noThrift.4.patch, HIVE-5155-noThrift.5.patch, HIVE-5155-noThrift.6.patch, HIVE-5155.1.patch, HIVE-5155.2.patch, HIVE-5155.3.patch, ProxyAuth.java, ProxyAuth.out, TestKERBEROS_Hive_JDBC.java The HiveServer2 can authenticate a client using via Kerberos and impersonate the connecting user with underlying secure hadoop. This becomes a gateway for a remote client to access secure hadoop cluster. Now this works fine for when the client obtains Kerberos ticket and directly connects to HiveServer2. There's another big use case for middleware tools where the end user wants to access Hive via another server. For example Oozie action or Hue submitting queries or a BI tool server accessing to HiveServer2. In these cases, the third party server doesn't have end user's Kerberos credentials and hence it can't submit queries to HiveServer2 on behalf of the end user. This ticket is for enabling proxy access to HiveServer2 for third party tools on behalf of end users. There are two parts of the solution proposed in this ticket: 1) Delegation token based connection for Oozie (OOZIE-1457) This is the common mechanism for Hadoop ecosystem components. Hive Remote Metastore and HCatalog already support this. This is suitable for tool like Oozie that submits the MR jobs as actions on behalf of its client. Oozie already uses similar mechanism for Metastore/HCatalog access. 2) Direct proxy access for privileged hadoop users The delegation token implementation can be a challenge for non-hadoop (especially non-java) components. This second part enables a privileged user to directly specify an alternate session user during the connection. If the connecting user has hadoop level privilege to impersonate the requested userid, then HiveServer2 will run the session as that requested user. For example, user Hue is allowed to impersonate user Bob (via core-site.xml proxy user configuration). Then user Hue can connect to HiveServer2 and specify Bob as session user via a session property. HiveServer2 will verify Hue's proxy user privilege and then impersonate user Bob instead of Hue. This will enable any third party tool to impersonate alternate userid without having to implement delegation token connection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6537) NullPointerException when loading hashtable for MapJoin directly
[
https://issues.apache.org/jira/browse/HIVE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-6537:
---
Status: Patch Available (was: Open)
NullPointerException when loading hashtable for MapJoin directly
Key: HIVE-6537
URL: https://issues.apache.org/jira/browse/HIVE-6537
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-6537.patch
We see the following error:
{noformat}
2014-02-20 23:33:15,743 FATAL [main]
org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:103)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:149)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:164)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1026)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.NullPointerException
at java.util.Arrays.fill(Arrays.java:2685)
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.loadDirectly(HashTableLoader.java:155)
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:81)
... 15 more
{noformat}
It appears that the tables in Arrays.fill call is nulls. I don't really have
full understanding of this path, but what I gleaned so far is this...
From what I see, tables would be set unconditionally in initializeOp of the
sink, and in no other place, so I assume for this code to ever work that
startForward calls it at least some time.
Here, it doesn't call it, so it's null.
Previous loop also uses tables, and should have NPE-d before fill was ever
called; it didn't, so I'd assume it never executed.
There's a little bit of inconsistency in the above code where directWorks are
added to parents unconditionally but sink is only added as child
conditionally. I think it may be that some of the direct works are not table
scans; in fact given that loop never executes they may be null (which is
rather strange).
Regardless, it seems that the logic should be fixed, it may be the root cause
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6537) NullPointerException when loading hashtable for MapJoin directly
[
https://issues.apache.org/jira/browse/HIVE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-6537:
---
Attachment: HIVE-6537.patch
NullPointerException when loading hashtable for MapJoin directly
Key: HIVE-6537
URL: https://issues.apache.org/jira/browse/HIVE-6537
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-6537.patch
We see the following error:
{noformat}
2014-02-20 23:33:15,743 FATAL [main]
org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:103)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:149)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:164)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1026)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.NullPointerException
at java.util.Arrays.fill(Arrays.java:2685)
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.loadDirectly(HashTableLoader.java:155)
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:81)
... 15 more
{noformat}
It appears that the tables in Arrays.fill call is nulls. I don't really have
full understanding of this path, but what I gleaned so far is this...
From what I see, tables would be set unconditionally in initializeOp of the
sink, and in no other place, so I assume for this code to ever work that
startForward calls it at least some time.
Here, it doesn't call it, so it's null.
Previous loop also uses tables, and should have NPE-d before fill was ever
called; it didn't, so I'd assume it never executed.
There's a little bit of inconsistency in the above code where directWorks are
added to parents unconditionally but sink is only added as child
conditionally. I think it may be that some of the direct works are not table
scans; in fact given that loop never executes they may be null (which is
rather strange).
Regardless, it seems that the logic should be fixed, it may be the root cause
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5193) Columnar Pushdown for RC/ORC File not happening in HCatLoader
[ https://issues.apache.org/jira/browse/HIVE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918460#comment-13918460 ] Viraj Bhat commented on HIVE-5193: -- Sushanth, Thanks for your update. Really appreciate Viraj Columnar Pushdown for RC/ORC File not happening in HCatLoader -- Key: HIVE-5193 URL: https://issues.apache.org/jira/browse/HIVE-5193 Project: Hive Issue Type: Improvement Components: HCatalog Affects Versions: 0.10.0, 0.11.0, 0.12.0 Reporter: Viraj Bhat Assignee: Viraj Bhat Labels: hcatalog Fix For: 0.13.0 Attachments: HIVE-5193.2.patch, HIVE-5193.patch Currently the HCatLoader is not taking advantage of the ColumnProjectionUtils. where it could skip columns during read. The information is available in Pig it just needs to get to the Readers. Viraj -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6379) webhcat is not operational due to Slf4j conflicts
[ https://issues.apache.org/jira/browse/HIVE-6379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918456#comment-13918456 ] Eugene Koifman commented on HIVE-6379: -- to include .properties fiels in webhcat.jar code +++ b/hcatalog/webhcat/svr/pom.xml @@ -141,6 +141,12 @@ /profiles build + resources + resource + targetPath./targetPath + directorysrc/main/config/directory + /resource + /resources /code webhcat is not operational due to Slf4j conflicts - Key: HIVE-6379 URL: https://issues.apache.org/jira/browse/HIVE-6379 Project: Hive Issue Type: Bug Components: WebHCat Affects Versions: 0.12.0 Reporter: Eugene Koifman Assignee: Eugene Koifman 1. build current trunk: mvn clean install -Pdist,hadoop-1 -DskipTests 2. /Users/ekoifman/dev/hive/packaging/target/apache-hive-0.13.0-SNAPSHOT-bin/apache-hive-0.13.0-SNAPSHOT-bin/hcatalog 3. ./sbin/webhcat_server.sh start 4. http://localhost:50111/templeton/v1/ddl/database?user.name=ekoifman webhcat-console-error.log has log4j:WARN No appenders could be found for logger (org.apache.hive.hcatalog.templeton.AppConfig). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread main java.lang.NoSuchMethodError: org.slf4j.helpers.MessageFormatter.arrayFormat(Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple; at org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:613) at org.eclipse.jetty.util.log.JettyAwareLogger.debug(JettyAwareLogger.java:200) at org.eclipse.jetty.util.log.Slf4jLog.debug(Slf4jLog.java:94) at org.eclipse.jetty.util.component.Container.add(Container.java:201) at org.eclipse.jetty.util.component.Container.update(Container.java:164) at org.eclipse.jetty.util.component.Container.update(Container.java:106) at org.eclipse.jetty.server.Server.setConnectors(Server.java:184) at org.eclipse.jetty.server.Server.init(Server.java:97) at org.apache.hive.hcatalog.templeton.Main.runServer(Main.java:162) at org.apache.hive.hcatalog.templeton.Main.run(Main.java:114) at org.apache.hive.hcatalog.templeton.Main.main(Main.java:259) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:160) webhcat-console-error.log (END) -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 18588: sql std auth - add command to change owner of database
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18588/#review36013 --- ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java https://reviews.apache.org/r/18588/#comment66804 nit: spacing here and above ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java https://reviews.apache.org/r/18588/#comment66805 why is this change in this patch? - Sergey Shelukhin On March 2, 2014, 12:52 a.m., Thejas Nair wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18588/ --- (Updated March 2, 2014, 12:52 a.m.) Review request for hive and Ashutosh Chauhan. Bugs: HIVE-6440 https://issues.apache.org/jira/browse/HIVE-6440 Repository: hive-git Description --- It should be possible to change the owner of a database once it is created. Also includes a small change to authorize drop database, as it is better tested with this alter-db-owner functionality. Diffs - hbase-handler/src/test/results/negative/cascade_dbdrop_hadoop20.q.out c5bfd6a itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 7fbb67e metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java d60cb1b ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 4310ab8 ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 2484576 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 8356155 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 129c6c6 ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 6bf2a17 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java b1d3371 ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/AuthorizationParseUtils.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactoryImpl.java 4dbe78c ql/src/java/org/apache/hadoop/hive/ql/plan/AlterDatabaseDesc.java 16bb95b ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java e661f00 ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java f1671ba ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java c43bcea ql/src/test/queries/clientnegative/authorization_alter_db_owner.q PRE-CREATION ql/src/test/queries/clientnegative/authorization_alter_db_owner_default.q PRE-CREATION ql/src/test/queries/clientnegative/authorization_drop_db_cascade.q PRE-CREATION ql/src/test/queries/clientnegative/authorization_drop_db_empty.q PRE-CREATION ql/src/test/queries/clientpositive/alter_db_owner.q PRE-CREATION ql/src/test/results/clientnegative/authorization_alter_db_owner.q.out PRE-CREATION ql/src/test/results/clientnegative/authorization_alter_db_owner_default.q.out PRE-CREATION ql/src/test/results/clientnegative/authorization_drop_db_cascade.q.out PRE-CREATION ql/src/test/results/clientnegative/authorization_drop_db_empty.q.out PRE-CREATION ql/src/test/results/clientpositive/alter_db_owner.q.out PRE-CREATION ql/src/test/results/clientpositive/database.q.out cdaa405 ql/src/test/results/clientpositive/database_drop.q.out 3128561 ql/src/test/results/clientpositive/database_properties.q.out e3e2816 ql/src/test/results/clientpositive/drop_database_removes_partition_dirs.q.out 475d3d0 ql/src/test/results/clientpositive/input46.q.out a891362 ql/src/test/results/clientpositive/udf_using.q.out 69e5f3b Diff: https://reviews.apache.org/r/18588/diff/ Testing --- Thanks, Thejas Nair
[jira] [Commented] (HIVE-6537) NullPointerException when loading hashtable for MapJoin directly
[
https://issues.apache.org/jira/browse/HIVE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918472#comment-13918472
]
Sergey Shelukhin commented on HIVE-6537:
[~navis] [~vikram.dixit] fyi
NullPointerException when loading hashtable for MapJoin directly
Key: HIVE-6537
URL: https://issues.apache.org/jira/browse/HIVE-6537
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-6537.patch
We see the following error:
{noformat}
2014-02-20 23:33:15,743 FATAL [main]
org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:103)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:149)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:164)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1026)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.NullPointerException
at java.util.Arrays.fill(Arrays.java:2685)
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.loadDirectly(HashTableLoader.java:155)
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:81)
... 15 more
{noformat}
It appears that the tables in Arrays.fill call is nulls. I don't really have
full understanding of this path, but what I gleaned so far is this...
From what I see, tables would be set unconditionally in initializeOp of the
sink, and in no other place, so I assume for this code to ever work that
startForward calls it at least some time.
Here, it doesn't call it, so it's null.
Previous loop also uses tables, and should have NPE-d before fill was ever
called; it didn't, so I'd assume it never executed.
There's a little bit of inconsistency in the above code where directWorks are
added to parents unconditionally but sink is only added as child
conditionally. I think it may be that some of the direct works are not table
scans; in fact given that loop never executes they may be null (which is
rather strange).
Regardless, it seems that the logic should be fixed, it may be the root cause
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5761) Implement vectorized support for the DATE data type
[ https://issues.apache.org/jira/browse/HIVE-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918476#comment-13918476 ] Jitendra Nath Pandey commented on HIVE-5761: Committed to trunk. Thanks to Teddy! Implement vectorized support for the DATE data type --- Key: HIVE-5761 URL: https://issues.apache.org/jira/browse/HIVE-5761 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Teddy Choi Attachments: HIVE-5761.1.patch, HIVE-5761.2.patch, HIVE-5761.3.patch, HIVE-5761.4.patch, HIVE-5761.5.patch, HIVE-5761.6.patch, HIVE-5761.6.patch Add support to allow queries referencing DATE columns and expression results to run efficiently in vectorized mode. This should re-use the code for the the integer/timestamp types to the extent possible and beneficial. Include unit tests and end-to-end tests. Consider re-using or extending existing end-to-end tests for vectorized integer and/or timestamp operations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5761) Implement vectorized support for the DATE data type
[ https://issues.apache.org/jira/browse/HIVE-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-5761: --- Resolution: Fixed Status: Resolved (was: Patch Available) Implement vectorized support for the DATE data type --- Key: HIVE-5761 URL: https://issues.apache.org/jira/browse/HIVE-5761 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Teddy Choi Attachments: HIVE-5761.1.patch, HIVE-5761.2.patch, HIVE-5761.3.patch, HIVE-5761.4.patch, HIVE-5761.5.patch, HIVE-5761.6.patch, HIVE-5761.6.patch Add support to allow queries referencing DATE columns and expression results to run efficiently in vectorized mode. This should re-use the code for the the integer/timestamp types to the extent possible and beneficial. Include unit tests and end-to-end tests. Consider re-using or extending existing end-to-end tests for vectorized integer and/or timestamp operations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6538) yet another annoying exception in test logs
Sergey Shelukhin created HIVE-6538:
--
Summary: yet another annoying exception in test logs
Key: HIVE-6538
URL: https://issues.apache.org/jira/browse/HIVE-6538
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Priority: Trivial
Whenever you look at failed q tests you have to go thru this useless exception.
{noformat}
2014-03-03 11:22:54,872 ERROR metastore.RetryingHMSHandler
(RetryingHMSHandler.java:invoke(143)) -
MetaException(message:NoSuchObjectException(message:Function
default.qtest_get_java_boolean does not exist))
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:4575)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_function(HiveMetaStore.java:4702)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at $Proxy8.get_function(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunction(HiveMetaStoreClient.java:1526)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
at $Proxy9.getFunction(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getFunction(Hive.java:2603)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfoFromMetastore(FunctionRegistry.java:546)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getQualifiedFunctionInfo(FunctionRegistry.java:578)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:599)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:606)
at
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:94)
at
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:60)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:445)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:345)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1078)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1121)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1014)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
at org.apache.hadoop.hive.ql.QTestUtil.runCmd(QTestUtil.java:655)
at org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:772)
at
org.apache.hadoop.hive.cli.TestCliDriver.clinit(TestCliDriver.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:34)
at org.junit.internal.runners.SuiteMethod.init(SuiteMethod.java:23)
at
org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:14)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at
org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
[jira] [Created] (HIVE-6539) Couple of issues in fs based stats collection
Ashutosh Chauhan created HIVE-6539: -- Summary: Couple of issues in fs based stats collection Key: HIVE-6539 URL: https://issues.apache.org/jira/browse/HIVE-6539 Project: Hive Issue Type: Bug Reporter: Ashutosh Chauhan While testing on cluster found couple of bugs: * NPE in certain case. * map object reuse causing problem -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6539) Couple of issues in fs based stats collection
[ https://issues.apache.org/jira/browse/HIVE-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6539: --- Attachment: HIVE-6539.patch * Fixes NPE * Do new Hashmap to avoid object reuse issue. * Increased test coverage by putting fs based stats collection in tests. Couple of issues in fs based stats collection - Key: HIVE-6539 URL: https://issues.apache.org/jira/browse/HIVE-6539 Project: Hive Issue Type: Bug Reporter: Ashutosh Chauhan Attachments: HIVE-6539.patch While testing on cluster found couple of bugs: * NPE in certain case. * map object reuse causing problem -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6539) Couple of issues in fs based stats collection
[ https://issues.apache.org/jira/browse/HIVE-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6539: --- Assignee: Ashutosh Chauhan Status: Patch Available (was: Open) Couple of issues in fs based stats collection - Key: HIVE-6539 URL: https://issues.apache.org/jira/browse/HIVE-6539 Project: Hive Issue Type: Bug Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-6539.patch While testing on cluster found couple of bugs: * NPE in certain case. * map object reuse causing problem -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HIVE-5787) Move non-type specific column statistics to ColumnStatistics object in ORC
[
https://issues.apache.org/jira/browse/HIVE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth J reassigned HIVE-5787:
Assignee: Prasanth J
Move non-type specific column statistics to ColumnStatistics object in ORC
--
Key: HIVE-5787
URL: https://issues.apache.org/jira/browse/HIVE-5787
Project: Hive
Issue Type: Improvement
Affects Versions: 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
Priority: Minor
Labels: orcfile
Non-type statistics like min, max, numNulls etc. can be moved to
ColumnStatistics class. This avoids ugly checks like
{code} if (index instanceof IntegerColumnStatistics) {
return ((IntegerColumnStatistics) index).getMinimum();
} else if (index instanceof DoubleColumnStatistics) {
return ((DoubleColumnStatistics) index).getMinimum();
} else if (index instanceof StringColumnStatistics) {
return ((StringColumnStatistics) index).getMinimum();
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6492) limit partition number involved in a table scan
[ https://issues.apache.org/jira/browse/HIVE-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Selina Zhang updated HIVE-6492: --- Attachment: HIVE-6492.3.patch.txt The test failure seems not related to this patch. Re-submit the patch. limit partition number involved in a table scan --- Key: HIVE-6492 URL: https://issues.apache.org/jira/browse/HIVE-6492 Project: Hive Issue Type: New Feature Components: Query Processor Affects Versions: 0.12.0 Reporter: Selina Zhang Fix For: 0.13.0 Attachments: HIVE-6492.1.patch.txt, HIVE-6492.2.patch.txt, HIVE-6492.3.patch.txt Original Estimate: 24h Remaining Estimate: 24h To protect the cluster, a new configure variable hive.limit.query.max.table.partition is added to hive configuration to limit the table partitions involved in a table scan. The default value will be set to -1 which means there is no limit by default. This variable will not affect metadata only query. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6486) Support secure Subject.doAs() in HiveServer2 JDBC client.
[ https://issues.apache.org/jira/browse/HIVE-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6486: --- Assignee: Shivaraju Gowda Support secure Subject.doAs() in HiveServer2 JDBC client. - Key: HIVE-6486 URL: https://issues.apache.org/jira/browse/HIVE-6486 Project: Hive Issue Type: Improvement Components: Authentication, HiveServer2, JDBC Affects Versions: 0.11.0, 0.12.0 Reporter: Shivaraju Gowda Assignee: Shivaraju Gowda Fix For: 0.13.0 Attachments: HIVE-6486.1.patch, Hive_011_Support-Subject_doAS.patch, TestHive_SujectDoAs.java HIVE-5155 addresses the problem of kerberos authentication in multi-user middleware server using proxy user. In this mode the principal used by the middle ware server has privileges to impersonate selected users in Hive/Hadoop. This enhancement is to support Subject.doAs() authentication in Hive JDBC layer so that the end users Kerberos Subject is passed through in the middle ware server. With this improvement there won't be any additional setup in the server to grant proxy privileges to some users and there won't be need to specify a proxy user in the JDBC client. This version should also be more secure since it won't require principals with the privileges to impersonate other users in Hive/Hadoop setup. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5728) Make ORC InputFormat/OutputFormat usable outside Hive
[ https://issues.apache.org/jira/browse/HIVE-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918590#comment-13918590 ] Rohini Palaniswamy commented on HIVE-5728: -- [~owen.omalley], Shouldn't this jira be marked Resolved or is there some reason to keep it open? Make ORC InputFormat/OutputFormat usable outside Hive - Key: HIVE-5728 URL: https://issues.apache.org/jira/browse/HIVE-5728 Project: Hive Issue Type: Improvement Components: File Formats Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.13.0 Attachments: HIVE-5728-1.patch, HIVE-5728-10.patch, HIVE-5728-2.patch, HIVE-5728-3.patch, HIVE-5728-4.patch, HIVE-5728-5.patch, HIVE-5728-6.patch, HIVE-5728-7.patch, HIVE-5728-8.patch, HIVE-5728-9.patch, HIVE-5728.10.patch, HIVE-5728.11.patch, HIVE-5728.12.patch, HIVE-5728.13.patch ORC InputFormat/OutputFormat is currently not usable outside Hive. There are several issues need to solve: 1. Several class is not public, eg: OrcStruct 2. There is no InputFormat/OutputFormat for new api (Some tools such as Pig need new api) 3. Has no way to push WriteOption to OutputFormat outside Hive -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6538) yet another annoying exception in test logs
[
https://issues.apache.org/jira/browse/HIVE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918593#comment-13918593
]
Szehon Ho commented on HIVE-6538:
-
In RetryingHMSHandler, there are checks for get_table and get_partitions not to
log to error, I guess similar thing needs to apply for new method
get_function.
yet another annoying exception in test logs
---
Key: HIVE-6538
URL: https://issues.apache.org/jira/browse/HIVE-6538
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Priority: Trivial
Whenever you look at failed q tests you have to go thru this useless
exception.
{noformat}
2014-03-03 11:22:54,872 ERROR metastore.RetryingHMSHandler
(RetryingHMSHandler.java:invoke(143)) -
MetaException(message:NoSuchObjectException(message:Function
default.qtest_get_java_boolean does not exist))
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:4575)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_function(HiveMetaStore.java:4702)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at $Proxy8.get_function(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunction(HiveMetaStoreClient.java:1526)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
at $Proxy9.getFunction(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getFunction(Hive.java:2603)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfoFromMetastore(FunctionRegistry.java:546)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getQualifiedFunctionInfo(FunctionRegistry.java:578)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:599)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:606)
at
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:94)
at
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:60)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:445)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:345)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1078)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1121)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1014)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
at org.apache.hadoop.hive.ql.QTestUtil.runCmd(QTestUtil.java:655)
at org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:772)
at
org.apache.hadoop.hive.cli.TestCliDriver.clinit(TestCliDriver.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:34)
at org.junit.internal.runners.SuiteMethod.init(SuiteMethod.java:23)
at
org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:14)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at
org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
Review Request 18709: Bug fix for fs based stats collection.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18709/ --- Review request for hive. Bugs: HIVE-6539 https://issues.apache.org/jira/browse/HIVE-6539 Repository: hive-git Description --- Bug fix for fs based stats collection. Diffs - ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java 3255031 ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsPublisher.java d898a26 ql/src/test/queries/clientpositive/metadata_only_queries.q 9cec4da ql/src/test/queries/clientpositive/metadata_only_queries_with_filters.q 998f6a1 ql/src/test/queries/clientpositive/stats_only_null.q 38524b4 Diff: https://reviews.apache.org/r/18709/diff/ Testing --- Made few stats based test to use fs based stats collection. Thanks, Ashutosh Chauhan
[jira] [Created] (HIVE-6540) Support Multi Column Stats
Laljo John Pullokkaran created HIVE-6540: Summary: Support Multi Column Stats Key: HIVE-6540 URL: https://issues.apache.org/jira/browse/HIVE-6540 Project: Hive Issue Type: Improvement Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran For Joins involving compound predicates, multi column stats can be used to accurately compute the NDV. Objective is to compute NDV of more than one columns. Compute NDV of (x,y,z). R1 IJ R2 on R1.x=R2.x and R1.y=R2.y and R1.z=R2.z can use max(NDV(R1.x, R1.y, R1.z), NDV(R2.x, R2.y, R2.z)) for Join NDV ( hence selectivity). http://www.oracle-base.com/articles/11g/statistics-collection-enhancements-11gr1.php#multi_column_statistics http://blogs.msdn.com/b/ianjo/archive/2005/11/10/491548.aspx http://developer.teradata.com/database/articles/removing-multi-column-statistics-a-process-for-identification-of-redundant-statist -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6507) OrcFile table property names are specified as strings
[ https://issues.apache.org/jira/browse/HIVE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-6507: --- Status: Open (was: Patch Available) OrcFile table property names are specified as strings - Key: HIVE-6507 URL: https://issues.apache.org/jira/browse/HIVE-6507 Project: Hive Issue Type: Bug Components: HCatalog, Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: HIVE-6507.patch In HIVE-5504, we had to do some special casing in HCatalog to add a particular set of orc table properties from table properties to job properties. In doing so, it's obvious that that is a bit cumbersome, and ideally, the list of all orc file table properties should really be an enum, rather than individual loosely tied constant strings. If we were to clean this up, we can clean up other code that references this to reference the entire enum, and avoid future errors when new table properties are introduced, but other referencing code is not updated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6409) FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.
[
https://issues.apache.org/jira/browse/HIVE-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-6409:
---
Resolution: Fixed
Fix Version/s: 0.13.0
Status: Resolved (was: Patch Available)
FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.
--
Key: HIVE-6409
URL: https://issues.apache.org/jira/browse/HIVE-6409
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.12.0, 0.13.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Fix For: 0.13.0
Attachments: HIVE-6409.patch
When HCatalog's FileOutputCommitterContainer::commitJob() is run, it calls
the underlying OutputCommitter and then attempts to register partitions in
HCatalog.
If the commit fails (for example, because of HIVE-4996), commitJob() cancels
delegation tokens retrieved from HCatalog before the exception is rethrown.
{code}
java.io.IOException: java.lang.reflect.InvocationTargetException
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:183)
... 5 more
Caused by: org.apache.hcatalog.common.HCatException : 2006 : Error adding
partition to metastore. Cause :
MetaException(message:java.lang.RuntimeException: commitTransaction was called
but openTransactionCalls = 0. This probably indicates that there are
unbalanced
calls to openTransaction/commitTransaction)
at
org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:712)
{code}
The problem is that this happens before abortJob() has had a chance to run,
thus yielding the following error:
{code}
MetaException(message:Could not connect to meta store using any of the URIs
provided. Most recent failure:
org.apache.thrift.transport.TTransportException: Peer indicated failure:
DIGEST-MD5: IO error acquiring password
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
...
{code}
I'll have a patch out that only cancels delegation tokens if the commitJob()
has succeeded.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6507) OrcFile table property names are specified as strings
[ https://issues.apache.org/jira/browse/HIVE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-6507: --- Status: Patch Available (was: Open) OrcFile table property names are specified as strings - Key: HIVE-6507 URL: https://issues.apache.org/jira/browse/HIVE-6507 Project: Hive Issue Type: Bug Components: HCatalog, Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: HIVE-6507.patch In HIVE-5504, we had to do some special casing in HCatalog to add a particular set of orc table properties from table properties to job properties. In doing so, it's obvious that that is a bit cumbersome, and ideally, the list of all orc file table properties should really be an enum, rather than individual loosely tied constant strings. If we were to clean this up, we can clean up other code that references this to reference the entire enum, and avoid future errors when new table properties are introduced, but other referencing code is not updated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6537) NullPointerException when loading hashtable for MapJoin directly
[
https://issues.apache.org/jira/browse/HIVE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918618#comment-13918618
]
Hive QA commented on HIVE-6537:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12632331/HIVE-6537.patch
{color:red}ERROR:{color} -1 due to 61 failed/errored test(s), 5236 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cp_mj_rc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join35
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_star
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_subquery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_subquery2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_reduce_deduplicate
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1606/testReport
[jira] [Updated] (HIVE-5950) ORC SARG creation fails with NPE for predicate conditions with decimal/date/char/varchar datatypes
[ https://issues.apache.org/jira/browse/HIVE-5950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-5950: - Attachment: HIVE-5950.5.patch Addressed [~hagleitn]'s code review comments. ORC SARG creation fails with NPE for predicate conditions with decimal/date/char/varchar datatypes -- Key: HIVE-5950 URL: https://issues.apache.org/jira/browse/HIVE-5950 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile Attachments: HIVE-5950.1.patch, HIVE-5950.2.patch, HIVE-5950.3.patch, HIVE-5950.4.patch, HIVE-5950.5.patch When decimal or date column is used, the type field in PredicateLeafImpl will be set to null. This will result in NPE during predicate leaf generation because of null dereferencing in hashcode computation. SARG creation should be extended to support/handle decimal and date data types. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 17471: HIVE-6325: Enable using multiple concurrent sessions in tez
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17471/#review36039 --- ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java https://reviews.apache.org/r/17471/#comment66866 can you remove trailing ws in the files. This shouldn't require a new run of the tests... service/src/java/org/apache/hive/service/server/HiveServer2.java https://reviews.apache.org/r/17471/#comment66868 i believe this should be hidden behind conf check - Gunther Hagleitner On Feb. 28, 2014, 2:25 a.m., Vikram Dixit Kumaraswamy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17471/ --- (Updated Feb. 28, 2014, 2:25 a.m.) Review request for hive. Bugs: HIVE-6325 https://issues.apache.org/jira/browse/HIVE-6325 Repository: hive-git Description --- Enable using multiple concurrent sessions in tez. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6802b4d conf/hive-default.xml.template 0b86b9c ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b8552a3 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java bef5ba3 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 16e9166 ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java PRE-CREATION service/src/java/org/apache/hive/service/server/HiveServer2.java 8700c5b Diff: https://reviews.apache.org/r/17471/diff/ Testing --- Added multi-threaded junit tests. Thanks, Vikram Dixit Kumaraswamy
[jira] [Commented] (HIVE-6325) Enable using multiple concurrent sessions in tez
[ https://issues.apache.org/jira/browse/HIVE-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918645#comment-13918645 ] Gunther Hagleitner commented on HIVE-6325: -- minor comments on rb. Otherwise +1 LGTM. Enable using multiple concurrent sessions in tez Key: HIVE-6325 URL: https://issues.apache.org/jira/browse/HIVE-6325 Project: Hive Issue Type: Improvement Components: Tez Affects Versions: 0.13.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-6325.1.patch, HIVE-6325.2.patch, HIVE-6325.3.patch, HIVE-6325.4.patch, HIVE-6325.5.patch, HIVE-6325.6.patch, HIVE-6325.7.patch, HIVE-6325.8.patch, HIVE-6325.9.patch We would like to enable multiple concurrent sessions in tez via hive server 2. This will enable users to make efficient use of the cluster when it has been partitioned using yarn queues. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5193) Columnar Pushdown for RC/ORC File not happening in HCatLoader
[ https://issues.apache.org/jira/browse/HIVE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918660#comment-13918660 ] Sushanth Sowmyan commented on HIVE-5193: Looks good to me, +1. Columnar Pushdown for RC/ORC File not happening in HCatLoader -- Key: HIVE-5193 URL: https://issues.apache.org/jira/browse/HIVE-5193 Project: Hive Issue Type: Improvement Components: HCatalog Affects Versions: 0.10.0, 0.11.0, 0.12.0 Reporter: Viraj Bhat Assignee: Viraj Bhat Labels: hcatalog Fix For: 0.13.0 Attachments: HIVE-5193.2.patch, HIVE-5193.patch Currently the HCatLoader is not taking advantage of the ColumnProjectionUtils. where it could skip columns during read. The information is available in Pig it just needs to get to the Readers. Viraj -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6392) Hive (and HCatalog) don't allow super-users to add partitions to tables.
[ https://issues.apache.org/jira/browse/HIVE-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-6392: --- Status: Patch Available (was: Open) Hive (and HCatalog) don't allow super-users to add partitions to tables. Key: HIVE-6392 URL: https://issues.apache.org/jira/browse/HIVE-6392 Project: Hive Issue Type: Bug Components: Authorization Affects Versions: 0.12.0, 0.13.0 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: HIVE-6392.branch-0.12.patch, HIVE-6392.patch HDFS allows for users to be added to a supergroup (identified by the dfs.permissions.superusergroup key in hdfs-site.xml). Users in this group are allowed to modify HDFS contents regardless of the path's ogw permissions. However, Hive's StorageBasedAuthProvider disallows such a superuser from adding partitions to any table that doesn't explicitly grant write permissions to said superuser. This causes the odd scenario where the superuser writes data to a partition-directory (under the table's path), but can't register the appropriate partition. I have a patch that brings the Metastore's behaviour in line with what the HDFS allows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6392) Hive (and HCatalog) don't allow super-users to add partitions to tables.
[ https://issues.apache.org/jira/browse/HIVE-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-6392: --- Status: Open (was: Patch Available) Hive (and HCatalog) don't allow super-users to add partitions to tables. Key: HIVE-6392 URL: https://issues.apache.org/jira/browse/HIVE-6392 Project: Hive Issue Type: Bug Components: Authorization Affects Versions: 0.12.0, 0.13.0 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: HIVE-6392.branch-0.12.patch, HIVE-6392.patch HDFS allows for users to be added to a supergroup (identified by the dfs.permissions.superusergroup key in hdfs-site.xml). Users in this group are allowed to modify HDFS contents regardless of the path's ogw permissions. However, Hive's StorageBasedAuthProvider disallows such a superuser from adding partitions to any table that doesn't explicitly grant write permissions to said superuser. This causes the odd scenario where the superuser writes data to a partition-directory (under the table's path), but can't register the appropriate partition. I have a patch that brings the Metastore's behaviour in line with what the HDFS allows. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 17471: HIVE-6325: Enable using multiple concurrent sessions in tez
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17471/ --- (Updated March 3, 2014, 10:39 p.m.) Review request for hive. Changes --- Addressed comments. Bugs: HIVE-6325 https://issues.apache.org/jira/browse/HIVE-6325 Repository: hive-git Description --- Enable using multiple concurrent sessions in tez. Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 388a604 conf/hive-default.xml.template 3f01e0b ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java aef6e68 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java d30ec8e ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 16e9166 ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java PRE-CREATION service/src/java/org/apache/hive/service/server/HiveServer2.java 8700c5b Diff: https://reviews.apache.org/r/17471/diff/ Testing --- Added multi-threaded junit tests. Thanks, Vikram Dixit Kumaraswamy
[jira] [Updated] (HIVE-6538) yet another annoying exception in test logs
[
https://issues.apache.org/jira/browse/HIVE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-6538:
Attachment: HIVE-6538.patch
This seems to work and get rid of the stack trace in the log.
However, FunctionRegistry.getFunctionInfoFromMetastore() still seems to print
the top-level error, not sure if its by design. + [~jdere]
yet another annoying exception in test logs
---
Key: HIVE-6538
URL: https://issues.apache.org/jira/browse/HIVE-6538
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Priority: Trivial
Attachments: HIVE-6538.patch
Whenever you look at failed q tests you have to go thru this useless
exception.
{noformat}
2014-03-03 11:22:54,872 ERROR metastore.RetryingHMSHandler
(RetryingHMSHandler.java:invoke(143)) -
MetaException(message:NoSuchObjectException(message:Function
default.qtest_get_java_boolean does not exist))
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:4575)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_function(HiveMetaStore.java:4702)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at $Proxy8.get_function(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunction(HiveMetaStoreClient.java:1526)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
at $Proxy9.getFunction(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getFunction(Hive.java:2603)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfoFromMetastore(FunctionRegistry.java:546)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getQualifiedFunctionInfo(FunctionRegistry.java:578)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:599)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:606)
at
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:94)
at
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:60)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:445)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:345)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1078)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1121)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1014)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
at org.apache.hadoop.hive.ql.QTestUtil.runCmd(QTestUtil.java:655)
at org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:772)
at
org.apache.hadoop.hive.cli.TestCliDriver.clinit(TestCliDriver.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:34)
at org.junit.internal.runners.SuiteMethod.init(SuiteMethod.java:23)
at
org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:14)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at
org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
at
[jira] [Updated] (HIVE-6538) yet another annoying exception in test logs
[
https://issues.apache.org/jira/browse/HIVE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-6538:
Status: Patch Available (was: Open)
yet another annoying exception in test logs
---
Key: HIVE-6538
URL: https://issues.apache.org/jira/browse/HIVE-6538
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Szehon Ho
Priority: Trivial
Attachments: HIVE-6538.patch
Whenever you look at failed q tests you have to go thru this useless
exception.
{noformat}
2014-03-03 11:22:54,872 ERROR metastore.RetryingHMSHandler
(RetryingHMSHandler.java:invoke(143)) -
MetaException(message:NoSuchObjectException(message:Function
default.qtest_get_java_boolean does not exist))
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:4575)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_function(HiveMetaStore.java:4702)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at $Proxy8.get_function(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunction(HiveMetaStoreClient.java:1526)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
at $Proxy9.getFunction(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getFunction(Hive.java:2603)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfoFromMetastore(FunctionRegistry.java:546)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getQualifiedFunctionInfo(FunctionRegistry.java:578)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:599)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:606)
at
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:94)
at
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:60)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:445)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:345)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1078)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1121)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1014)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
at org.apache.hadoop.hive.ql.QTestUtil.runCmd(QTestUtil.java:655)
at org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:772)
at
org.apache.hadoop.hive.cli.TestCliDriver.clinit(TestCliDriver.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:34)
at org.junit.internal.runners.SuiteMethod.init(SuiteMethod.java:23)
at
org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:14)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at
org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
[jira] [Assigned] (HIVE-6538) yet another annoying exception in test logs
[
https://issues.apache.org/jira/browse/HIVE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho reassigned HIVE-6538:
---
Assignee: Szehon Ho
yet another annoying exception in test logs
---
Key: HIVE-6538
URL: https://issues.apache.org/jira/browse/HIVE-6538
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Szehon Ho
Priority: Trivial
Attachments: HIVE-6538.patch
Whenever you look at failed q tests you have to go thru this useless
exception.
{noformat}
2014-03-03 11:22:54,872 ERROR metastore.RetryingHMSHandler
(RetryingHMSHandler.java:invoke(143)) -
MetaException(message:NoSuchObjectException(message:Function
default.qtest_get_java_boolean does not exist))
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:4575)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_function(HiveMetaStore.java:4702)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at $Proxy8.get_function(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunction(HiveMetaStoreClient.java:1526)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
at $Proxy9.getFunction(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getFunction(Hive.java:2603)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfoFromMetastore(FunctionRegistry.java:546)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getQualifiedFunctionInfo(FunctionRegistry.java:578)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:599)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:606)
at
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:94)
at
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:60)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:445)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:345)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1078)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1121)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1014)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
at org.apache.hadoop.hive.ql.QTestUtil.runCmd(QTestUtil.java:655)
at org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:772)
at
org.apache.hadoop.hive.cli.TestCliDriver.clinit(TestCliDriver.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:34)
at org.junit.internal.runners.SuiteMethod.init(SuiteMethod.java:23)
at
org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:14)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at
org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
[jira] [Commented] (HIVE-6532) ORC NPE on null Map
[ https://issues.apache.org/jira/browse/HIVE-6532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918675#comment-13918675 ] Prasanth J commented on HIVE-6532: -- ORC creates SARG (searchable argument) when there are where predicates in the query. SARG creation fails for non-primitive columns. This is already tracked in HIVE-5567. The fix for it should go in next release. ORC NPE on null Map --- Key: HIVE-6532 URL: https://issues.apache.org/jira/browse/HIVE-6532 Project: Hive Issue Type: Bug Components: File Formats Affects Versions: 0.12.0 Environment: ubuntu 12.04 Reporter: nicolas maillard Priority: Trivial On a select query map tasks will fail with npe for a where clause on a null map. on an ORC table example select * from my table where mymap['entry'] = 'something' If my map where to be null in a line we will get an npe exception. Maybe the line should just be ignored. The same query on a text format table will resolve correctly -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6325) Enable using multiple concurrent sessions in tez
[ https://issues.apache.org/jira/browse/HIVE-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6325: - Status: Open (was: Patch Available) Enable using multiple concurrent sessions in tez Key: HIVE-6325 URL: https://issues.apache.org/jira/browse/HIVE-6325 Project: Hive Issue Type: Improvement Components: Tez Affects Versions: 0.13.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-6325.1.patch, HIVE-6325.10.patch, HIVE-6325.2.patch, HIVE-6325.3.patch, HIVE-6325.4.patch, HIVE-6325.5.patch, HIVE-6325.6.patch, HIVE-6325.7.patch, HIVE-6325.8.patch, HIVE-6325.9.patch We would like to enable multiple concurrent sessions in tez via hive server 2. This will enable users to make efficient use of the cluster when it has been partitioned using yarn queues. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6325) Enable using multiple concurrent sessions in tez
[ https://issues.apache.org/jira/browse/HIVE-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6325: - Attachment: HIVE-6325.10.patch Address comments. Enable using multiple concurrent sessions in tez Key: HIVE-6325 URL: https://issues.apache.org/jira/browse/HIVE-6325 Project: Hive Issue Type: Improvement Components: Tez Affects Versions: 0.13.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-6325.1.patch, HIVE-6325.10.patch, HIVE-6325.2.patch, HIVE-6325.3.patch, HIVE-6325.4.patch, HIVE-6325.5.patch, HIVE-6325.6.patch, HIVE-6325.7.patch, HIVE-6325.8.patch, HIVE-6325.9.patch We would like to enable multiple concurrent sessions in tez via hive server 2. This will enable users to make efficient use of the cluster when it has been partitioned using yarn queues. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5193) Columnar Pushdown for RC/ORC File not happening in HCatLoader
[ https://issues.apache.org/jira/browse/HIVE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918679#comment-13918679 ] Sushanth Sowmyan commented on HIVE-5193: Committed. Thanks, Viraj! Columnar Pushdown for RC/ORC File not happening in HCatLoader -- Key: HIVE-5193 URL: https://issues.apache.org/jira/browse/HIVE-5193 Project: Hive Issue Type: Improvement Components: HCatalog Affects Versions: 0.10.0, 0.11.0, 0.12.0 Reporter: Viraj Bhat Assignee: Viraj Bhat Labels: hcatalog Fix For: 0.13.0 Attachments: HIVE-5193.2.patch, HIVE-5193.patch Currently the HCatLoader is not taking advantage of the ColumnProjectionUtils. where it could skip columns during read. The information is available in Pig it just needs to get to the Readers. Viraj -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5193) Columnar Pushdown for RC/ORC File not happening in HCatLoader
[ https://issues.apache.org/jira/browse/HIVE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-5193: --- Resolution: Fixed Status: Resolved (was: Patch Available) Columnar Pushdown for RC/ORC File not happening in HCatLoader -- Key: HIVE-5193 URL: https://issues.apache.org/jira/browse/HIVE-5193 Project: Hive Issue Type: Improvement Components: HCatalog Affects Versions: 0.10.0, 0.11.0, 0.12.0 Reporter: Viraj Bhat Assignee: Viraj Bhat Labels: hcatalog Fix For: 0.13.0 Attachments: HIVE-5193.2.patch, HIVE-5193.patch Currently the HCatLoader is not taking advantage of the ColumnProjectionUtils. where it could skip columns during read. The information is available in Pig it just needs to get to the Readers. Viraj -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5193) Columnar Pushdown for RC/ORC File not happening in HCatLoader
[ https://issues.apache.org/jira/browse/HIVE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-5193: --- Release Note: (Columnar pushdown will now work with HCatLoader - basically, what this means is that when using RC/ORC tables, not all rows are necessarily read and processed if pig would discard (by filtering) it anyway. This allows reads to be a bit faster.) Columnar Pushdown for RC/ORC File not happening in HCatLoader -- Key: HIVE-5193 URL: https://issues.apache.org/jira/browse/HIVE-5193 Project: Hive Issue Type: Improvement Components: HCatalog Affects Versions: 0.10.0, 0.11.0, 0.12.0 Reporter: Viraj Bhat Assignee: Viraj Bhat Labels: hcatalog Fix For: 0.13.0 Attachments: HIVE-5193.2.patch, HIVE-5193.patch Currently the HCatLoader is not taking advantage of the ColumnProjectionUtils. where it could skip columns during read. The information is available in Pig it just needs to get to the Readers. Viraj -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6538) yet another annoying exception in test logs
[
https://issues.apache.org/jira/browse/HIVE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918684#comment-13918684
]
Sergey Shelukhin commented on HIVE-6538:
+1 assuming tests pass, esp. negative cli
yet another annoying exception in test logs
---
Key: HIVE-6538
URL: https://issues.apache.org/jira/browse/HIVE-6538
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Szehon Ho
Priority: Trivial
Attachments: HIVE-6538.patch
Whenever you look at failed q tests you have to go thru this useless
exception.
{noformat}
2014-03-03 11:22:54,872 ERROR metastore.RetryingHMSHandler
(RetryingHMSHandler.java:invoke(143)) -
MetaException(message:NoSuchObjectException(message:Function
default.qtest_get_java_boolean does not exist))
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:4575)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_function(HiveMetaStore.java:4702)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at $Proxy8.get_function(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunction(HiveMetaStoreClient.java:1526)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
at $Proxy9.getFunction(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getFunction(Hive.java:2603)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfoFromMetastore(FunctionRegistry.java:546)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getQualifiedFunctionInfo(FunctionRegistry.java:578)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:599)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:606)
at
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:94)
at
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:60)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:445)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:345)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1078)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1121)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1014)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
at org.apache.hadoop.hive.ql.QTestUtil.runCmd(QTestUtil.java:655)
at org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:772)
at
org.apache.hadoop.hive.cli.TestCliDriver.clinit(TestCliDriver.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:34)
at org.junit.internal.runners.SuiteMethod.init(SuiteMethod.java:23)
at
org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:14)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at
org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
[jira] [Commented] (HIVE-5950) ORC SARG creation fails with NPE for predicate conditions with decimal/date/char/varchar datatypes
[ https://issues.apache.org/jira/browse/HIVE-5950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918652#comment-13918652 ] Gunther Hagleitner commented on HIVE-5950: -- +1 ORC SARG creation fails with NPE for predicate conditions with decimal/date/char/varchar datatypes -- Key: HIVE-5950 URL: https://issues.apache.org/jira/browse/HIVE-5950 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile Attachments: HIVE-5950.1.patch, HIVE-5950.2.patch, HIVE-5950.3.patch, HIVE-5950.4.patch, HIVE-5950.5.patch When decimal or date column is used, the type field in PredicateLeafImpl will be set to null. This will result in NPE during predicate leaf generation because of null dereferencing in hashcode computation. SARG creation should be extended to support/handle decimal and date data types. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6541) Need to write documentation for ACID work
Alan Gates created HIVE-6541: Summary: Need to write documentation for ACID work Key: HIVE-6541 URL: https://issues.apache.org/jira/browse/HIVE-6541 Project: Hive Issue Type: Sub-task Components: Documentation Affects Versions: 0.13.0 Reporter: Alan Gates Assignee: Alan Gates Fix For: 0.13.0 ACID introduces a number of new config file options, tables in the metastore, keywords in the grammar, and a new interface for use of tools like storm and flume. These need to be documented. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6518) Add a GC canary to the VectorGroupByOperator to flush whenever a GC is triggered
[ https://issues.apache.org/jira/browse/HIVE-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-6518: -- Status: Patch Available (was: Open) Add a GC canary to the VectorGroupByOperator to flush whenever a GC is triggered Key: HIVE-6518 URL: https://issues.apache.org/jira/browse/HIVE-6518 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.13.0 Reporter: Gopal V Assignee: Gopal V Priority: Minor Attachments: HIVE-6518.1-tez.patch, HIVE-6518.2-tez.patch, HIVE-6518.2.patch The current VectorGroupByOperator implementation flushes the in-memory hashes when the maximum entries or fraction of memory is hit. This works for most cases, but there are some corner cases where we hit GC ovehead limits or heap size limits before either of those conditions are reached due to the rest of the pipeline. This patch adds a SoftReference as a GC canary. If the soft reference is dead, then a full GC pass happened sometime in the near past the aggregation hashtables should be flushed immediately before another full GC is triggered. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6518) Add a GC canary to the VectorGroupByOperator to flush whenever a GC is triggered
[ https://issues.apache.org/jira/browse/HIVE-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-6518: -- Status: Open (was: Patch Available) Add a GC canary to the VectorGroupByOperator to flush whenever a GC is triggered Key: HIVE-6518 URL: https://issues.apache.org/jira/browse/HIVE-6518 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.13.0 Reporter: Gopal V Assignee: Gopal V Priority: Minor Attachments: HIVE-6518.1-tez.patch, HIVE-6518.2-tez.patch, HIVE-6518.2.patch The current VectorGroupByOperator implementation flushes the in-memory hashes when the maximum entries or fraction of memory is hit. This works for most cases, but there are some corner cases where we hit GC ovehead limits or heap size limits before either of those conditions are reached due to the rest of the pipeline. This patch adds a SoftReference as a GC canary. If the soft reference is dead, then a full GC pass happened sometime in the near past the aggregation hashtables should be flushed immediately before another full GC is triggered. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6392) Hive (and HCatalog) don't allow super-users to add partitions to tables.
[ https://issues.apache.org/jira/browse/HIVE-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918697#comment-13918697 ] Mithun Radhakrishnan commented on HIVE-6392: Hey, Thejas. This failure doesn't look related to the code change. I've resubmitted the patch. I'm hoping for a cleaner run. Hive (and HCatalog) don't allow super-users to add partitions to tables. Key: HIVE-6392 URL: https://issues.apache.org/jira/browse/HIVE-6392 Project: Hive Issue Type: Bug Components: Authorization Affects Versions: 0.12.0, 0.13.0 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: HIVE-6392.branch-0.12.patch, HIVE-6392.patch HDFS allows for users to be added to a supergroup (identified by the dfs.permissions.superusergroup key in hdfs-site.xml). Users in this group are allowed to modify HDFS contents regardless of the path's ogw permissions. However, Hive's StorageBasedAuthProvider disallows such a superuser from adding partitions to any table that doesn't explicitly grant write permissions to said superuser. This causes the odd scenario where the superuser writes data to a partition-directory (under the table's path), but can't register the appropriate partition. I have a patch that brings the Metastore's behaviour in line with what the HDFS allows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6290) Add support for hbase filters for composite keys
[ https://issues.apache.org/jira/browse/HIVE-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918718#comment-13918718 ] Xuefu Zhang commented on HIVE-6290: --- [~swarnim] The review board (https://reviews.apache.org/r/17483/) doesn't seem having your latest code change. Do you mind updating that? You can reuse the same review request, but uploading diff with your latest patch. Add support for hbase filters for composite keys Key: HIVE-6290 URL: https://issues.apache.org/jira/browse/HIVE-6290 Project: Hive Issue Type: Sub-task Components: HBase Handler Affects Versions: 0.12.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-6290.1.patch.txt, HIVE-6290.2.patch.txt, HIVE-6290.3.patch.txt Add support for filters to be provided via the composite key class -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6539) Couple of issues in fs based stats collection
[
https://issues.apache.org/jira/browse/HIVE-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918732#comment-13918732
]
Hive QA commented on HIVE-6539:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12632345/HIVE-6539.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5236 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hive.hcatalog.mapreduce.TestHCatMutablePartitioned.testHCatPartitionedTable
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1607/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1607/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12632345
Couple of issues in fs based stats collection
-
Key: HIVE-6539
URL: https://issues.apache.org/jira/browse/HIVE-6539
Project: Hive
Issue Type: Bug
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-6539.patch
While testing on cluster found couple of bugs:
* NPE in certain case.
* map object reuse causing problem
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 17483: Support for hbase filters for composite keys
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17483/ --- (Updated March 3, 2014, 11:40 p.m.) Review request for hive. Changes --- Updating with latest patch. Bugs: HIVE-6290 https://issues.apache.org/jira/browse/HIVE-6290 Repository: hive-git Description --- Support to be able to provide scan filter for hive/hbase integration for composite keys. Diffs (updated) - hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java 5008f15 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 8cd594b hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 704fcb9 hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java 13c344b ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java d39ee2e ql/src/java/org/apache/hadoop/hive/ql/index/IndexSearchCondition.java 5f1329c ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java e50026b Diff: https://reviews.apache.org/r/17483/diff/ Testing --- Tested using locally created data and debug logs that the filter gets applied on the scan. Thanks, Swarnim Kulkarni
