[jira] [Updated] (HIVE-8456) Support Hive Counter to collect spark job metric[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-8456: Attachment: HIVE-8456.3-spark.patch set javaSparkContext and hiveConf as transient variables, as they are only used in driver side. Support Hive Counter to collect spark job metric[Spark Branch] -- Key: HIVE-8456 URL: https://issues.apache.org/jira/browse/HIVE-8456 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Chengxiang Li Labels: Spark-M3 Attachments: HIVE-8456.1-spark.patch, HIVE-8456.2-spark.patch, HIVE-8456.3-spark.patch Several Hive query metric in Hive operators is collected by Hive Counter, such as CREATEDFILES and DESERIALIZE_ERRORS, Besides, Hive use Counter as an option to collect table stats info. Spark support Accumulator which is pretty similiar with Hive Counter, we could try to enable Hive Counter based on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8456) Support Hive Counter to collect spark job metric[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173414#comment-14173414 ] Rui Li commented on HIVE-8456: -- [~chengxiang li] - thanks for the explanation! I agree we don't have to identify all the needed counters for now. For #3, I don't see hive create counters for those enums. So do you mean it's an improvement to add counters for them on spark? Support Hive Counter to collect spark job metric[Spark Branch] -- Key: HIVE-8456 URL: https://issues.apache.org/jira/browse/HIVE-8456 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Chengxiang Li Labels: Spark-M3 Attachments: HIVE-8456.1-spark.patch, HIVE-8456.2-spark.patch, HIVE-8456.3-spark.patch Several Hive query metric in Hive operators is collected by Hive Counter, such as CREATEDFILES and DESERIALIZE_ERRORS, Besides, Hive use Counter as an option to collect table stats info. Spark support Accumulator which is pretty similiar with Hive Counter, we could try to enable Hive Counter based on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8482) hive HashTableLoader error
q79969786 created HIVE-8482: --- Summary: hive HashTableLoader error Key: HIVE-8482 URL: https://issues.apache.org/jira/browse/HIVE-8482 Project: Hive Issue Type: Bug Reporter: q79969786 java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.FileNotFoundException: /data/data01/data/tmp/hadoop-hadoop/mapred/local/taskTracker/hduser0401/distcache/4554609282698230126_-1275563112_394094114/CNSH041566.app.paic.com.cn/tmp/hduser0401/hive_2014-10-16_13-42-26_451_2739170706811325303-1/-mr-10024/HashTable-Stage-31/Stage-31.tar.gz/MapJoin-mapfile21--.hashtable (No such file or directory) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.FileNotFoundException: /data/data01/data/tmp/hadoop-hadoop/mapred/local/taskTracker/hduser0401/distcache/4554609282698230126_-1275563112_394094114/CNSH041566.app.paic.com.cn/tmp/hduser0401/hive_2014-10-16_13-42-26_451_2739170706811325303-1/-mr-10024/HashTable-Stage-31/Stage-31.tar.gz/MapJoin-mapfile21--.hashtable (No such file or directory) at org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:104) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:152) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:178) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1029) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1033) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1033) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:505) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177) ... 8 more Caused by: java.io.FileNotFoundException: /data/data01/data/tmp/hadoop-hadoop/mapred/local/taskTracker/hduser0401/distcache/4554609282698230126_-1275563112_394094114/CNSH041566.app.paic.com.cn/tmp/hduser0401/hive_2014-10-16_13-42-26_451_2739170706811325303-1/-mr-10024/HashTable-Stage-31/Stage-31.tar.gz/MapJoin-mapfile21--.hashtable (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.init(FileInputStream.java:120) at java.io.FileInputStream.init(FileInputStream.java:79) at org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:95) ... 15 more FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-2573) Create per-session function registry
[
https://issues.apache.org/jira/browse/HIVE-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173431#comment-14173431
]
Hive QA commented on HIVE-2573:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12674959/HIVE-2573.7.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 6559 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1292/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1292/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1292/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12674959
- PreCommit-HIVE-TRUNK-Build
Create per-session function registry
-
Key: HIVE-2573
URL: https://issues.apache.org/jira/browse/HIVE-2573
Project: Hive
Issue Type: Improvement
Components: Server Infrastructure
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2573.D3231.1.patch,
HIVE-2573.1.patch.txt, HIVE-2573.2.patch.txt, HIVE-2573.3.patch.txt,
HIVE-2573.4.patch.txt, HIVE-2573.5.patch, HIVE-2573.6.patch, HIVE-2573.7.patch
Currently the function registry is shared resource and could be overrided by
other users when using HiveServer. If per-session function registry is
provided, this situation could be prevented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8456) Support Hive Counter to collect spark job metric[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173437#comment-14173437 ] Chengxiang Li commented on HIVE-8456: - I haven't look into the details for these enums, we can register these counters while enable the feature depends on these enums. I only submit Spark counter implementation here, and left the usage of spark counter in the following tickets. Support Hive Counter to collect spark job metric[Spark Branch] -- Key: HIVE-8456 URL: https://issues.apache.org/jira/browse/HIVE-8456 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Chengxiang Li Labels: Spark-M3 Attachments: HIVE-8456.1-spark.patch, HIVE-8456.2-spark.patch, HIVE-8456.3-spark.patch Several Hive query metric in Hive operators is collected by Hive Counter, such as CREATEDFILES and DESERIALIZE_ERRORS, Besides, Hive use Counter as an option to collect table stats info. Spark support Accumulator which is pretty similiar with Hive Counter, we could try to enable Hive Counter based on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8456) Support Hive Counter to collect spark job metric[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173443#comment-14173443 ] Rui Li commented on HIVE-8456: -- I see. That makes sense. +1 The patch looks good to me. Support Hive Counter to collect spark job metric[Spark Branch] -- Key: HIVE-8456 URL: https://issues.apache.org/jira/browse/HIVE-8456 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Chengxiang Li Labels: Spark-M3 Attachments: HIVE-8456.1-spark.patch, HIVE-8456.2-spark.patch, HIVE-8456.3-spark.patch Several Hive query metric in Hive operators is collected by Hive Counter, such as CREATEDFILES and DESERIALIZE_ERRORS, Besides, Hive use Counter as an option to collect table stats info. Spark support Accumulator which is pretty similiar with Hive Counter, we could try to enable Hive Counter based on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8406) Research on skewed join [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173449#comment-14173449 ] Rui Li commented on HIVE-8406: -- Skew join optimization depends on map join. Research on skewed join [Spark Branch] -- Key: HIVE-8406 URL: https://issues.apache.org/jira/browse/HIVE-8406 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Xuefu Zhang Assignee: Rui Li Research on how to handle skewed join for hive on spark. Here is original hive's design doc for skewed join, https://cwiki.apache.org/confluence/display/Hive/Skewed+Join+Optimization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8320) Error in MetaException(message:Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out)
[ https://issues.apache.org/jira/browse/HIVE-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gavin kim updated HIVE-8320: Attachment: (was: 0001-make-to-synchronize-hiveserver2-session-s-metastore-.patch) Error in MetaException(message:Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out) -- Key: HIVE-8320 URL: https://issues.apache.org/jira/browse/HIVE-8320 Project: Hive Issue Type: Improvement Components: HiveServer2 Affects Versions: 0.13.1 Reporter: gavin kim Assignee: gavin kim Priority: Minor Labels: patch Fix For: 0.13.1 Attachments: HIVE-8320.1.patch I'm using Hive 13.1 in cdh environment. Using hue's beeswax, sometimes hiveserver2 occur MetaException. And after that, hive meta data request timed out. error log's detail is below. 2014-09-29 12:05:44,829 ERROR hive.log: Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:600) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:587) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:826) at org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:62) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:562) at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:315) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:493) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60) at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:273) at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:402) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1429) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1414) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at
[jira] [Updated] (HIVE-8320) Error in MetaException(message:Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out)
[ https://issues.apache.org/jira/browse/HIVE-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gavin kim updated HIVE-8320: Attachment: HIVE-8320.2.patch I'm done my code. But, I'm confused how to use review board. Is it right posted in review board automatically when i submit patch in jira?? Error in MetaException(message:Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out) -- Key: HIVE-8320 URL: https://issues.apache.org/jira/browse/HIVE-8320 Project: Hive Issue Type: Improvement Components: HiveServer2 Affects Versions: 0.13.1 Reporter: gavin kim Assignee: gavin kim Priority: Minor Labels: patch Fix For: 0.13.1 Attachments: HIVE-8320.1.patch, HIVE-8320.2.patch I'm using Hive 13.1 in cdh environment. Using hue's beeswax, sometimes hiveserver2 occur MetaException. And after that, hive meta data request timed out. error log's detail is below. 2014-09-29 12:05:44,829 ERROR hive.log: Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:600) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:587) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:826) at org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:62) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:562) at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:315) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:493) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60) at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:273) at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:402) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1429) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1414) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at
[jira] [Commented] (HIVE-8467) Table Copy - Background, incremental data load
[ https://issues.apache.org/jira/browse/HIVE-8467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173455#comment-14173455 ] Gunther Hagleitner commented on HIVE-8467: -- Materialized views don't necessarily have to keep the tables in sync, do they? Other vendors allow deferred refreshes and for the user to specify integrity levels. I.e.: You can still put the onus on the user and you don't necessarily have to offer a background sync method (you can choose to additional options later.) As far as other engines go - you have the same problem right? You can expose the table copy or view, but the smarts how and when to rewrite queries has to be built into each of those, or left to the user. With materialized views, other engines will also know how the tables are derived, which seems beneficial (well, if they speak SQL at least). For Pig and MR you will likely have to bake assumptions into the scripts/code. Could say more about retention policy, max size and in general how you have seen ppl choose which partitions to add to the table copy? Is it typically the newest n partition? Or the last month of data? That'd be interesting - to see if it can be mapped on materialized views and how hard it'd be for the CBO can handle it. Table Copy - Background, incremental data load -- Key: HIVE-8467 URL: https://issues.apache.org/jira/browse/HIVE-8467 Project: Hive Issue Type: New Feature Reporter: Rajat Venkatesh Attachments: Table Copies.pdf Traditionally, Hive and other tools in the Hadoop eco-system havent required a load stage. However, with recent developments, Hive is much more performant when data is stored in specific formats like ORC, Parquet, Avro etc. Technologies like Presto, also work much better with certain data formats. At the same time, data is generated or obtained from 3rd parties in non-optimal formats such as CSV, tab-limited or JSON. Many a times, its not an option to change the data format at the source. We've found that users either use sub-optimal formats or spend a large amount of effort creating and maintaining copies. We want to propose a new construct - Table Copy - to help “load” data into an optimal storage format. I am going to attach a PDF document with a lot more details especially addressing how is this different from bulk loads in relational DBs or materialized views. Looking forward to hear if others see a similar need to formalize conversion of data to different storage formats. If yes, are the details in the PDF document a good start ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8320) Error in MetaException(message:Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out)
[ https://issues.apache.org/jira/browse/HIVE-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gavin kim updated HIVE-8320: Status: Patch Available (was: Open) Error in MetaException(message:Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out) -- Key: HIVE-8320 URL: https://issues.apache.org/jira/browse/HIVE-8320 Project: Hive Issue Type: Improvement Components: HiveServer2 Affects Versions: 0.13.1 Reporter: gavin kim Assignee: gavin kim Priority: Minor Labels: patch Fix For: 0.13.1 Attachments: HIVE-8320.1.patch, HIVE-8320.2.patch I'm using Hive 13.1 in cdh environment. Using hue's beeswax, sometimes hiveserver2 occur MetaException. And after that, hive meta data request timed out. error log's detail is below. 2014-09-29 12:05:44,829 ERROR hive.log: Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:600) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:587) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:826) at org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:62) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:562) at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:315) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:493) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60) at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:273) at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:402) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1429) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1414) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at
[jira] [Commented] (HIVE-8320) Error in MetaException(message:Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out)
[ https://issues.apache.org/jira/browse/HIVE-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173463#comment-14173463 ] gavin kim commented on HIVE-8320: - I posted patch manullay. The url is https://reviews.apache.org/r/26805/ How can i get that there is no problem in code convention?? Error in MetaException(message:Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out) -- Key: HIVE-8320 URL: https://issues.apache.org/jira/browse/HIVE-8320 Project: Hive Issue Type: Improvement Components: HiveServer2 Affects Versions: 0.13.1 Reporter: gavin kim Assignee: gavin kim Priority: Minor Labels: patch Fix For: 0.13.1 Attachments: HIVE-8320.1.patch, HIVE-8320.2.patch I'm using Hive 13.1 in cdh environment. Using hue's beeswax, sometimes hiveserver2 occur MetaException. And after that, hive meta data request timed out. error log's detail is below. 2014-09-29 12:05:44,829 ERROR hive.log: Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:600) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:587) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:826) at org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:62) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:562) at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:315) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:493) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60) at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:273) at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:402) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1429) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1414) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at
Review Request 26805: HIVE-8320: edit hiveserver2 session's metastore client to use ThreadLocal client
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26805/ --- Review request for hive and Thejas Nair. Bugs: HIVE-8320 https://issues.apache.org/jira/browse/HIVE-8320 Repository: hive-git Description --- HIVE-8320: edit hiveserver2 session's metastore client to use ThreadLocal client Diffs - service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java a9d5902 Diff: https://reviews.apache.org/r/26805/diff/ Testing --- Thanks, Gavin Kim
[jira] [Updated] (HIVE-8465) Fix some minor test fails on trunk
[ https://issues.apache.org/jira/browse/HIVE-8465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-8465: Description: org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parallel -- due to HIVE-2828 org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2 org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3 was: org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parallel Fix some minor test fails on trunk -- Key: HIVE-8465 URL: https://issues.apache.org/jira/browse/HIVE-8465 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Jason Dere Priority: Minor Attachments: HIVE-8465.1.patch.txt, HIVE-8465.2.patch org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parallel -- due to HIVE-2828 org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2 org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-2828) make timestamp accessible in the hbase KeyValue
[
https://issues.apache.org/jira/browse/HIVE-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173468#comment-14173468
]
Navis commented on HIVE-2828:
-
My bad. Some codes not in the patch is included by mistake, failing
hbase_custom_key* tests. I'll merge patch for that with HIVE-8465.
make timestamp accessible in the hbase KeyValue
Key: HIVE-2828
URL: https://issues.apache.org/jira/browse/HIVE-2828
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Reporter: Navis
Assignee: Navis
Priority: Trivial
Fix For: 0.15.0
Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.1.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.2.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.3.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.4.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.5.patch, HIVE-2828.6.patch.txt,
HIVE-2828.7.patch.txt, HIVE-2828.8.patch.txt, HIVE-2828.9.patch.txt
Originated from HIVE-2781 and not accepted, but I think this could be helpful
to someone.
By using special column notation ':timestamp' in HBASE_COLUMNS_MAPPING, user
might access timestamp value in hbase KeyValue.
{code}
CREATE TABLE hbase_table (key int, value string, time timestamp)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf:string,:timestamp)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8320) Error in MetaException(message:Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out)
[ https://issues.apache.org/jira/browse/HIVE-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gavin kim updated HIVE-8320: Attachment: 0001-make-to-synchronize-hiveserver2-session-s-metastore-.patch Error in MetaException(message:Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out) -- Key: HIVE-8320 URL: https://issues.apache.org/jira/browse/HIVE-8320 Project: Hive Issue Type: Improvement Components: HiveServer2 Affects Versions: 0.13.1 Reporter: gavin kim Assignee: gavin kim Priority: Minor Labels: patch Fix For: 0.13.1 Attachments: 0001-make-to-synchronize-hiveserver2-session-s-metastore-.patch, HIVE-8320.1.patch, HIVE-8320.2.patch I'm using Hive 13.1 in cdh environment. Using hue's beeswax, sometimes hiveserver2 occur MetaException. And after that, hive meta data request timed out. error log's detail is below. 2014-09-29 12:05:44,829 ERROR hive.log: Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:600) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:587) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:826) at org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:62) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:562) at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:315) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:493) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60) at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:273) at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:402) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1429) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1414) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at
[jira] [Updated] (HIVE-2828) make timestamp accessible in the hbase KeyValue
[
https://issues.apache.org/jira/browse/HIVE-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-2828:
-
Labels: TODOC15 (was: )
make timestamp accessible in the hbase KeyValue
Key: HIVE-2828
URL: https://issues.apache.org/jira/browse/HIVE-2828
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Reporter: Navis
Assignee: Navis
Priority: Trivial
Labels: TODOC15
Fix For: 0.15.0
Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.1.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.2.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.3.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.4.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.5.patch, HIVE-2828.6.patch.txt,
HIVE-2828.7.patch.txt, HIVE-2828.8.patch.txt, HIVE-2828.9.patch.txt
Originated from HIVE-2781 and not accepted, but I think this could be helpful
to someone.
By using special column notation ':timestamp' in HBASE_COLUMNS_MAPPING, user
might access timestamp value in hbase KeyValue.
{code}
CREATE TABLE hbase_table (key int, value string, time timestamp)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf:string,:timestamp)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8465) Fix some minor test fails on trunk
[ https://issues.apache.org/jira/browse/HIVE-8465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-8465: Attachment: HIVE-8465.3.patch.txt Fix some minor test fails on trunk -- Key: HIVE-8465 URL: https://issues.apache.org/jira/browse/HIVE-8465 Project: Hive Issue Type: Task Components: Tests Reporter: Navis Assignee: Jason Dere Priority: Minor Attachments: HIVE-8465.1.patch.txt, HIVE-8465.2.patch, HIVE-8465.3.patch.txt org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parallel -- due to HIVE-2828 org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2 org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8466) nonReserved keywords can not be used as table alias
[ https://issues.apache.org/jira/browse/HIVE-8466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173482#comment-14173482 ] Navis commented on HIVE-8466: - TestSchemaTool is not related to this and missing_overwrite just needs an update of golden file. But *dynamic_partition_pruning tests are showing a bug in trunk, not recognizing alias of join source. nonReserved keywords can not be used as table alias --- Key: HIVE-8466 URL: https://issues.apache.org/jira/browse/HIVE-8466 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.12.0, 0.13.0, 0.13.1 Reporter: cw Priority: Minor Attachments: HIVE-8466.1.patch There is a small mistake in the patch of issue HIVE-2906. See the change of FromClauseParser.g -: tabname=tableName (ts=tableSample)? (KW_AS? alias=identifier)? -- ^(TOK_TABREF $tabname $ts? $alias?) +: tabname=tableName (props=tableProperties)? (ts=tableSample)? (KW_AS? alias=Identifier)? +- ^(TOK_TABREF $tabname $props? $ts? $alias?) With the 'identifier' changed to 'Identifier' we can not use nonReserved keywords as table alias. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-2828) make timestamp accessible in the hbase KeyValue
[
https://issues.apache.org/jira/browse/HIVE-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173481#comment-14173481
]
Lefty Leverenz commented on HIVE-2828:
--
Doc note: This should be documented in the HBase Integration wikidoc, with
version information, and a release note would be nice.
* [HBase Integration |
https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration]
make timestamp accessible in the hbase KeyValue
Key: HIVE-2828
URL: https://issues.apache.org/jira/browse/HIVE-2828
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Reporter: Navis
Assignee: Navis
Priority: Trivial
Labels: TODOC15
Fix For: 0.15.0
Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.1.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.2.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.3.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.4.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.5.patch, HIVE-2828.6.patch.txt,
HIVE-2828.7.patch.txt, HIVE-2828.8.patch.txt, HIVE-2828.9.patch.txt
Originated from HIVE-2781 and not accepted, but I think this could be helpful
to someone.
By using special column notation ':timestamp' in HBASE_COLUMNS_MAPPING, user
might access timestamp value in hbase KeyValue.
{code}
CREATE TABLE hbase_table (key int, value string, time timestamp)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf:string,:timestamp)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 26481: HIVE-8411: Support partial partition spec for certain ALTER PARTITION statements
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26481/#review56905 --- ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java https://reviews.apache.org/r/26481/#comment97340 there's a flag for enabling dynamic partitioning. is this checked somewhere in this path? ppl use that to avoid someone accidentally changing everything. ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java https://reviews.apache.org/r/26481/#comment97339 why does this fail? it seems more consistent to just alter 0 partitions in this case. or at least just print a warning instead of failing. ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java https://reviews.apache.org/r/26481/#comment97341 LOG.error? ql/src/test/queries/clientpositive/alter_partition_change_col.q https://reviews.apache.org/r/26481/#comment97342 does this support multi level partitioning? if so can you specify only some of the partitions? ql/src/test/queries/clientpositive/alter_partition_change_col.q https://reviews.apache.org/r/26481/#comment97343 add a query to verify the output? - Gunther Hagleitner On Oct. 9, 2014, 3:14 a.m., Jason Dere wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26481/ --- (Updated Oct. 9, 2014, 3:14 a.m.) Review request for hive, Gunther Hagleitner and Thejas Nair. Bugs: HIVE-8411 https://issues.apache.org/jira/browse/HIVE-8411 Repository: hive-git Description --- Allow partial partition spec for some alter table statements, such as: alter table alter_partition_change_col1 partition (p1) change column c2 c2 decimal(10,0); Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java d5374bc ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 4e58ad8 ql/src/test/queries/clientpositive/alter_partition_change_col.q baabb9f ql/src/test/results/clientpositive/alter_partition_change_col.q.out 7123e40 Diff: https://reviews.apache.org/r/26481/diff/ Testing --- updated qfile test Thanks, Jason Dere
[jira] [Commented] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173485#comment-14173485
]
Hive QA commented on HIVE-8448:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12675081/HIVE-8448.4.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 6560 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1293/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1293/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1293/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12675081
- PreCommit-HIVE-TRUNK-Build
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
Attachments: HIVE-8448.4.patch
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8411) Support partial partition spec for certain ALTER PARTITION statements
[ https://issues.apache.org/jira/browse/HIVE-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173484#comment-14173484 ] Gunther Hagleitner commented on HIVE-8411: -- Some comments on rb. Otherwise +1 Support partial partition spec for certain ALTER PARTITION statements - Key: HIVE-8411 URL: https://issues.apache.org/jira/browse/HIVE-8411 Project: Hive Issue Type: Bug Reporter: Jason Dere Assignee: Jason Dere Fix For: 0.14.0 Attachments: HIVE-8411.1.patch To help address concerns [~hagleitn] had about having to update many partitions [here|https://issues.apache.org/jira/browse/HIVE-7971?focusedCommentId=14131948page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14131948] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8341) Transaction information in config file can grow excessively large
[ https://issues.apache.org/jira/browse/HIVE-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173494#comment-14173494 ] Gunther Hagleitner commented on HIVE-8341: -- [~alangates] could you comment on the test failures. At least testScriptOperatorEvnVarsProcessing seems related. Also, do you have a reviewboard entry for this? Patch looks good though. I think the blacklist is a good approach. Will test tomorrow. Transaction information in config file can grow excessively large - Key: HIVE-8341 URL: https://issues.apache.org/jira/browse/HIVE-8341 Project: Hive Issue Type: Bug Components: Transactions Affects Versions: 0.14.0 Reporter: Alan Gates Assignee: Alan Gates Priority: Critical Attachments: HIVE-8341.2.patch, HIVE-8341.patch In our testing we have seen cases where the transaction list grows very large. We need a more efficient way of communicating the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8466) nonReserved keywords can not be used as table alias
[ https://issues.apache.org/jira/browse/HIVE-8466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-8466: Attachment: HIVE-8466.2.patch.txt nonReserved keywords can not be used as table alias --- Key: HIVE-8466 URL: https://issues.apache.org/jira/browse/HIVE-8466 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.12.0, 0.13.0, 0.13.1 Reporter: cw Priority: Minor Attachments: HIVE-8466.1.patch, HIVE-8466.2.patch.txt There is a small mistake in the patch of issue HIVE-2906. See the change of FromClauseParser.g -: tabname=tableName (ts=tableSample)? (KW_AS? alias=identifier)? -- ^(TOK_TABREF $tabname $ts? $alias?) +: tabname=tableName (props=tableProperties)? (ts=tableSample)? (KW_AS? alias=Identifier)? +- ^(TOK_TABREF $tabname $props? $ts? $alias?) With the 'identifier' changed to 'Identifier' we can not use nonReserved keywords as table alias. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7156) Group-By operator stat-annotation only uses distinct approx to generate rollups
[
https://issues.apache.org/jira/browse/HIVE-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-7156:
-
Labels: (was: TODOC14)
Group-By operator stat-annotation only uses distinct approx to generate
rollups
---
Key: HIVE-7156
URL: https://issues.apache.org/jira/browse/HIVE-7156
Project: Hive
Issue Type: Sub-task
Affects Versions: 0.14.0
Reporter: Gopal V
Assignee: Prasanth J
Priority: Blocker
Fix For: 0.14.0
Attachments: HIVE-7156.1.patch, HIVE-7156.2.patch, HIVE-7156.3.patch,
HIVE-7156.4.patch, HIVE-7156.5.patch, HIVE-7156.6.patch, HIVE-7156.7.patch,
HIVE-7156.8.patch, HIVE-7156.8.patch, HIVE-7156.9.patch, hive-debug.log.bz2
The stats annotation for a group-by only annotates the reduce-side row-count
with the distinct values.
The map-side gets the row-count as the rows output instead of distinct *
parallelism, while the reducer side gets the correct parallelism.
{code}
hive explain select distinct L_SHIPDATE from lineitem;
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: lineitem
Statistics: Num rows: 589709 Data size: 4745677733354
Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: l_shipdate (type: string)
outputColumnNames: l_shipdate
Statistics: Num rows: 589709 Data size: 4745677733354
Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
keys: l_shipdate (type: string)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 589709 Data size:
563999032646 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 589709 Data size:
563999032646 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: vectorized
Reducer 2
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1955 Data size: 183770 Basic stats:
COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1955 Data size: 183770 Basic stats:
COMPLETE Column stats: COMPLETE
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7156) Group-By operator stat-annotation only uses distinct approx to generate rollups
[
https://issues.apache.org/jira/browse/HIVE-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173506#comment-14173506
]
Lefty Leverenz commented on HIVE-7156:
--
Doc done: Removal of *hive.stats.map.parallelism* in 0.14 is noted in two
places in the wiki:
* [Configuration Properties -- hive.stats.map.parallelism |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.stats.map.parallelism]
* [Configuration Properties -- Tez (list of related parameters) |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Tez]
Group-By operator stat-annotation only uses distinct approx to generate
rollups
---
Key: HIVE-7156
URL: https://issues.apache.org/jira/browse/HIVE-7156
Project: Hive
Issue Type: Sub-task
Affects Versions: 0.14.0
Reporter: Gopal V
Assignee: Prasanth J
Priority: Blocker
Fix For: 0.14.0
Attachments: HIVE-7156.1.patch, HIVE-7156.2.patch, HIVE-7156.3.patch,
HIVE-7156.4.patch, HIVE-7156.5.patch, HIVE-7156.6.patch, HIVE-7156.7.patch,
HIVE-7156.8.patch, HIVE-7156.8.patch, HIVE-7156.9.patch, hive-debug.log.bz2
The stats annotation for a group-by only annotates the reduce-side row-count
with the distinct values.
The map-side gets the row-count as the rows output instead of distinct *
parallelism, while the reducer side gets the correct parallelism.
{code}
hive explain select distinct L_SHIPDATE from lineitem;
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: lineitem
Statistics: Num rows: 589709 Data size: 4745677733354
Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: l_shipdate (type: string)
outputColumnNames: l_shipdate
Statistics: Num rows: 589709 Data size: 4745677733354
Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
keys: l_shipdate (type: string)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 589709 Data size:
563999032646 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 589709 Data size:
563999032646 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: vectorized
Reducer 2
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1955 Data size: 183770 Basic stats:
COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1955 Data size: 183770 Basic stats:
COMPLETE Column stats: COMPLETE
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6715) Hive JDBC should include username into open session request for non-sasl connection
[
https://issues.apache.org/jira/browse/HIVE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173529#comment-14173529
]
Hive QA commented on HIVE-6715:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12675019/HIVE-6715.3.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 6560 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1294/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1294/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1294/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12675019
- PreCommit-HIVE-TRUNK-Build
Hive JDBC should include username into open session request for non-sasl
connection
---
Key: HIVE-6715
URL: https://issues.apache.org/jira/browse/HIVE-6715
Project: Hive
Issue Type: Bug
Components: JDBC
Reporter: Srinath
Assignee: Prasad Mujumdar
Priority: Critical
Fix For: 0.14.0
Attachments: HIVE-6715.1.patch, HIVE-6715.2.patch, HIVE-6715.3.patch
The only parameter from sessVars that's being set in
HiveConnection.openSession() is HS2_PROXY_USER.
HIVE_AUTH_USER must also be set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8467) Table Copy - Background, incremental data load
[ https://issues.apache.org/jira/browse/HIVE-8467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173534#comment-14173534 ] Rajat Venkatesh commented on HIVE-8467: --- No they dont have to. The databases I know provide both options - sync on user input or automatically. I am not confident we can support automatic sync on external tables. Since it feels like a big feature gap, I chose a different name. Yes - we also have diffs we would like to contribute in other projects to use Table Copy. Since the optimization is at the storage level, its very simple. Replace partitions from the table copy when possible. Directories when it comes to Pig or M/R. If materialized views are chosen, then the optimizers have to mature in more or less lock step. WRT to retention policy, the common case is to only keep the newest n partitions limited by size of the copy. We didnt chose a date range. Sometimes the date partition is not the top level one. This is a moving window. If older partitions are accessed then it will fall back to reading partitions from the Hive Table. Table Copy - Background, incremental data load -- Key: HIVE-8467 URL: https://issues.apache.org/jira/browse/HIVE-8467 Project: Hive Issue Type: New Feature Reporter: Rajat Venkatesh Attachments: Table Copies.pdf Traditionally, Hive and other tools in the Hadoop eco-system havent required a load stage. However, with recent developments, Hive is much more performant when data is stored in specific formats like ORC, Parquet, Avro etc. Technologies like Presto, also work much better with certain data formats. At the same time, data is generated or obtained from 3rd parties in non-optimal formats such as CSV, tab-limited or JSON. Many a times, its not an option to change the data format at the source. We've found that users either use sub-optimal formats or spend a large amount of effort creating and maintaining copies. We want to propose a new construct - Table Copy - to help “load” data into an optimal storage format. I am going to attach a PDF document with a lot more details especially addressing how is this different from bulk loads in relational DBs or materialized views. Looking forward to hear if others see a similar need to formalize conversion of data to different storage formats. If yes, are the details in the PDF document a good start ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8482) set hive.auto.convert.join=true has a bug when create table as select with join
[ https://issues.apache.org/jira/browse/HIVE-8482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] q79969786 updated HIVE-8482: Description: 014-10-16 16:37:22,899 Stage-31 map = 0%, reduce = 0% 2014-10-16 16:38:19,081 Stage-31 map = 100%, reduce = 100% Ended Job = job_201409101641_249763 with errors Error during job, obtaining debugging information... Job Tracking URL: http://CNSH041566.app.paic.com.cn:50030/jobdetails.jsp?user.name=hadoopjobid=job_201409101641_249763 Examining task ID: task_201409101641_249763_m_02 (and more) from job job_201409101641_249763 Task with the most failures(4): - Task ID: task_201409101641_249763_m_00 URL: http://CNSH041566.app.paic.com.cn:50030/taskdetails.jsp?jobid=job_201409101641_249763tipid=task_201409101641_249763_m_00 - Diagnostic Messages for this Task: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.FileNotFoundException: /data/data01/data/tmp/hadoop-hadoop/mapred/local/taskTracker/hduser0401/distcache/-5448139015889594124_-204538_404393650/CNSH041566.app.paic.com.cn/tmp/hduser0401/hive_2014-10-16_16-33-51_335_4584605482989653833-1/-mr-10024/HashTable-Stage-31/Stage-31.tar.gz/MapJoin-mapfile21--.hashtable (No such file or directory) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.FileNotFoundException: /data/data01/data/tmp/hadoop-hadoop/mapred/local/taskTracker/hduser0401/distcache/-5448139015889594124_-204538_404393650/CNSH041566.app.paic.com.cn/tmp/hduser0401/hive_2014-10-16_16-33-51_335_4584605482989653833-1/-mr-10024/HashTable-Stage-31/Stage-31.tar.gz/MapJoin-mapfile21--.hashtable (No such file or directory) at org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:104) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:152) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:178) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1029) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1033) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1033) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:505) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177) ... 8 more Caused by: java.io.FileNotFoundException: /data/data01/data/tmp/hadoop-hadoop/mapred/local/taskTracker/hduser0401/distcache/-5448139015889594124_-204538_404393650/CNSH041566.app.paic.com.cn/tmp/hduser0401/hive_2014-10-16_16-33-51_335_4584605482989653833-1/-mr-10024/HashTable-Stage-31/Stage-31.tar.gz/MapJoin-mapfile21--.hashtable (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.init(FileInputStream.java:120) at java.io.FileInputStream.init(FileInputStream.java:79) at org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:95) ... 15 more FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched: was: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.FileNotFoundException: /data/data01/data/tmp/hadoop-hadoop/mapred/local/taskTracker/hduser0401/distcache/4554609282698230126_-1275563112_394094114/CNSH041566.app.paic.com.cn/tmp/hduser0401/hive_2014-10-16_13-42-26_451_2739170706811325303-1/-mr-10024/HashTable-Stage-31/Stage-31.tar.gz/MapJoin-mapfile21--.hashtable (No such file or directory) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) at
[jira] [Commented] (HIVE-8387) add retry logic to ZooKeeperStorage in WebHCat
[
https://issues.apache.org/jira/browse/HIVE-8387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173574#comment-14173574
]
Hive QA commented on HIVE-8387:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12675058/HIVE-8387.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 6559 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1295/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1295/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1295/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12675058
- PreCommit-HIVE-TRUNK-Build
add retry logic to ZooKeeperStorage in WebHCat
--
Key: HIVE-8387
URL: https://issues.apache.org/jira/browse/HIVE-8387
Project: Hive
Issue Type: Bug
Components: WebHCat
Affects Versions: 0.13.1
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Attachments: HIVE-8387.patch
ZK interactions may run into transient errors that should be retried.
Currently there is no retry logic in WebHCat for this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-7709) Create SparkReporter [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li reassigned HIVE-7709: --- Assignee: Chengxiang Li Create SparkReporter [Spark Branch] --- Key: HIVE-7709 URL: https://issues.apache.org/jira/browse/HIVE-7709 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Chengxiang Li Hive operators use Reporter to collect global information, with Hive on Spark mode, we need a new implementation of Reporter to collect hive operator level information based on spark specified Counter. This task should depends on HIVE-7551. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8483) HiveServer2 can`t correct re-establish metastore communication
Stanilovsky Evgeny created HIVE-8483: Summary: HiveServer2 can`t correct re-establish metastore communication Key: HIVE-8483 URL: https://issues.apache.org/jira/browse/HIVE-8483 Project: Hive Issue Type: Improvement Affects Versions: 0.12.0 Environment: unbuntu 14.04 java 1.7 Reporter: Stanilovsky Evgeny with hive-site.xml config like: property namehive.metastore.uris/name valuethrift://mashine1:9083,thrift://mashine1:9083/value /property log: 2014-10-16 14:04:04,502 INFO hive.metastore (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with URI thrift://mashine1:9083 2014-10-16 14:04:04,515 INFO hive.metastore (HiveMetaStoreClient.java:open(345)) - Connected to metastore. if split current connection, for example with command: iptables -I INPUT -d mashine1 -j REJECT, and try to refresh database list through beeline : show tables; we have an stack like : 2014-10-07 10:14:33,898 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketException: Broken pipe org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_databases(ThriftHiveMetastore.java:498) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:490) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:735) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1425) at com.sun.proxy.$Proxy14.getDatabases(Unknown Source) at org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:73) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:514) at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:284) at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:556) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57) at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:222) at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:360) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8483) HiveServer2 can`t correct re-establish metastore communication
[ https://issues.apache.org/jira/browse/HIVE-8483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanilovsky Evgeny updated HIVE-8483: - Attachment: patch1.patch HiveServer2 can`t correct re-establish metastore communication -- Key: HIVE-8483 URL: https://issues.apache.org/jira/browse/HIVE-8483 Project: Hive Issue Type: Improvement Affects Versions: 0.12.0 Environment: unbuntu 14.04 java 1.7 Reporter: Stanilovsky Evgeny Labels: patch Attachments: patch1.patch with hive-site.xml config like: property namehive.metastore.uris/name valuethrift://mashine1:9083,thrift://mashine1:9083/value /property log: 2014-10-16 14:04:04,502 INFO hive.metastore (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with URI thrift://mashine1:9083 2014-10-16 14:04:04,515 INFO hive.metastore (HiveMetaStoreClient.java:open(345)) - Connected to metastore. if split current connection, for example with command: iptables -I INPUT -d mashine1 -j REJECT, and try to refresh database list through beeline : show tables; we have an stack like : 2014-10-07 10:14:33,898 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketException: Broken pipe org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_databases(ThriftHiveMetastore.java:498) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:490) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:735) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1425) at com.sun.proxy.$Proxy14.getDatabases(Unknown Source) at org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:73) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:514) at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:284) at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:556) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57) at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:222) at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:360) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8483) HiveServer2 can`t correct re-establish metastore communication
[ https://issues.apache.org/jira/browse/HIVE-8483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanilovsky Evgeny updated HIVE-8483: - Description: with hive-site.xml config like: property namehive.metastore.uris/name valuethrift://mashine1:9083,thrift://mashine1:9083/value /property log: 2014-10-16 14:04:04,502 INFO hive.metastore (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with URI thrift://mashine1:9083 2014-10-16 14:04:04,515 INFO hive.metastore (HiveMetaStoreClient.java:open(345)) - Connected to metastore. if split current connection, for example with command: iptables -I INPUT -d mashine1 -j REJECT, and try to refresh database list through hue\beeline : show tables; we have an stack like : 2014-10-07 10:14:33,898 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketException: Broken pipe org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_databases(ThriftHiveMetastore.java:498) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:490) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:735) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1425) at com.sun.proxy.$Proxy14.getDatabases(Unknown Source) at org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:73) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:514) at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:284) at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:556) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57) at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:222) at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:360) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) was: with hive-site.xml config like: property namehive.metastore.uris/name valuethrift://mashine1:9083,thrift://mashine1:9083/value /property log: 2014-10-16 14:04:04,502 INFO hive.metastore (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with URI thrift://mashine1:9083 2014-10-16 14:04:04,515 INFO hive.metastore (HiveMetaStoreClient.java:open(345)) - Connected to metastore. if split current connection, for example with command: iptables -I INPUT -d mashine1 -j REJECT, and try to refresh database list through beeline : show tables; we have an stack like : 2014-10-07 10:14:33,898 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketException: Broken pipe
[jira] [Updated] (HIVE-8483) HiveServer2 can`t correct re-establish metastore communication
[ https://issues.apache.org/jira/browse/HIVE-8483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanilovsky Evgeny updated HIVE-8483: - Description: with hive-site.xml config like: property namehive.metastore.uris/name valuethrift://mashine1:9083,thrift://mashine2:9083/value /property log: 2014-10-16 14:04:04,502 INFO hive.metastore (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with URI thrift://mashine1:9083 2014-10-16 14:04:04,515 INFO hive.metastore (HiveMetaStoreClient.java:open(345)) - Connected to metastore. if split current connection, for example with command: iptables -I INPUT -d mashine1 -j REJECT, and try to refresh database list through hue\beeline : show tables; we have an stack like : 2014-10-07 10:14:33,898 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketException: Broken pipe org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_databases(ThriftHiveMetastore.java:498) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:490) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:735) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1425) at com.sun.proxy.$Proxy14.getDatabases(Unknown Source) at org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:73) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:514) at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:284) at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:556) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57) at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:222) at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:360) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) was: with hive-site.xml config like: property namehive.metastore.uris/name valuethrift://mashine1:9083,thrift://mashine1:9083/value /property log: 2014-10-16 14:04:04,502 INFO hive.metastore (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with URI thrift://mashine1:9083 2014-10-16 14:04:04,515 INFO hive.metastore (HiveMetaStoreClient.java:open(345)) - Connected to metastore. if split current connection, for example with command: iptables -I INPUT -d mashine1 -j REJECT, and try to refresh database list through hue\beeline : show tables; we have an stack like : 2014-10-07 10:14:33,898 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketException: Broken pipe
[jira] [Updated] (HIVE-8483) HiveServer2 can`t correct re-establish metastore communication
[ https://issues.apache.org/jira/browse/HIVE-8483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanilovsky Evgeny updated HIVE-8483: - Status: Patch Available (was: Open) HiveServer2 can`t correct re-establish metastore communication -- Key: HIVE-8483 URL: https://issues.apache.org/jira/browse/HIVE-8483 Project: Hive Issue Type: Improvement Affects Versions: 0.12.0 Environment: unbuntu 14.04 java 1.7 Reporter: Stanilovsky Evgeny Labels: patch Attachments: patch1.patch with hive-site.xml config like: property namehive.metastore.uris/name valuethrift://mashine1:9083,thrift://mashine1:9083/value /property log: 2014-10-16 14:04:04,502 INFO hive.metastore (HiveMetaStoreClient.java:open(257)) - Trying to connect to metastore with URI thrift://mashine1:9083 2014-10-16 14:04:04,515 INFO hive.metastore (HiveMetaStoreClient.java:open(345)) - Connected to metastore. if split current connection, for example with command: iptables -I INPUT -d mashine1 -j REJECT, and try to refresh database list through hue\beeline : show tables; we have an stack like : 2014-10-07 10:14:33,898 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(911)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketException: Broken pipe org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_databases(ThriftHiveMetastore.java:498) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:490) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:735) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1425) at com.sun.proxy.$Proxy14.getDatabases(Unknown Source) at org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:73) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:514) at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:284) at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:556) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57) at com.sun.proxy.$Proxy13.getSchemas(Unknown Source) at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:222) at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:360) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8428) PCR doesnt remove filters involving casts
[
https://issues.apache.org/jira/browse/HIVE-8428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173606#comment-14173606
]
Hive QA commented on HIVE-8428:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12675080/HIVE-8428.3.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 6559 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1296/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1296/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1296/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12675080
- PreCommit-HIVE-TRUNK-Build
PCR doesnt remove filters involving casts
-
Key: HIVE-8428
URL: https://issues.apache.org/jira/browse/HIVE-8428
Project: Hive
Issue Type: Improvement
Components: Logical Optimizer
Affects Versions: 0.11.0, 0.12.0, 0.13.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-8428.1.patch, HIVE-8428.2.patch, HIVE-8428.3.patch,
HIVE-8428.patch
e.g.,
select key,value from srcpart where hr = cast(11 as double);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-8484) HCatalog throws an exception if Pig job is of type 'fetch'
Lorand Bendig created HIVE-8484:
---
Summary: HCatalog throws an exception if Pig job is of type 'fetch'
Key: HIVE-8484
URL: https://issues.apache.org/jira/browse/HIVE-8484
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.14.0
Reporter: Lorand Bendig
Fix For: 0.14.0
When Pig tries to retrieve result in fetch mode through HCatalog then
HCatLoader#setLocation(String location, Job job) can't set the outputschema
because HCatUtil#checkJobContextIfRunningFromBackend(job) always returns false :
{code}
public static boolean checkJobContextIfRunningFromBackend(JobContext j) {
if (j.getConfiguration().get(mapred.task.id, ).equals()
!(true.equals(j.getConfiguration().get(pig.illustrating {
return false;
}
return true;
}
{code}
This is because in fetch mode we don't have a mapred.task.id. A null
outputschema will raise an exception when HCatBaseLoader#getNext() is called:
(ERROR 6018: Error converting read value to tuple).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8471) Cleanup TestSSL
[
https://issues.apache.org/jira/browse/HIVE-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173633#comment-14173633
]
Hive QA commented on HIVE-8471:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12675083/HIVE-8471.patch
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 6559 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithProperty
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithURL
org.apache.hive.jdbc.TestSSL.testSSLFetch
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1297/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1297/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1297/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12675083
- PreCommit-HIVE-TRUNK-Build
Cleanup TestSSL
---
Key: HIVE-8471
URL: https://issues.apache.org/jira/browse/HIVE-8471
Project: Hive
Issue Type: Improvement
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Minor
Attachments: HIVE-8471.patch
conf is static and there are some other minor cleanups
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8484) HCatalog throws an exception if Pig job is of type 'fetch'
[
https://issues.apache.org/jira/browse/HIVE-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lorand Bendig updated HIVE-8484:
Attachment: HIVE-8484.patch
As a solution I'd check for the property 'pig.job.converted.fetch' and if set,
would return true.
HCatalog throws an exception if Pig job is of type 'fetch'
--
Key: HIVE-8484
URL: https://issues.apache.org/jira/browse/HIVE-8484
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.14.0
Reporter: Lorand Bendig
Fix For: 0.14.0
Attachments: HIVE-8484.patch
When Pig tries to retrieve result in fetch mode through HCatalog then
HCatLoader#setLocation(String location, Job job) can't set the outputschema
because HCatUtil#checkJobContextIfRunningFromBackend(job) always returns
false :
{code}
public static boolean checkJobContextIfRunningFromBackend(JobContext j) {
if (j.getConfiguration().get(mapred.task.id, ).equals()
!(true.equals(j.getConfiguration().get(pig.illustrating {
return false;
}
return true;
}
{code}
This is because in fetch mode we don't have a mapred.task.id. A null
outputschema will raise an exception when HCatBaseLoader#getNext() is called:
(ERROR 6018: Error converting read value to tuple).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8484) HCatalog throws an exception if Pig job is of type 'fetch'
[
https://issues.apache.org/jira/browse/HIVE-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lorand Bendig updated HIVE-8484:
Status: Patch Available (was: Open)
HCatalog throws an exception if Pig job is of type 'fetch'
--
Key: HIVE-8484
URL: https://issues.apache.org/jira/browse/HIVE-8484
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.14.0
Reporter: Lorand Bendig
Fix For: 0.14.0
Attachments: HIVE-8484.patch
When Pig tries to retrieve result in fetch mode through HCatalog then
HCatLoader#setLocation(String location, Job job) can't set the outputschema
because HCatUtil#checkJobContextIfRunningFromBackend(job) always returns
false :
{code}
public static boolean checkJobContextIfRunningFromBackend(JobContext j) {
if (j.getConfiguration().get(mapred.task.id, ).equals()
!(true.equals(j.getConfiguration().get(pig.illustrating {
return false;
}
return true;
}
{code}
This is because in fetch mode we don't have a mapred.task.id. A null
outputschema will raise an exception when HCatBaseLoader#getNext() is called:
(ERROR 6018: Error converting read value to tuple).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8395) CBO: enable by default
[
https://issues.apache.org/jira/browse/HIVE-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173679#comment-14173679
]
Hive QA commented on HIVE-8395:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12675104/HIVE-8395.06.patch
{color:red}ERROR:{color} -1 due to 102 failed/errored test(s), 6559 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver_accumulo_predicate_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguous_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_file_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_multiple
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_self_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_unused
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_update
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_compression
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compression
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_stale
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_stale_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merge_multi_expressions
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_partition_metadataonly
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_oneskew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_oneskew_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_oneskew_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mergejoins
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_udf_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_print_header
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_unquote_and
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_unquote_not
[jira] [Created] (HIVE-8485) HMS on Oracle incompatibility
Ryan Pridgeon created HIVE-8485:
---
Summary: HMS on Oracle incompatibility
Key: HIVE-8485
URL: https://issues.apache.org/jira/browse/HIVE-8485
Project: Hive
Issue Type: Bug
Components: Metastore
Environment: Oracle as metastore DB
Reporter: Ryan Pridgeon
Oracle does not distinguish between empty strings and NULL,which proves
problematic for DataNucleus.
In the event a user creates a table with some property stored as an empty
string the table will no longer be accessible.
i.e. TBLPROPERTIES ('serialization.null.format'='')
If they try to select, describe, drop, etc the client prints the following
exception.
ERROR ql.Driver: FAILED: SemanticException [Error 10001]: Table not found
table name
The work around for this was to go into the hive metastore on the Oracle
database and replace NULL with some other string. Users could then drop the
tables or alter their data to use the new null format they just set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8476) JavaDoc updates to HiveEndPoint.newConnection() for secure streaming with Kerberos
[
https://issues.apache.org/jira/browse/HIVE-8476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173730#comment-14173730
]
Hive QA commented on HIVE-8476:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12675131/HIVE-8476.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 6559 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1299/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1299/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1299/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12675131
- PreCommit-HIVE-TRUNK-Build
JavaDoc updates to HiveEndPoint.newConnection() for secure streaming with
Kerberos
--
Key: HIVE-8476
URL: https://issues.apache.org/jira/browse/HIVE-8476
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.13.1
Reporter: Roshan Naik
Assignee: Roshan Naik
Fix For: 0.14.0
Attachments: HIVE-8476.patch
Add additional notes on using kerberos authenticated streaming connection in
HiveEndPoint.newConnection() method
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7898) HCatStorer should ignore namespaces generated by Pig
[ https://issues.apache.org/jira/browse/HIVE-7898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173733#comment-14173733 ] Justin Leet commented on HIVE-7898: --- Anybody willing to review this? https://reviews.apache.org/r/25140/ HCatStorer should ignore namespaces generated by Pig Key: HIVE-7898 URL: https://issues.apache.org/jira/browse/HIVE-7898 Project: Hive Issue Type: Improvement Components: HCatalog Affects Versions: 0.13.1 Reporter: Justin Leet Assignee: Justin Leet Priority: Minor Attachments: HIVE-7898.1.patch Currently, Pig aliases must exactly match the names of HCat columns for HCatStorer to be successful. However, several Pig operations prepend a namespace to the alias in order to differentiate fields (e.g. after a group with field b, you might have A::b). In this case, even if the fields are in the right order and the alias without namespace matches, the store will fail because it tries to match the long form of the alias, despite the namespace being extraneous information in this case. Note that multiple aliases can be applied (e.g. A::B::C::d). A workaround is possible by doing a FOREACH relation GENERATE field1 AS field1, field2 AS field2, etc. This quickly becomes tedious and bloated for tables with many fields. Changing this would normally require care around columns named, for example, `A::b` as has been introduced in Hive 13. However, a different function call only validates Pig aliases if they follow the old rules for Hive columns. As such, a direct change (rather than attempting to match either the namespace::alias or just alias) maintains compatibility for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173744#comment-14173744
]
Yongzhi Chen commented on HIVE-8448:
[~szehon] and [~jdere], I checked the failure tests. 2 of them are new ones,
but it related to login so it should not be caused by the change.
The rest of the failures are aged more than 2 days, and do not have union
operations in the queries. So I think the all the test failures
are not related to the change.
Thanks.
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
Attachments: HIVE-8448.4.patch
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7205) Wrong results when union all of grouping followed by group by with correlation optimization
[
https://issues.apache.org/jira/browse/HIVE-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173804#comment-14173804
]
Yin Huai commented on HIVE-7205:
[~navis] Can you update the review board? I will take a look. Thank you.
Wrong results when union all of grouping followed by group by with
correlation optimization
---
Key: HIVE-7205
URL: https://issues.apache.org/jira/browse/HIVE-7205
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 0.13.0, 0.13.1
Reporter: dima machlin
Assignee: Navis
Priority: Critical
Attachments: HIVE-7205.1.patch.txt, HIVE-7205.2.patch.txt,
HIVE-7205.3.patch.txt, HIVE-7205.4.patch.txt
use case :
table TBL (a string,b string) contains single row : 'a','a'
the following query :
{code:sql}
select b, sum(cc) from (
select b,count(1) as cc from TBL group by b
union all
select a as b,count(1) as cc from TBL group by a
) z
group by b
{code}
returns
a 1
a 1
while set hive.optimize.correlation=true;
if we change set hive.optimize.correlation=false;
it returns correct results : a 2
The plan with correlation optimization :
{code:sql}
ABSTRACT SYNTAX TREE:
(TOK_QUERY (TOK_FROM (TOK_SUBQUERY (TOK_UNION (TOK_QUERY (TOK_FROM
(TOK_TABREF (TOK_TABNAME DB TBL))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR
TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL b)) (TOK_SELEXPR
(TOK_FUNCTION count 1) cc)) (TOK_GROUPBY (TOK_TABLE_OR_COL b (TOK_QUERY
(TOK_FROM (TOK_TABREF (TOK_TABNAME DB TBL))) (TOK_INSERT (TOK_DESTINATION
(TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL a) b)
(TOK_SELEXPR (TOK_FUNCTION count 1) cc)) (TOK_GROUPBY (TOK_TABLE_OR_COL
a) z)) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT
(TOK_SELEXPR (TOK_TABLE_OR_COL b)) (TOK_SELEXPR (TOK_FUNCTION sum
(TOK_TABLE_OR_COL cc (TOK_GROUPBY (TOK_TABLE_OR_COL b
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Alias - Map Operator Tree:
null-subquery1:z-subquery1:TBL
TableScan
alias: TBL
Select Operator
expressions:
expr: b
type: string
outputColumnNames: b
Group By Operator
aggregations:
expr: count(1)
bucketGroup: false
keys:
expr: b
type: string
mode: hash
outputColumnNames: _col0, _col1
Reduce Output Operator
key expressions:
expr: _col0
type: string
sort order: +
Map-reduce partition columns:
expr: _col0
type: string
tag: 0
value expressions:
expr: _col1
type: bigint
null-subquery2:z-subquery2:TBL
TableScan
alias: TBL
Select Operator
expressions:
expr: a
type: string
outputColumnNames: a
Group By Operator
aggregations:
expr: count(1)
bucketGroup: false
keys:
expr: a
type: string
mode: hash
outputColumnNames: _col0, _col1
Reduce Output Operator
key expressions:
expr: _col0
type: string
sort order: +
Map-reduce partition columns:
expr: _col0
type: string
tag: 1
value expressions:
expr: _col1
type: bigint
Reduce Operator Tree:
Demux Operator
Group By Operator
aggregations:
expr: count(VALUE._col0)
bucketGroup: false
keys:
expr: KEY._col0
type: string
mode: mergepartial
outputColumnNames: _col0, _col1
Select Operator
expressions:
expr: _col0
type: string
expr: _col1
type: bigint
outputColumnNames: _col0, _col1
Union
Select Operator
expressions:
[jira] [Updated] (HIVE-8428) PCR doesnt remove filters involving casts
[ https://issues.apache.org/jira/browse/HIVE-8428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8428: --- Fix Version/s: 0.15.0 PCR doesnt remove filters involving casts - Key: HIVE-8428 URL: https://issues.apache.org/jira/browse/HIVE-8428 Project: Hive Issue Type: Improvement Components: Logical Optimizer Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.15.0 Attachments: HIVE-8428.1.patch, HIVE-8428.2.patch, HIVE-8428.3.patch, HIVE-8428.patch e.g., select key,value from srcpart where hr = cast(11 as double); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8428) PCR doesnt remove filters involving casts
[ https://issues.apache.org/jira/browse/HIVE-8428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8428: --- Resolution: Fixed Status: Resolved (was: Patch Available) Cant repro udaf_context_histogram which anyways should not be impacted by this change. Committed to trunk. [~vikram.dixit] This will be good to have in 0.14 as well. PCR doesnt remove filters involving casts - Key: HIVE-8428 URL: https://issues.apache.org/jira/browse/HIVE-8428 Project: Hive Issue Type: Improvement Components: Logical Optimizer Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-8428.1.patch, HIVE-8428.2.patch, HIVE-8428.3.patch, HIVE-8428.patch e.g., select key,value from srcpart where hr = cast(11 as double); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8464) Vectorized reducer nested group by query returns wrong results
[ https://issues.apache.org/jira/browse/HIVE-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173860#comment-14173860 ] Ashutosh Chauhan commented on HIVE-8464: +1 Vectorized reducer nested group by query returns wrong results -- Key: HIVE-8464 URL: https://issues.apache.org/jira/browse/HIVE-8464 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8464.01.patch Query has 2 Group By operators in reduce-stage and VectorGroupByOperator can only handle one Group By operator being fed key batches by reduce-shuffle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8486) TPC-DS Query 96 parallelism is not set correcly
Brock Noland created HIVE-8486: -- Summary: TPC-DS Query 96 parallelism is not set correcly Key: HIVE-8486 URL: https://issues.apache.org/jira/browse/HIVE-8486 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Chao When we run the query on a 20B we only have a parallelism factor of 1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8477) Update timestamp in status console [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-8477:
---
Attachment: HIVE-8477-spark.patch
Update timestamp in status console [Spark Branch]
-
Key: HIVE-8477
URL: https://issues.apache.org/jira/browse/HIVE-8477
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Brock Noland
Assignee: Brock Noland
Attachments: HIVE-8477-spark.patch, HIVE-8477.patch
Note our time versus the log time
{noformat}
2014-10-15 19:18:07,646 INFO [main]: status.SparkJobMonitor
(SessionState.java:printInfo(820)) - 2014-18-15 07:18:07,646 Stage-0_0:
2(+9)/11
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8456) Support Hive Counter to collect spark job metric[Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173925#comment-14173925
]
Hive QA commented on HIVE-8456:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12675227/HIVE-8456.3-spark.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6769 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_tez_smb_1
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/223/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/223/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-223/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12675227
Support Hive Counter to collect spark job metric[Spark Branch]
--
Key: HIVE-8456
URL: https://issues.apache.org/jira/browse/HIVE-8456
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chengxiang Li
Assignee: Chengxiang Li
Labels: Spark-M3
Attachments: HIVE-8456.1-spark.patch, HIVE-8456.2-spark.patch,
HIVE-8456.3-spark.patch
Several Hive query metric in Hive operators is collected by Hive Counter,
such as CREATEDFILES and DESERIALIZE_ERRORS, Besides, Hive use Counter as an
option to collect table stats info. Spark support Accumulator which is
pretty similiar with Hive Counter, we could try to enable Hive Counter based
on it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7366) getDatabase using direct sql
[ https://issues.apache.org/jira/browse/HIVE-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-7366: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks, Sergey, committed to 0.14 and trunk. getDatabase using direct sql Key: HIVE-7366 URL: https://issues.apache.org/jira/browse/HIVE-7366 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.14.0 Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Priority: Critical Fix For: 0.14.0 Attachments: HIVE-7366.2.patch, HIVE-7366.3.patch, HIVE-7366.4.patch, HIVE-7366.patch Given that get_database is easily one of the most frequent calls made on the metastore, we should have the ability to bypass datanucleus for that, and use direct SQL instead. This was something that I did initially as part of debugging HIVE-7368, but I think that given the frequency of this call, it's useful to have it in mainline direct sql. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8389) Fix CBO when indexes are used
[ https://issues.apache.org/jira/browse/HIVE-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174000#comment-14174000 ] Pengcheng Xiong commented on HIVE-8389: --- Done. Thanks [~leftylev] Fix CBO when indexes are used - Key: HIVE-8389 URL: https://issues.apache.org/jira/browse/HIVE-8389 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Labels: TODOC14 Fix For: 0.15.0 Attachments: HIVE-8389.1.patch, HIVE-8389.2.patch, HIVE-8389.3.patch Disable CBO when indexes are used -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8484) HCatalog throws an exception if Pig job is of type 'fetch'
[
https://issues.apache.org/jira/browse/HIVE-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174004#comment-14174004
]
Hive QA commented on HIVE-8484:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12675260/HIVE-8484.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 6559 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1300/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1300/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1300/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12675260
- PreCommit-HIVE-TRUNK-Build
HCatalog throws an exception if Pig job is of type 'fetch'
--
Key: HIVE-8484
URL: https://issues.apache.org/jira/browse/HIVE-8484
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.14.0
Reporter: Lorand Bendig
Fix For: 0.14.0
Attachments: HIVE-8484.patch
When Pig tries to retrieve result in fetch mode through HCatalog then
HCatLoader#setLocation(String location, Job job) can't set the outputschema
because HCatUtil#checkJobContextIfRunningFromBackend(job) always returns
false :
{code}
public static boolean checkJobContextIfRunningFromBackend(JobContext j) {
if (j.getConfiguration().get(mapred.task.id, ).equals()
!(true.equals(j.getConfiguration().get(pig.illustrating {
return false;
}
return true;
}
{code}
This is because in fetch mode we don't have a mapred.task.id. A null
outputschema will raise an exception when HCatBaseLoader#getNext() is called:
(ERROR 6018: Error converting read value to tuple).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-8448:
Resolution: Fixed
Fix Version/s: 0.15.0
Status: Resolved (was: Patch Available)
Makes sense. Committed to trunk, thanks Yongzhi for the contribution and Jason
for additional review !
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
Fix For: 0.15.0
Attachments: HIVE-8448.4.patch
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8477) Update timestamp in status console [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174006#comment-14174006
]
Hive QA commented on HIVE-8477:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12675290/HIVE-8477-spark.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6769 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_tez_smb_1
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/224/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/224/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-224/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12675290
Update timestamp in status console [Spark Branch]
-
Key: HIVE-8477
URL: https://issues.apache.org/jira/browse/HIVE-8477
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Brock Noland
Assignee: Brock Noland
Attachments: HIVE-8477-spark.patch, HIVE-8477.patch
Note our time versus the log time
{noformat}
2014-10-15 19:18:07,646 INFO [main]: status.SparkJobMonitor
(SessionState.java:printInfo(820)) - 2014-18-15 07:18:07,646 Stage-0_0:
2(+9)/11
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8477) Update timestamp in status console [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174012#comment-14174012
]
Xuefu Zhang commented on HIVE-8477:
---
+1
Update timestamp in status console [Spark Branch]
-
Key: HIVE-8477
URL: https://issues.apache.org/jira/browse/HIVE-8477
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Brock Noland
Assignee: Brock Noland
Attachments: HIVE-8477-spark.patch, HIVE-8477.patch
Note our time versus the log time
{noformat}
2014-10-15 19:18:07,646 INFO [main]: status.SparkJobMonitor
(SessionState.java:printInfo(820)) - 2014-18-15 07:18:07,646 Stage-0_0:
2(+9)/11
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8456) Support Hive Counter to collect spark job metric[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-8456: Resolution: Fixed Fix Version/s: spark-branch Status: Resolved (was: Patch Available) Committed to spark, you can begin the work to use these counters. Thanks Chengxiang for the contribution, and Rui for review! Support Hive Counter to collect spark job metric[Spark Branch] -- Key: HIVE-8456 URL: https://issues.apache.org/jira/browse/HIVE-8456 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Chengxiang Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8456.1-spark.patch, HIVE-8456.2-spark.patch, HIVE-8456.3-spark.patch Several Hive query metric in Hive operators is collected by Hive Counter, such as CREATEDFILES and DESERIALIZE_ERRORS, Besides, Hive use Counter as an option to collect table stats info. Spark support Accumulator which is pretty similiar with Hive Counter, we could try to enable Hive Counter based on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8389) Fix CBO when indexes are used
[ https://issues.apache.org/jira/browse/HIVE-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-8389: --- Fix Version/s: (was: 0.15.0) 0.14.0 Fix CBO when indexes are used - Key: HIVE-8389 URL: https://issues.apache.org/jira/browse/HIVE-8389 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Labels: TODOC14 Fix For: 0.14.0 Attachments: HIVE-8389.1.patch, HIVE-8389.2.patch, HIVE-8389.3.patch Disable CBO when indexes are used -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8389) Fix CBO when indexes are used
[ https://issues.apache.org/jira/browse/HIVE-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174033#comment-14174033 ] Sergey Shelukhin commented on HIVE-8389: committed to 14 Fix CBO when indexes are used - Key: HIVE-8389 URL: https://issues.apache.org/jira/browse/HIVE-8389 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Labels: TODOC14 Fix For: 0.14.0 Attachments: HIVE-8389.1.patch, HIVE-8389.2.patch, HIVE-8389.3.patch Disable CBO when indexes are used -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8460) ORC SARG literal creation for double from float may lead to wrong evaluation of SARG
[ https://issues.apache.org/jira/browse/HIVE-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-8460: - Attachment: HIVE-8460.2.patch ORC SARG literal creation for double from float may lead to wrong evaluation of SARG Key: HIVE-8460 URL: https://issues.apache.org/jira/browse/HIVE-8460 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.13.0, 0.13.1 Reporter: Ashutosh Chauhan Assignee: Prasanth J Attachments: HIVE-8460.1.patch, HIVE-8460.2.patch e.g., expression like d = 0.22 where 0.22 is of float type, gets converted to double d = 0.21986 in SARG creation. This will cause erroneous evaluation of SARG expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8460) ORC SARG literal creation for double from float may lead to wrong evaluation of SARG
[ https://issues.apache.org/jira/browse/HIVE-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174086#comment-14174086 ] Ashutosh Chauhan commented on HIVE-8460: +1 ORC SARG literal creation for double from float may lead to wrong evaluation of SARG Key: HIVE-8460 URL: https://issues.apache.org/jira/browse/HIVE-8460 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.13.0, 0.13.1 Reporter: Ashutosh Chauhan Assignee: Prasanth J Attachments: HIVE-8460.1.patch, HIVE-8460.2.patch e.g., expression like d = 0.22 where 0.22 is of float type, gets converted to double d = 0.21986 in SARG creation. This will cause erroneous evaluation of SARG expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8460) ORC SARG literal creation for double from float may lead to wrong evaluation of SARG
[ https://issues.apache.org/jira/browse/HIVE-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174090#comment-14174090 ] Ashutosh Chauhan commented on HIVE-8460: [~vikram.dixit] It will be good to have this in 0.14 as well. ORC SARG literal creation for double from float may lead to wrong evaluation of SARG Key: HIVE-8460 URL: https://issues.apache.org/jira/browse/HIVE-8460 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.13.0, 0.13.1 Reporter: Ashutosh Chauhan Assignee: Prasanth J Attachments: HIVE-8460.1.patch, HIVE-8460.2.patch e.g., expression like d = 0.22 where 0.22 is of float type, gets converted to double d = 0.21986 in SARG creation. This will cause erroneous evaluation of SARG expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8428) PCR doesnt remove filters involving casts
[ https://issues.apache.org/jira/browse/HIVE-8428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174122#comment-14174122 ] Vikram Dixit K commented on HIVE-8428: -- +1 for 0.14 PCR doesnt remove filters involving casts - Key: HIVE-8428 URL: https://issues.apache.org/jira/browse/HIVE-8428 Project: Hive Issue Type: Improvement Components: Logical Optimizer Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.15.0 Attachments: HIVE-8428.1.patch, HIVE-8428.2.patch, HIVE-8428.3.patch, HIVE-8428.patch e.g., select key,value from srcpart where hr = cast(11 as double); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8395) CBO: enable by default
[ https://issues.apache.org/jira/browse/HIVE-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-8395: --- Attachment: HIVE-8395.07.patch Update with changes to out files for recently fixed issues. CBO: enable by default -- Key: HIVE-8395 URL: https://issues.apache.org/jira/browse/HIVE-8395 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.15.0 Attachments: HIVE-8395.01.patch, HIVE-8395.02.patch, HIVE-8395.03.patch, HIVE-8395.04.patch, HIVE-8395.05.patch, HIVE-8395.06.patch, HIVE-8395.07.patch, HIVE-8395.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8487) CBO causes hbase_custom_key tests to fail
Sergey Shelukhin created HIVE-8487:
--
Summary: CBO causes hbase_custom_key tests to fail
Key: HIVE-8487
URL: https://issues.apache.org/jira/browse/HIVE-8487
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Priority: Critical
{noformat}
FAILED: SemanticException java.lang.NullPointerException
org.apache.hadoop.hive.ql.parse.SemanticException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6291)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8899)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8790)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9609)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9503)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9960)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:417)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1070)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1132)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1007)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:997)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:345)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:832)
at
org.apache.hadoop.hive.cli.TestHBaseCliDriver.runTest(TestHBaseCliDriver.java:112)
at
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key(TestHBaseCliDriver.java:94)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.extensions.TestSetup.run(TestSetup.java:27)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: org.apache.hadoop.hive.serde2.SerDeException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.hbase.HBaseSerDeParameters.initKeyFactory(HBaseSerDeParameters.java:178)
at
org.apache.hadoop.hive.hbase.HBaseSerDeParameters.init(HBaseSerDeParameters.java:103)
at
org.apache.hadoop.hive.hbase.HBaseSerDe.initialize(HBaseSerDe.java:93)
at
org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:51)
at
org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:519)
at
org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:84)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6281)
... 42 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.hbase.HBaseSerDeParameters.createKeyFactory(HBaseSerDeParameters.java:186)
at
[jira] [Updated] (HIVE-6715) Hive JDBC should include username into open session request for non-sasl connection
[ https://issues.apache.org/jira/browse/HIVE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6715: Resolution: Fixed Status: Resolved (was: Patch Available) Patch committed to trunk and 0.14 branch. Thanks [~prasadm]! Hive JDBC should include username into open session request for non-sasl connection --- Key: HIVE-6715 URL: https://issues.apache.org/jira/browse/HIVE-6715 Project: Hive Issue Type: Bug Components: JDBC Reporter: Srinath Assignee: Prasad Mujumdar Priority: Critical Fix For: 0.14.0 Attachments: HIVE-6715.1.patch, HIVE-6715.2.patch, HIVE-6715.3.patch The only parameter from sessVars that's being set in HiveConnection.openSession() is HS2_PROXY_USER. HIVE_AUTH_USER must also be set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8488) hash() doesn't match between string and char/varchar
Jason Dere created HIVE-8488:
Summary: hash() doesn't match between string and char/varchar
Key: HIVE-8488
URL: https://issues.apache.org/jira/browse/HIVE-8488
Project: Hive
Issue Type: Bug
Components: UDF
Reporter: Jason Dere
Assignee: Jason Dere
{noformat}
hive select * from tab1;
OK
val_484 val_484 val_484
hive select hash(c1), hash(c2), hash(c3) from tab1;
OK
230901778 1973712113 1973712113
{noformat}
This may throw off users expecting string/varchar/char types to be fairly
interchangeable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8488) hash() doesn't match between string and char/varchar
[
https://issues.apache.org/jira/browse/HIVE-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174151#comment-14174151
]
Jason Dere commented on HIVE-8488:
--
hash() uses ObjectInspectorUtils.hashCode(), which has special case handling
for Text so that the hashcode matches String.hashCode(). This special case
handling should have been done for char/varchar as well.
Note that fixing this will break existing behavior of hash() for char/varchar.
hash() doesn't match between string and char/varchar
Key: HIVE-8488
URL: https://issues.apache.org/jira/browse/HIVE-8488
Project: Hive
Issue Type: Bug
Components: UDF
Reporter: Jason Dere
Assignee: Jason Dere
{noformat}
hive select * from tab1;
OK
val_484 val_484 val_484
hive select hash(c1), hash(c2), hash(c3) from tab1;
OK
230901778 1973712113 1973712113
{noformat}
This may throw off users expecting string/varchar/char types to be fairly
interchangeable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8488) hash() doesn't match between string and char/varchar
[
https://issues.apache.org/jira/browse/HIVE-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-8488:
-
Attachment: HIVE-8488.1.patch
hash() doesn't match between string and char/varchar
Key: HIVE-8488
URL: https://issues.apache.org/jira/browse/HIVE-8488
Project: Hive
Issue Type: Bug
Components: UDF
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-8488.1.patch
{noformat}
hive select * from tab1;
OK
val_484 val_484 val_484
hive select hash(c1), hash(c2), hash(c3) from tab1;
OK
230901778 1973712113 1973712113
{noformat}
This may throw off users expecting string/varchar/char types to be fairly
interchangeable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8488) hash() doesn't match between string and char/varchar
[
https://issues.apache.org/jira/browse/HIVE-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-8488:
-
Status: Patch Available (was: Open)
hash() doesn't match between string and char/varchar
Key: HIVE-8488
URL: https://issues.apache.org/jira/browse/HIVE-8488
Project: Hive
Issue Type: Bug
Components: UDF
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-8488.1.patch
{noformat}
hive select * from tab1;
OK
val_484 val_484 val_484
hive select hash(c1), hash(c2), hash(c3) from tab1;
OK
230901778 1973712113 1973712113
{noformat}
This may throw off users expecting string/varchar/char types to be fairly
interchangeable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8341) Transaction information in config file can grow excessively large
[ https://issues.apache.org/jira/browse/HIVE-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8341: - Status: Open (was: Patch Available) The TestOperators failure is caused by this patch. The rest I believe are unrelated. I'll put up a new version of the patch that addresses the TestOperators failure. Transaction information in config file can grow excessively large - Key: HIVE-8341 URL: https://issues.apache.org/jira/browse/HIVE-8341 Project: Hive Issue Type: Bug Components: Transactions Affects Versions: 0.14.0 Reporter: Alan Gates Assignee: Alan Gates Priority: Critical Attachments: HIVE-8341.2.patch, HIVE-8341.patch In our testing we have seen cases where the transaction list grows very large. We need a more efficient way of communicating the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8341) Transaction information in config file can grow excessively large
[ https://issues.apache.org/jira/browse/HIVE-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8341: - Status: Patch Available (was: Open) Transaction information in config file can grow excessively large - Key: HIVE-8341 URL: https://issues.apache.org/jira/browse/HIVE-8341 Project: Hive Issue Type: Bug Components: Transactions Affects Versions: 0.14.0 Reporter: Alan Gates Assignee: Alan Gates Priority: Critical Attachments: HIVE-8341.2.patch, HIVE-8341.3.patch, HIVE-8341.patch In our testing we have seen cases where the transaction list grows very large. We need a more efficient way of communicating the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8341) Transaction information in config file can grow excessively large
[ https://issues.apache.org/jira/browse/HIVE-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8341: - Attachment: HIVE-8341.3.patch Transaction information in config file can grow excessively large - Key: HIVE-8341 URL: https://issues.apache.org/jira/browse/HIVE-8341 Project: Hive Issue Type: Bug Components: Transactions Affects Versions: 0.14.0 Reporter: Alan Gates Assignee: Alan Gates Priority: Critical Attachments: HIVE-8341.2.patch, HIVE-8341.3.patch, HIVE-8341.patch In our testing we have seen cases where the transaction list grows very large. We need a more efficient way of communicating the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8477) Update timestamp in status console [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-8477:
---
Resolution: Fixed
Fix Version/s: spark-branch
Status: Resolved (was: Patch Available)
Thank you! I have committed to spark.
Update timestamp in status console [Spark Branch]
-
Key: HIVE-8477
URL: https://issues.apache.org/jira/browse/HIVE-8477
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Brock Noland
Assignee: Brock Noland
Fix For: spark-branch
Attachments: HIVE-8477-spark.patch, HIVE-8477.patch
Note our time versus the log time
{noformat}
2014-10-15 19:18:07,646 INFO [main]: status.SparkJobMonitor
(SessionState.java:printInfo(820)) - 2014-18-15 07:18:07,646 Stage-0_0:
2(+9)/11
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8429) Add records in/out counters
[ https://issues.apache.org/jira/browse/HIVE-8429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8429: - Fix Version/s: 0.14.0 Add records in/out counters --- Key: HIVE-8429 URL: https://issues.apache.org/jira/browse/HIVE-8429 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 0.14.0 Attachments: HIVE-8429.1.patch, HIVE-8429.2.patch, HIVE-8429.3.patch, HIVE-8429.4.patch We don't do counters for input/output records right now. That would help for debugging though (if it can be done with minimal overhead). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8429) Add records in/out counters
[ https://issues.apache.org/jira/browse/HIVE-8429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174189#comment-14174189 ] Gunther Hagleitner commented on HIVE-8429: -- Failures are unrelated. [~gopalv] can you take another look? Add records in/out counters --- Key: HIVE-8429 URL: https://issues.apache.org/jira/browse/HIVE-8429 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 0.14.0 Attachments: HIVE-8429.1.patch, HIVE-8429.2.patch, HIVE-8429.3.patch, HIVE-8429.4.patch We don't do counters for input/output records right now. That would help for debugging though (if it can be done with minimal overhead). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8465) Fix some minor test fails on trunk
[
https://issues.apache.org/jira/browse/HIVE-8465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174193#comment-14174193
]
Hive QA commented on HIVE-8465:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12675240/HIVE-8465.3.patch.txt
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1301/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1301/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1301/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-1301/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target
shims/0.20S/target shims/0.23/target shims/aggregator/target
shims/common/target shims/common-secure/target packaging/target
hbase-handler/target testutils/target jdbc/target metastore/target
itests/target itests/hcatalog-unit/target itests/test-serde/target
itests/qtest/target itests/hive-unit/target itests/custom-serde/target
itests/util/target hcatalog/target hcatalog/core/target
hcatalog/streaming/target hcatalog/server-extensions/target
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target
hcatalog/hcatalog-pig-adapter/target accumulo-handler/target hwi/target
common/target common/src/gen contrib/target service/target serde/target
beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update
Ujdbc/src/java/org/apache/hive/jdbc/HiveConnection.java
Aitests/hive-unit/src/test/java/org/apache/hive/jdbc/TestNoSaslAuth.java
Fetching external item into 'hcatalog/src/test/e2e/harness'
Updated external to revision 1632432.
Updated to revision 1632432.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12675240
- PreCommit-HIVE-TRUNK-Build
Fix some minor test fails on trunk
--
Key: HIVE-8465
URL: https://issues.apache.org/jira/browse/HIVE-8465
Project: Hive
Issue Type: Task
Components: Tests
Reporter: Navis
Assignee: Jason Dere
Priority: Minor
Attachments: HIVE-8465.1.patch.txt, HIVE-8465.2.patch,
HIVE-8465.3.patch.txt
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parallel
-- due to HIVE-2828
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6758) Beeline doesn't work with -e option when started in background
[
https://issues.apache.org/jira/browse/HIVE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174204#comment-14174204
]
Mohit Sabharwal commented on HIVE-6758:
---
I looked into it a bit, and here's my understanding:
In beeline, for pretty table formatting (which hive cli does not do), we get
the terminal width (and height) using
jline.Terminal.setupTerminal().getTerminalWidth(), which [sets terminal
properties|http://jline.sourcearchive.com/documentation/0.9.94-5/classjline_1_1UnixTerminal_da5160e7ec686c49b8f39e03d986cc2f.html#da5160e7ec686c49b8f39e03d986cc2f]
by calling stty.
However, when beeline is backgrounded, kernel's terminal driver sends a SIGTTOU
when stty is called, which if not caught, defaults to stopping the process. I
tried ignoring the signal in beeline:
{code}
Signal.handle(new Signal(TTOU), SignalHandler.SIG_IGN);
{code}
This puts beeline process state back to running, though it remains hung and
doesn't resume normal operation. I didn't see anything helpful using jstack.
Not sure what's happening here, will need some more digging.
However, since we don't care about pretty table formatting in background mode,
solution proposed by [~qwertymaniac] and [~brocknoland] also works fine.
Attached patch does that. It essentially sets the terminal type to
[UnsupportedTerminal|http://jline.sourcearchive.com/documentation/0.9.94-5/classjline_1_1UnsupportedTerminal.html]
(as opposed to UnixTerminal) which doesn't call stty.
Alternatively, instead of setting this system property, we could simply not
initialize a Terminal object if the process is backgrounded. Both approaches
result in default terminal height and width values. Am going with beeline
script-only approach.
Would appreciate comments from anyone more familiar with beeline/hive cli.
Thanks.
Beeline doesn't work with -e option when started in background
--
Key: HIVE-6758
URL: https://issues.apache.org/jira/browse/HIVE-6758
Project: Hive
Issue Type: Improvement
Components: CLI
Affects Versions: 0.11.0
Reporter: Johndee Burks
Assignee: Mohit Sabharwal
In hive CLI you could easily integrate its use into a script and back ground
the process like this:
hive -e some query
Beeline does not run when you do the same even with the -f switch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-6758) Beeline doesn't work with -e option when started in background
[ https://issues.apache.org/jira/browse/HIVE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-6758: -- Status: Patch Available (was: Reopened) Beeline doesn't work with -e option when started in background -- Key: HIVE-6758 URL: https://issues.apache.org/jira/browse/HIVE-6758 Project: Hive Issue Type: Improvement Components: CLI Affects Versions: 0.11.0 Reporter: Johndee Burks Assignee: Mohit Sabharwal Attachments: HIVE-6758.patch In hive CLI you could easily integrate its use into a script and back ground the process like this: hive -e some query Beeline does not run when you do the same even with the -f switch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6758) Beeline doesn't work with -e option when started in background
[ https://issues.apache.org/jira/browse/HIVE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-6758: -- Attachment: HIVE-6758.patch Beeline doesn't work with -e option when started in background -- Key: HIVE-6758 URL: https://issues.apache.org/jira/browse/HIVE-6758 Project: Hive Issue Type: Improvement Components: CLI Affects Versions: 0.11.0 Reporter: Johndee Burks Assignee: Mohit Sabharwal Attachments: HIVE-6758.patch In hive CLI you could easily integrate its use into a script and back ground the process like this: hive -e some query Beeline does not run when you do the same even with the -f switch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7858) Parquet compression should be configurable via table property
[
https://issues.apache.org/jira/browse/HIVE-7858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-7858:
---
Release Note: The property parquet.compression can now be configured as a
table property. (was: The property parquet.compression can not be configured
as a table property.)
Parquet compression should be configurable via table property
-
Key: HIVE-7858
URL: https://issues.apache.org/jira/browse/HIVE-7858
Project: Hive
Issue Type: Improvement
Reporter: Brock Noland
Assignee: Ferdinand Xu
Fix For: 0.15.0
Attachments: HIVE-7858.1.patch, HIVE-7858.patch, HIVE-7858.patch
ORC supports the orc.compress table property:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC
{noformat}
create table Addresses (
name string,
street string,
city string,
state string,
zip int
) stored as orc tblproperties (orc.compress=NONE);
{noformat}
I think it'd be great to support the same for Parquet.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7858) Parquet compression should be configurable via table property
[
https://issues.apache.org/jira/browse/HIVE-7858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-7858:
---
Labels: TODOC15 (was: )
We should doc it.. I put a release note The property parquet.compression can
now be configured as a table property.
I think we should put a item here:
https://cwiki.apache.org/confluence/display/Hive/Parquet
* Parquet Table Properties *
with this one. We can update it as we add more..
Parquet compression should be configurable via table property
-
Key: HIVE-7858
URL: https://issues.apache.org/jira/browse/HIVE-7858
Project: Hive
Issue Type: Improvement
Reporter: Brock Noland
Assignee: Ferdinand Xu
Labels: TODOC15
Fix For: 0.15.0
Attachments: HIVE-7858.1.patch, HIVE-7858.patch, HIVE-7858.patch
ORC supports the orc.compress table property:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC
{noformat}
create table Addresses (
name string,
street string,
city string,
state string,
zip int
) stored as orc tblproperties (orc.compress=NONE);
{noformat}
I think it'd be great to support the same for Parquet.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8486) TPC-DS Query 96 parallelism is not set correcly
[
https://issues.apache.org/jira/browse/HIVE-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174223#comment-14174223
]
Chao commented on HIVE-8486:
I just talked with [~szehon] offline, and he pointed me to
{{ReduceSinkOperator::computeHashCode}}. This function returns:
{noformat}
return bucketNumber 0 ? keyHashCode : keyHashCode * 31 + bucketNumber;
{noformat}
So, if the {{bucketNumber}} is 0, and we set the number of partitions to 31,
then all keys will go to the same partition. I think this explains why the
issue only happens only when we set {{mapreduce.job.reduces}} to 31. I also
verified it locally.
TPC-DS Query 96 parallelism is not set correcly
---
Key: HIVE-8486
URL: https://issues.apache.org/jira/browse/HIVE-8486
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Chao
When we run the query on a 20B we only have a parallelism factor of 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 26481: HIVE-8411: Support partial partition spec for certain ALTER PARTITION statements
On Oct. 16, 2014, 7:32 a.m., Gunther Hagleitner wrote: ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java, line 3265 https://reviews.apache.org/r/26481/diff/1/?file=716405#file716405line3265 there's a flag for enabling dynamic partitioning. is this checked somewhere in this path? ppl use that to avoid someone accidentally changing everything. Good point, will add that check. On Oct. 16, 2014, 7:32 a.m., Gunther Hagleitner wrote: ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java, line 3269 https://reviews.apache.org/r/26481/diff/1/?file=716405#file716405line3269 why does this fail? it seems more consistent to just alter 0 partitions in this case. or at least just print a warning instead of failing. This is just the existing behavior right? This is if the user specifies a non-existent partition, including if they specify all partition values in the spec, like (part1='nonexistent', part2='partition'). I think this is sensible behavior, but reply if you think otherwise. We could just fail in the event that the partition spec specifies values for all of the partition columns, if you think that is better. On Oct. 16, 2014, 7:32 a.m., Gunther Hagleitner wrote: ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java, line 3301 https://reviews.apache.org/r/26481/diff/1/?file=716405#file716405line3301 LOG.error? Whoops, will fix. On Oct. 16, 2014, 7:32 a.m., Gunther Hagleitner wrote: ql/src/test/queries/clientpositive/alter_partition_change_col.q, line 60 https://reviews.apache.org/r/26481/diff/1/?file=716407#file716407line60 does this support multi level partitioning? if so can you specify only some of the partitions? You mean like multiple partition columns? I'll change the test to include that. On Oct. 16, 2014, 7:32 a.m., Gunther Hagleitner wrote: ql/src/test/queries/clientpositive/alter_partition_change_col.q, line 63 https://reviews.apache.org/r/26481/diff/1/?file=716407#file716407line63 add a query to verify the output? Will fix. - Jason --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26481/#review56905 --- On Oct. 9, 2014, 3:14 a.m., Jason Dere wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26481/ --- (Updated Oct. 9, 2014, 3:14 a.m.) Review request for hive, Gunther Hagleitner and Thejas Nair. Bugs: HIVE-8411 https://issues.apache.org/jira/browse/HIVE-8411 Repository: hive-git Description --- Allow partial partition spec for some alter table statements, such as: alter table alter_partition_change_col1 partition (p1) change column c2 c2 decimal(10,0); Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java d5374bc ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 4e58ad8 ql/src/test/queries/clientpositive/alter_partition_change_col.q baabb9f ql/src/test/results/clientpositive/alter_partition_change_col.q.out 7123e40 Diff: https://reviews.apache.org/r/26481/diff/ Testing --- updated qfile test Thanks, Jason Dere
[jira] [Commented] (HIVE-8486) TPC-DS Query 96 parallelism is not set correcly
[ https://issues.apache.org/jira/browse/HIVE-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174277#comment-14174277 ] Brock Noland commented on HIVE-8486: Thanks Chao! Two questions: 1) Can we fix this the calculation described above? 2) Why are we estimating a single reducer in the normal case? The join covers the entire store_sales table which is 2.2GB. Perhap we need to add some debug statements to SetSparkReducerParallelism? TPC-DS Query 96 parallelism is not set correcly --- Key: HIVE-8486 URL: https://issues.apache.org/jira/browse/HIVE-8486 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Brock Noland Assignee: Chao When we run the query on a 20B we only have a parallelism factor of 1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8460) ORC SARG literal creation for double from float may lead to wrong evaluation of SARG
[
https://issues.apache.org/jira/browse/HIVE-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174279#comment-14174279
]
Hive QA commented on HIVE-8460:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12675323/HIVE-8460.2.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 6561 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1302/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1302/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1302/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12675323
- PreCommit-HIVE-TRUNK-Build
ORC SARG literal creation for double from float may lead to wrong evaluation
of SARG
Key: HIVE-8460
URL: https://issues.apache.org/jira/browse/HIVE-8460
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.0, 0.13.1
Reporter: Ashutosh Chauhan
Assignee: Prasanth J
Attachments: HIVE-8460.1.patch, HIVE-8460.2.patch
e.g., expression like d = 0.22 where 0.22 is of float type, gets converted to
double d = 0.21986 in SARG creation. This will cause erroneous evaluation
of SARG expression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8486) TPC-DS Query 96 parallelism is not set correcly
[
https://issues.apache.org/jira/browse/HIVE-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174292#comment-14174292
]
Chao commented on HIVE-8486:
1) I discussed with [~szehon] about this, and seems we cannot change the
calculation now, since it will affect the way how the buckets are calculated.
2) In the Spark branch, we are not doing anything to estimate the number of
reducers. In MR, in case this number is not set, it will estimate and set it at
runtime. This is done in {{MapRedTask}}. Tez also uses Auto Reducer
Parallelism (see HIVE-7158) to control this. As result, I think we should do
something to resolve this.
TPC-DS Query 96 parallelism is not set correcly
---
Key: HIVE-8486
URL: https://issues.apache.org/jira/browse/HIVE-8486
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Chao
When we run the query on a 20B we only have a parallelism factor of 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8486) TPC-DS Query 96 parallelism is not set correcly
[
https://issues.apache.org/jira/browse/HIVE-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174308#comment-14174308
]
Brock Noland commented on HIVE-8486:
bq. In the Spark branch, we are not doing anything to estimate the number of
reducers.
What is {{SetSparkReducerParallelism}} doing then?
TPC-DS Query 96 parallelism is not set correcly
---
Key: HIVE-8486
URL: https://issues.apache.org/jira/browse/HIVE-8486
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Chao
When we run the query on a 20B we only have a parallelism factor of 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8462) CBO duplicates columns
[
https://issues.apache.org/jira/browse/HIVE-8462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-8462:
---
Resolution: Fixed
Fix Version/s: 0.15.0
Status: Resolved (was: Patch Available)
committed to trunk. [~vikram.dixit] ok for 14?
CBO duplicates columns
--
Key: HIVE-8462
URL: https://issues.apache.org/jira/browse/HIVE-8462
Project: Hive
Issue Type: Bug
Components: CBO
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Priority: Critical
Fix For: 0.15.0
Attachments: HIVE-8462.01.patch, HIVE-8462.patch
{noformat}
select *, rank() over(partition by key order by value) as rr from src1
{noformat}
Original plan appears to be incorrect:
{noformat}
HiveProjectRel(key=[$0], value=[$1], (tok_function rank (tok_windowspec
(tok_partitioningspec (tok_distributeby (tok_table_or_col key)) (tok_orderby
(tok_tabsortcolnameasc (tok_table_or_col value))=[$5], rr=[$5])
HiveProjectRel(key=[$0], value=[$1], block__offset__inside__file=[$2],
input__file__name=[$3], row__id=[$4], (tok_function rank (tok_windowspec
(tok_partitioningspec (tok_distributeby (tok_table_or_col key)) (tok_orderby
(tok_tabsortcolnameasc (tok_table_or_col value))=[rank() OVER (PARTITION
BY $0 ORDER BY $1 ROWS BETWEEN 2147483647 FOLLOWING AND 2147483647
PRECEDING)])
HiveTableScanRel(table=[[default.src1]])
{noformat}
and final AST has {noformat}
TOK_SELEXPR
.
TOK_TABLE_OR_COL
$hdt$_0
(tok_function rank (tok_windowspec (tok_partitioningspec
(tok_distributeby (tok_table_or_col key)) (tok_orderby (tok_tabsortcolnameasc
(tok_table_or_col value))
(tok_function rank (tok_windowspec (tok_partitioningspec
(tok_distributeby (tok_table_or_col key)) (tok_orderby (tok_tabsortcolnameasc
(tok_table_or_col value))
TOK_SELEXPR
.
TOK_TABLE_OR_COL
$hdt$_0
(tok_function rank (tok_windowspec (tok_partitioningspec
(tok_distributeby (tok_table_or_col key)) (tok_orderby (tok_tabsortcolnameasc
(tok_table_or_col value))
rr
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8486) TPC-DS Query 96 parallelism is not set correcly
[ https://issues.apache.org/jira/browse/HIVE-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174313#comment-14174313 ] Chao commented on HIVE-8486: Oops. I didn't notice this class. Let me take a look and come back. TPC-DS Query 96 parallelism is not set correcly --- Key: HIVE-8486 URL: https://issues.apache.org/jira/browse/HIVE-8486 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Brock Noland Assignee: Chao When we run the query on a 20B we only have a parallelism factor of 1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8486) TPC-DS Query 96 parallelism is not set correcly
[
https://issues.apache.org/jira/browse/HIVE-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174341#comment-14174341
]
Chao commented on HIVE-8486:
OK, I debugged this query. In {{SetSparkReducerParallelism}}, in order to
estimate, it needs to obtain statistics from the siblings of the current reduce
sink, and adds up the total number of bytes. However, somehow the
{{statistics}} field of all the siblings are null, and hence the number of
bytes is 0 at end. As result, it will only use one reducer.
I'm wondering if this is something we haven't implemented yet, or is it a bug?
TPC-DS Query 96 parallelism is not set correcly
---
Key: HIVE-8486
URL: https://issues.apache.org/jira/browse/HIVE-8486
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Chao
When we run the query on a 20B we only have a parallelism factor of 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-8489) Add sanity check to dynamic partition pruning
Gunther Hagleitner created HIVE-8489: Summary: Add sanity check to dynamic partition pruning Key: HIVE-8489 URL: https://issues.apache.org/jira/browse/HIVE-8489 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 0.14.0 For debugging/safety it would help to make sure we received the right number of events from tez tasks. This is always supposed to be the case, but would be really hard to debug if something goes wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8490) Constant folding should happen before partition pruning
Ashutosh Chauhan created HIVE-8490: -- Summary: Constant folding should happen before partition pruning Key: HIVE-8490 URL: https://issues.apache.org/jira/browse/HIVE-8490 Project: Hive Issue Type: Improvement Components: Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan so that partition pruning see simplified expressions and present simpler expressions to metastore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8490) Constant folding should happen before partition pruning
[ https://issues.apache.org/jira/browse/HIVE-8490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8490: --- Attachment: HIVE-8490.patch Constant folding should happen before partition pruning --- Key: HIVE-8490 URL: https://issues.apache.org/jira/browse/HIVE-8490 Project: Hive Issue Type: Improvement Components: Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-8490.patch so that partition pruning see simplified expressions and present simpler expressions to metastore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8490) Constant folding should happen before partition pruning
[ https://issues.apache.org/jira/browse/HIVE-8490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8490: --- Status: Patch Available (was: Open) Constant folding should happen before partition pruning --- Key: HIVE-8490 URL: https://issues.apache.org/jira/browse/HIVE-8490 Project: Hive Issue Type: Improvement Components: Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-8490.patch so that partition pruning see simplified expressions and present simpler expressions to metastore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8490) Constant folding should happen before partition pruning
[ https://issues.apache.org/jira/browse/HIVE-8490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174348#comment-14174348 ] Ashutosh Chauhan commented on HIVE-8490: Needs HIVE-6934 Constant folding should happen before partition pruning --- Key: HIVE-8490 URL: https://issues.apache.org/jira/browse/HIVE-8490 Project: Hive Issue Type: Improvement Components: Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-8490.patch so that partition pruning see simplified expressions and present simpler expressions to metastore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
