[jira] [Commented] (HIVE-2390) Expand support for union types
[
https://issues.apache.org/jira/browse/HIVE-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125239#comment-14125239
]
Suma Shivaprasad commented on HIVE-2390:
[~amareshwari] Yes Test Case failure is unrelated to the patch
Expand support for union types
--
Key: HIVE-2390
URL: https://issues.apache.org/jira/browse/HIVE-2390
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Jakob Homan
Assignee: Suma Shivaprasad
Labels: uniontype
Fix For: 0.14.0
Attachments: HIVE-2390.1.patch, HIVE-2390.patch
When the union type was introduced, full support for it wasn't provided. For
instance, when working with a union that gets passed to LazyBinarySerde:
{noformat}Caused by: java.lang.RuntimeException: Unrecognized type: UNION
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:468)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serializeStruct(LazyBinarySerDe.java:230)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:184)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8017) Use HiveKey instead of BytesWritable as key type of the pair RDD [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-8017:
-
Attachment: HIVE-8017-spark.patch
Use HiveKey instead of BytesWritable as key type of the pair RDD [Spark
Branch]
---
Key: HIVE-8017
URL: https://issues.apache.org/jira/browse/HIVE-8017
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Rui Li
Assignee: Rui Li
Attachments: HIVE-8017-spark.patch
HiveKey should be used as the key type because it holds the hash code for
partitioning. While BytesWritable serves partitioning well for simple cases,
we have to use {{HiveKey.hashCode}} for more complicated ones, e.g. join,
bucketed table, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8017) Use HiveKey instead of BytesWritable as key type of the pair RDD [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-8017:
-
Status: Patch Available (was: Open)
Use HiveKey instead of BytesWritable as key type of the pair RDD [Spark
Branch]
---
Key: HIVE-8017
URL: https://issues.apache.org/jira/browse/HIVE-8017
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Rui Li
Assignee: Rui Li
Attachments: HIVE-8017-spark.patch
HiveKey should be used as the key type because it holds the hash code for
partitioning. While BytesWritable serves partitioning well for simple cases,
we have to use {{HiveKey.hashCode}} for more complicated ones, e.g. join,
bucketed table, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8017) Use HiveKey instead of BytesWritable as key type of the pair RDD [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125252#comment-14125252
]
Rui Li commented on HIVE-8017:
--
This patch change the RDD key type to HiveKey after map/reduce functions have
been applied. Original input RDD key type remains BytesWritable.
Use HiveKey instead of BytesWritable as key type of the pair RDD [Spark
Branch]
---
Key: HIVE-8017
URL: https://issues.apache.org/jira/browse/HIVE-8017
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Rui Li
Assignee: Rui Li
Attachments: HIVE-8017-spark.patch
HiveKey should be used as the key type because it holds the hash code for
partitioning. While BytesWritable serves partitioning well for simple cases,
we have to use {{HiveKey.hashCode}} for more complicated ones, e.g. join,
bucketed table, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7923) populate stats for test tables
[ https://issues.apache.org/jira/browse/HIVE-7923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-7923: --- Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Pengcheng! populate stats for test tables -- Key: HIVE-7923 URL: https://issues.apache.org/jira/browse/HIVE-7923 Project: Hive Issue Type: Improvement Reporter: pengcheng xiong Assignee: pengcheng xiong Priority: Minor Fix For: 0.14.0 Attachments: HIVE-7923.1.patch, HIVE-7923.2.patch, HIVE-7923.3.patch, HIVE-7923.4.patch, HIVE-7923.5.patch, HIVE-7923.6.patch Current q_test only generates tables, e.g., src only but does not create status. All the test cases will fail in CBO because CBO depends on the status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8018) Fix typo in config var name for dynamic partition pruning
Gunther Hagleitner created HIVE-8018: Summary: Fix typo in config var name for dynamic partition pruning Key: HIVE-8018 URL: https://issues.apache.org/jira/browse/HIVE-8018 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Found by [~leftylev] in HIVE-7826 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8018) Fix typo in config var name for dynamic partition pruning
[ https://issues.apache.org/jira/browse/HIVE-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8018: - Priority: Minor (was: Major) Fix typo in config var name for dynamic partition pruning - Key: HIVE-8018 URL: https://issues.apache.org/jira/browse/HIVE-8018 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Priority: Minor Found by [~leftylev] in HIVE-7826 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8018) Fix typo in config var name for dynamic partition pruning
[ https://issues.apache.org/jira/browse/HIVE-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8018: - Attachment: HIVE-8018.1.patch Fix typo in config var name for dynamic partition pruning - Key: HIVE-8018 URL: https://issues.apache.org/jira/browse/HIVE-8018 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Priority: Minor Attachments: HIVE-8018.1.patch Found by [~leftylev] in HIVE-7826 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8018) Fix typo in config var name for dynamic partition pruning
[ https://issues.apache.org/jira/browse/HIVE-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125278#comment-14125278 ] Gunther Hagleitner commented on HIVE-8018: -- [~leftylev] can you review/+1 this one? Fix typo in config var name for dynamic partition pruning - Key: HIVE-8018 URL: https://issues.apache.org/jira/browse/HIVE-8018 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Priority: Minor Attachments: HIVE-8018.1.patch Found by [~leftylev] in HIVE-7826 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 25245: Support dynamic service discovery for HiveServer2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25245/ --- (Updated Sept. 8, 2014, 7:43 a.m.) Review request for hive, Alan Gates, Navis Ryu, Szehon Ho, and Thejas Nair. Changes --- Patch rebased on trunk. Bugs: HIVE-7935 https://issues.apache.org/jira/browse/HIVE-7935 Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-7935 Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 31aeba9 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java ae128a9 jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java cbcfec7 jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java 6e248d6 jdbc/src/java/org/apache/hive/jdbc/JdbcUriParseException.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/Utils.java 58339bf jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientException.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientHelper.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 0919d2f ql/src/java/org/apache/hadoop/hive/ql/util/ZooKeeperHiveHelper.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java 59294b1 service/src/java/org/apache/hive/service/cli/CLIService.java a0bc905 service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f5a8f27 service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java b0bb8be service/src/java/org/apache/hive/service/cli/session/SessionManager.java 11d25cc service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java 2b80adc service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 443c371 service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java 4067106 service/src/java/org/apache/hive/service/server/HiveServer2.java 124996c service/src/test/org/apache/hive/service/cli/session/TestSessionGlobalInitFile.java 66fc1fc Diff: https://reviews.apache.org/r/25245/diff/ Testing --- Manual testing. Thanks, Vaibhav Gumashta
[jira] [Updated] (HIVE-7935) Support dynamic service discovery for HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-7935: --- Status: Open (was: Patch Available) Support dynamic service discovery for HiveServer2 - Key: HIVE-7935 URL: https://issues.apache.org/jira/browse/HIVE-7935 Project: Hive Issue Type: New Feature Components: HiveServer2, JDBC Affects Versions: 0.14.0 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Fix For: 0.14.0 Attachments: HIVE-7935.1.patch, HIVE-7935.2.patch To support Rolling Upgrade / HA, we need a mechanism by which a JDBC client can dynamically resolve an HiveServer2 to connect to. *High Level Design:* Whether, dynamic service discovery is supported or not, can be configured by setting HIVE_SERVER2_SUPPORT_DYNAMIC_SERVICE_DISCOVERY. ZooKeeper is used to support this. * When an instance of HiveServer2 comes up, it adds itself as a znode to ZooKeeper under a configurable namespace (HIVE_SERVER2_ZOOKEEPER_NAMESPACE). * A JDBC/ODBC client now specifies the ZooKeeper ensemble in its connection string, instead of pointing to a specific HiveServer2 instance. The JDBC driver, uses the ZooKeeper ensemble to pick an instance of HiveServer2 to connect for the entire session. * When an instance is removed from ZooKeeper, the existing client sessions continue till completion. When the last client session completes, the instance shuts down. * All new client connection pick one of the available HiveServer2 uris from ZooKeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7935) Support dynamic service discovery for HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-7935: --- Status: Patch Available (was: Open) Support dynamic service discovery for HiveServer2 - Key: HIVE-7935 URL: https://issues.apache.org/jira/browse/HIVE-7935 Project: Hive Issue Type: New Feature Components: HiveServer2, JDBC Affects Versions: 0.14.0 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Fix For: 0.14.0 Attachments: HIVE-7935.1.patch, HIVE-7935.2.patch, HIVE-7935.3.patch To support Rolling Upgrade / HA, we need a mechanism by which a JDBC client can dynamically resolve an HiveServer2 to connect to. *High Level Design:* Whether, dynamic service discovery is supported or not, can be configured by setting HIVE_SERVER2_SUPPORT_DYNAMIC_SERVICE_DISCOVERY. ZooKeeper is used to support this. * When an instance of HiveServer2 comes up, it adds itself as a znode to ZooKeeper under a configurable namespace (HIVE_SERVER2_ZOOKEEPER_NAMESPACE). * A JDBC/ODBC client now specifies the ZooKeeper ensemble in its connection string, instead of pointing to a specific HiveServer2 instance. The JDBC driver, uses the ZooKeeper ensemble to pick an instance of HiveServer2 to connect for the entire session. * When an instance is removed from ZooKeeper, the existing client sessions continue till completion. When the last client session completes, the instance shuts down. * All new client connection pick one of the available HiveServer2 uris from ZooKeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7935) Support dynamic service discovery for HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-7935: --- Attachment: HIVE-7935.3.patch Patch rebased on trunk. Support dynamic service discovery for HiveServer2 - Key: HIVE-7935 URL: https://issues.apache.org/jira/browse/HIVE-7935 Project: Hive Issue Type: New Feature Components: HiveServer2, JDBC Affects Versions: 0.14.0 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Fix For: 0.14.0 Attachments: HIVE-7935.1.patch, HIVE-7935.2.patch, HIVE-7935.3.patch To support Rolling Upgrade / HA, we need a mechanism by which a JDBC client can dynamically resolve an HiveServer2 to connect to. *High Level Design:* Whether, dynamic service discovery is supported or not, can be configured by setting HIVE_SERVER2_SUPPORT_DYNAMIC_SERVICE_DISCOVERY. ZooKeeper is used to support this. * When an instance of HiveServer2 comes up, it adds itself as a znode to ZooKeeper under a configurable namespace (HIVE_SERVER2_ZOOKEEPER_NAMESPACE). * A JDBC/ODBC client now specifies the ZooKeeper ensemble in its connection string, instead of pointing to a specific HiveServer2 instance. The JDBC driver, uses the ZooKeeper ensemble to pick an instance of HiveServer2 to connect for the entire session. * When an instance is removed from ZooKeeper, the existing client sessions continue till completion. When the last client session completes, the instance shuts down. * All new client connection pick one of the available HiveServer2 uris from ZooKeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-649) [UDF] now() for getting current time
[ https://issues.apache.org/jira/browse/HIVE-649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HIVE-649: -- Attachment: HIVE-649.2.patch I've taken the liberty to work on this a bit but I'm not too confident. * I'm returning a TIMESTAMP instead of a String * I'm assigning NOW in the constructor ** UDF instances will still be started at different times so now() won't be consistent. I'm not sure if I could get a better timestamp of when the job started using a GenericUDF [UDF] now() for getting current time Key: HIVE-649 URL: https://issues.apache.org/jira/browse/HIVE-649 Project: Hive Issue Type: New Feature Components: UDF Reporter: Min Zhou Attachments: HIVE-649.2.patch, HIVE-649.patch http://dev.mysql.com/doc/refman/5.1/en/date-and-time-functions.html#function_now -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-649) [UDF] now() for getting current time
[ https://issues.apache.org/jira/browse/HIVE-649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HIVE-649: -- Assignee: Lars Francke Status: Patch Available (was: Open) [UDF] now() for getting current time Key: HIVE-649 URL: https://issues.apache.org/jira/browse/HIVE-649 Project: Hive Issue Type: New Feature Components: UDF Reporter: Min Zhou Assignee: Lars Francke Attachments: HIVE-649.2.patch, HIVE-649.patch http://dev.mysql.com/doc/refman/5.1/en/date-and-time-functions.html#function_now -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-676) code cleanup in mapredWork
[ https://issues.apache.org/jira/browse/HIVE-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125282#comment-14125282 ] Lars Francke commented on HIVE-676: --- This seems to have been fixed in the meantime. I can't find any reference to pathToPartnInfo anymore. I'm closing this. code cleanup in mapredWork -- Key: HIVE-676 URL: https://issues.apache.org/jira/browse/HIVE-676 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: Carl Steinbach mapredWork currently contains: pathToAlias aliasToPartnInfo pathToPartnInfo pathToPartnInfo can be derived from pathToAlias and aliasToPartnInfo - it should be removed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-676) code cleanup in mapredWork
[ https://issues.apache.org/jira/browse/HIVE-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke resolved HIVE-676. --- Resolution: Fixed code cleanup in mapredWork -- Key: HIVE-676 URL: https://issues.apache.org/jira/browse/HIVE-676 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: Carl Steinbach mapredWork currently contains: pathToAlias aliasToPartnInfo pathToPartnInfo pathToPartnInfo can be derived from pathToAlias and aliasToPartnInfo - it should be removed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8017) Use HiveKey instead of BytesWritable as key type of the pair RDD [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125286#comment-14125286
]
Hive QA commented on HIVE-8017:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667133/HIVE-8017-spark.patch
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 6343 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_having
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_merge1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_merge2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_19
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_25
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/117/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/117/console
Test logs:
http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-117/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667133
Use HiveKey instead of BytesWritable as key type of the pair RDD [Spark
Branch]
---
Key: HIVE-8017
URL: https://issues.apache.org/jira/browse/HIVE-8017
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Rui Li
Assignee: Rui Li
Attachments: HIVE-8017-spark.patch
HiveKey should be used as the key type because it holds the hash code for
partitioning. While BytesWritable serves partitioning well for simple cases,
we have to use {{HiveKey.hashCode}} for more complicated ones, e.g. join,
bucketed table, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8018) Fix typo in config var name for dynamic partition pruning
[ https://issues.apache.org/jira/browse/HIVE-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125285#comment-14125285 ] Lars Francke commented on HIVE-8018: Not Lefty but still: +1 with the minor comment that it is a long line (exceeds 100 chars and violates checkstyle) Fix typo in config var name for dynamic partition pruning - Key: HIVE-8018 URL: https://issues.apache.org/jira/browse/HIVE-8018 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Priority: Minor Attachments: HIVE-8018.1.patch Found by [~leftylev] in HIVE-7826 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-709) support for adding new functions (not temporary)
[ https://issues.apache.org/jira/browse/HIVE-709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke resolved HIVE-709. --- Resolution: Duplicate support for adding new functions (not temporary) Key: HIVE-709 URL: https://issues.apache.org/jira/browse/HIVE-709 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain CREATE and DROP FUNCTIONS should be supported - these functions should not be session specific. As long as they are not dropped, users should be able to access them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8007) Use proper Thrift comments
[
https://issues.apache.org/jira/browse/HIVE-8007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Francke updated HIVE-8007:
---
Status: Patch Available (was: Open)
Use proper Thrift comments
--
Key: HIVE-8007
URL: https://issues.apache.org/jira/browse/HIVE-8007
Project: Hive
Issue Type: Improvement
Reporter: Lars Francke
Assignee: Lars Francke
Priority: Minor
Attachments: HIVE-8007.1.patch
Currently the thrift file uses {{//}} to denote comments. Thrift understands
the {{/** ... */}} syntax and converts that into documentation in the
generated code. This patch changes the syntax.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8007) Use proper Thrift comments
[
https://issues.apache.org/jira/browse/HIVE-8007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Francke updated HIVE-8007:
---
Attachment: HIVE-8007.1.patch
I can't get Thrift 0.9 to build on my Mac so I'm using Hive QA to verify that
everything still works.
Use proper Thrift comments
--
Key: HIVE-8007
URL: https://issues.apache.org/jira/browse/HIVE-8007
Project: Hive
Issue Type: Improvement
Reporter: Lars Francke
Assignee: Lars Francke
Priority: Minor
Attachments: HIVE-8007.1.patch
Currently the thrift file uses {{//}} to denote comments. Thrift understands
the {{/** ... */}} syntax and converts that into documentation in the
generated code. This patch changes the syntax.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-695) sort by does not support columns along with table alias when there is a group by
[ https://issues.apache.org/jira/browse/HIVE-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125293#comment-14125293 ] Lars Francke commented on HIVE-695: --- I just checked and it's still an issue in Hive 0.12 sort by does not support columns along with table alias when there is a group by Key: HIVE-695 URL: https://issues.apache.org/jira/browse/HIVE-695 Project: Hive Issue Type: Bug Reporter: Raghotham Murthy Priority: Minor The following query works: select a.userid from pageviews a where a.ds = '2009-07-21' sort by a.userid; However, the query below results in an error: select a.userid, count(1) from pageviews a where a.ds = '2009-07-21' group by a.userid sort by a.userid; FAILED: Error in semantic analysis: line 1:109 Invalid Table Alias or Column Reference a We should either not allow sort by after a group by or allow sort by to have column aliases always. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-781) Integrate Zebra with Hive
[ https://issues.apache.org/jira/browse/HIVE-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke resolved HIVE-781. --- Resolution: Won't Fix Zebra has been abandoned/deprecated (perhaps in favor of HCatalog?) as far as I know. Integrate Zebra with Hive - Key: HIVE-781 URL: https://issues.apache.org/jira/browse/HIVE-781 Project: Hive Issue Type: New Feature Reporter: He Yongqiang This is raised for my research. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-952) Support analytic NTILE function
[
https://issues.apache.org/jira/browse/HIVE-952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Francke resolved HIVE-952.
---
Resolution: Fixed
This has been fixed in the meantime.
Support analytic NTILE function
---
Key: HIVE-952
URL: https://issues.apache.org/jira/browse/HIVE-952
Project: Hive
Issue Type: New Feature
Components: OLAP, Query Processor, UDF
Reporter: Carl Steinbach
The NTILE function divides a set of ordered rows into equally sized buckets
and assigns a bucket number to each row.
Useful for calculating tertiles, quartiles, quintiles, etc.
Example:
{code:sql}
SELECT last_name, salary,
NTILE(4) OVER (ORDER BY salary DESC) AS quartile
FROM employees
WHERE department_id = 100;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-958) support ALTER PARTITION
[ https://issues.apache.org/jira/browse/HIVE-958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke resolved HIVE-958. --- Resolution: Fixed This has been fixed in the meantime support ALTER PARTITION --- Key: HIVE-958 URL: https://issues.apache.org/jira/browse/HIVE-958 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Paul Yang alter table table_name ... should be enhanced to also support: alter table table_name partition partition_name ... Only 1 partition can be specified at a time -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-958) support ALTER PARTITION
[ https://issues.apache.org/jira/browse/HIVE-958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HIVE-958: -- Assignee: (was: Paul Yang) support ALTER PARTITION --- Key: HIVE-958 URL: https://issues.apache.org/jira/browse/HIVE-958 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain alter table table_name ... should be enhanced to also support: alter table table_name partition partition_name ... Only 1 partition can be specified at a time -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-1033) change default value of hive.exec.parallel to true
[ https://issues.apache.org/jira/browse/HIVE-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HIVE-1033: --- Status: Patch Available (was: Open) change default value of hive.exec.parallel to true -- Key: HIVE-1033 URL: https://issues.apache.org/jira/browse/HIVE-1033 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Ashutosh Chauhan Attachments: HIVE-1033.2.patch, HIVE-1033.3.patch, hive.1033.1.patch There is no harm in changing it to true. Inside facebook, we have been testing it and it seems to be stable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-1033) change default value of hive.exec.parallel to true
[ https://issues.apache.org/jira/browse/HIVE-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HIVE-1033: --- Attachment: HIVE-1033.3.patch Rebased to trunk, letting Hive QA test this again. change default value of hive.exec.parallel to true -- Key: HIVE-1033 URL: https://issues.apache.org/jira/browse/HIVE-1033 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Ashutosh Chauhan Attachments: HIVE-1033.2.patch, HIVE-1033.3.patch, hive.1033.1.patch There is no harm in changing it to true. Inside facebook, we have been testing it and it seems to be stable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-1175) Enable automatic patch testing on Hudson
[ https://issues.apache.org/jira/browse/HIVE-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke resolved HIVE-1175. Resolution: Fixed The original link doesn't work anymore but from the title I assume that's already been fixed by the Hive QA bot. Enable automatic patch testing on Hudson Key: HIVE-1175 URL: https://issues.apache.org/jira/browse/HIVE-1175 Project: Hive Issue Type: Task Components: Build Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach Labels: test-patch See http://developer.yahoo.net/blogs/hadoop/2007/12/if_it_hurts_automate_it_1.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-1945) Support for ESCAPE BY not documented.
[ https://issues.apache.org/jira/browse/HIVE-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke resolved HIVE-1945. Resolution: Fixed This has been documented in the meantime Support for ESCAPE BY not documented. - Key: HIVE-1945 URL: https://issues.apache.org/jira/browse/HIVE-1945 Project: Hive Issue Type: Bug Reporter: Arvind Prabhakar Priority: Minor From the client positive test query files - create_escape.q and input_lazyserde.q, it is evident that there is support for specifying ESCAPED BY when using delimited row format. However this is not documented on Hive Wiki. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-1879) Remove hive.metastore.metadb.dir property from hive-default.xml and HiveConf
[ https://issues.apache.org/jira/browse/HIVE-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HIVE-1879: --- Attachment: HIVE-1879.1.patch My stab at this. Remove hive.metastore.metadb.dir property from hive-default.xml and HiveConf Key: HIVE-1879 URL: https://issues.apache.org/jira/browse/HIVE-1879 Project: Hive Issue Type: Bug Components: Configuration, Metastore Reporter: Carl Steinbach Assignee: Carl Steinbach Attachments: HIVE-1879.1.patch The file-based MetaStore implementation was removed in HIVE-143. We also need to remove the hive.metastore.metadb.dir property from hive-default.xml and HiveConf, as well as the references to this property that currently appear in HiveMetaStoreClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-1879) Remove hive.metastore.metadb.dir property from hive-default.xml and HiveConf
[ https://issues.apache.org/jira/browse/HIVE-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke reassigned HIVE-1879: -- Assignee: Lars Francke (was: Carl Steinbach) Remove hive.metastore.metadb.dir property from hive-default.xml and HiveConf Key: HIVE-1879 URL: https://issues.apache.org/jira/browse/HIVE-1879 Project: Hive Issue Type: Bug Components: Configuration, Metastore Reporter: Carl Steinbach Assignee: Lars Francke Attachments: HIVE-1879.1.patch The file-based MetaStore implementation was removed in HIVE-143. We also need to remove the hive.metastore.metadb.dir property from hive-default.xml and HiveConf, as well as the references to this property that currently appear in HiveMetaStoreClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-1879) Remove hive.metastore.metadb.dir property from hive-default.xml and HiveConf
[ https://issues.apache.org/jira/browse/HIVE-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HIVE-1879: --- Status: Patch Available (was: Open) Remove hive.metastore.metadb.dir property from hive-default.xml and HiveConf Key: HIVE-1879 URL: https://issues.apache.org/jira/browse/HIVE-1879 Project: Hive Issue Type: Bug Components: Configuration, Metastore Reporter: Carl Steinbach Assignee: Carl Steinbach Attachments: HIVE-1879.1.patch The file-based MetaStore implementation was removed in HIVE-143. We also need to remove the hive.metastore.metadb.dir property from hive-default.xml and HiveConf, as well as the references to this property that currently appear in HiveMetaStoreClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2019) Implement NOW() UDF
[ https://issues.apache.org/jira/browse/HIVE-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke resolved HIVE-2019. Resolution: Duplicate Implement NOW() UDF --- Key: HIVE-2019 URL: https://issues.apache.org/jira/browse/HIVE-2019 Project: Hive Issue Type: New Feature Components: UDF Reporter: Carl Steinbach Assignee: Priyadarshini Attachments: HIVE-2019.patch Reference: http://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html#function_now -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2149) Fix ant target generate-schema
[ https://issues.apache.org/jira/browse/HIVE-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke resolved HIVE-2149. Resolution: Invalid Invalid since the move to Maven Fix ant target generate-schema --- Key: HIVE-2149 URL: https://issues.apache.org/jira/browse/HIVE-2149 Project: Hive Issue Type: Bug Reporter: Ashutosh Chauhan Priority: Minor Running generate-schema target in metastore dir results in generate-schema: [java] Exception in thread main java.lang.NoClassDefFoundError: org/jpox/SchemaTool -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-2353) Optimize UDF LIKE
[ https://issues.apache.org/jira/browse/HIVE-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125331#comment-14125331 ] Lars Francke commented on HIVE-2353: [~cwsteinbach] do you have more details on what you meant here? Optimize UDF LIKE - Key: HIVE-2353 URL: https://issues.apache.org/jira/browse/HIVE-2353 Project: Hive Issue Type: Improvement Components: UDF Reporter: Carl Steinbach Assignee: Carl Steinbach -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7470) Wrong Thrift declaration for {{ShowCompactResponseElement}}
[
https://issues.apache.org/jira/browse/HIVE-7470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol updated HIVE-7470:
---
Attachment: HIVE-7470.2.patch
Rebased on last trunk
Wrong Thrift declaration for {{ShowCompactResponseElement}}
---
Key: HIVE-7470
URL: https://issues.apache.org/jira/browse/HIVE-7470
Project: Hive
Issue Type: Bug
Components: Metastore, Thrift API
Affects Versions: 0.14.0
Reporter: Damien Carol
Assignee: Damien Carol
Priority: Minor
Labels: metastore, thrift
Fix For: 0.14.0
Attachments: HIVE-7470.1.patch, HIVE-7470.2.patch
Prerequiste :
1. Remote metastore
2. Activate ACID and compactions
3. Launch ALTER TABLE foo COMPACT 'bar'
4. Call {{show_compact()}} on remote metastore
This use case throws exception in Thrift stack.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7470) Wrong Thrift declaration for {{ShowCompactResponseElement}}
[
https://issues.apache.org/jira/browse/HIVE-7470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol updated HIVE-7470:
---
Status: Patch Available (was: Open)
Wrong Thrift declaration for {{ShowCompactResponseElement}}
---
Key: HIVE-7470
URL: https://issues.apache.org/jira/browse/HIVE-7470
Project: Hive
Issue Type: Bug
Components: Metastore, Thrift API
Affects Versions: 0.14.0
Reporter: Damien Carol
Assignee: Damien Carol
Priority: Minor
Labels: metastore, thrift
Fix For: 0.14.0
Attachments: HIVE-7470.1.patch, HIVE-7470.2.patch
Prerequiste :
1. Remote metastore
2. Activate ACID and compactions
3. Launch ALTER TABLE foo COMPACT 'bar'
4. Call {{show_compact()}} on remote metastore
This use case throws exception in Thrift stack.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7689) Enable Postgres as METASTORE back-end
[ https://issues.apache.org/jira/browse/HIVE-7689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damien Carol updated HIVE-7689: --- Description: I maintain few patches to make Metastore works with Postgres back end in our production environment. The main goal of this JIRA is to push upstream these patches. This patch enable LOCKS, COMPACTION and fix error in STATS on postgres metastore. was: I maintain few patches to make Metastore works with Postgres back end in our production environment. The main goal of this JIRA is to push upstream these patches. This patch enable LOCKS, COMPACTION and fix error in STATS on metastore. Enable Postgres as METASTORE back-end - Key: HIVE-7689 URL: https://issues.apache.org/jira/browse/HIVE-7689 Project: Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.14.0 Reporter: Damien Carol Assignee: Damien Carol Priority: Minor Labels: metastore, postgres Fix For: 0.14.0 Attachments: HIVE-7889.1.patch, HIVE-7889.2.patch, HIVE-7889.3.patch, HIVE-7889.4.patch I maintain few patches to make Metastore works with Postgres back end in our production environment. The main goal of this JIRA is to push upstream these patches. This patch enable LOCKS, COMPACTION and fix error in STATS on postgres metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-649) [UDF] now() for getting current time
[
https://issues.apache.org/jira/browse/HIVE-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125350#comment-14125350
]
Hive QA commented on HIVE-649:
--
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667139/HIVE-649.2.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6185 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_functions
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTbl
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/688/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/688/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-688/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667139
[UDF] now() for getting current time
Key: HIVE-649
URL: https://issues.apache.org/jira/browse/HIVE-649
Project: Hive
Issue Type: New Feature
Components: UDF
Reporter: Min Zhou
Assignee: Lars Francke
Attachments: HIVE-649.2.patch, HIVE-649.patch
http://dev.mysql.com/doc/refman/5.1/en/date-and-time-functions.html#function_now
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Timeline for release of Hive 0.14
Would like to include https://issues.apache.org/jira/browse/HIVE-2390 and https://issues.apache.org/jira/browse/HIVE-7936. I can review and merge them. Thanks Amareshwari On Sat, Sep 6, 2014 at 6:23 AM, Vikram Dixit vik...@hortonworks.com wrote: Hi Folks, I am going to start consolidating the items mentioned in this list and create a wiki page to track it. I will wait till the end of next week to create the branch taking into account Ashutosh's request. Thanks Vikram. On Fri, Sep 5, 2014 at 5:39 PM, Ashutosh Chauhan hashut...@apache.org wrote: Vikram, Some of us are working on stabilizing cbo branch and trying to get it merged into trunk. We feel we are close. May I request to defer cutting the branch for few more days? Folks interested in this can track our progress here : https://issues.apache.org/jira/browse/HIVE-7946 Thanks, Ashutosh On Fri, Aug 22, 2014 at 4:09 PM, Lars Francke lars.fran...@gmail.com wrote: Thank you for volunteering to do the release. I think a 0.14 release is a good idea. I have a couple of issues I'd like to get in too: * Either HIVE-7107[0] (Fix an issue in the HiveServer1 JDBC driver) or HIVE-6977[1] (Delete HiveServer1). The former needs a review the latter a patch * HIVE-6123[2] Checkstyle in Maven needs a review HIVE-7622[3] HIVE-7543[4] are waiting for any reviews or comments on my previous thread[5]. I'd still appreciate any helpers for reviews or even just comments. I'd feel very sad if I had done all that work for nothing. Hoping this thread gives me a wider audience. Both patches fix up issues that should have been caught in earlier reviews as they are almost all Checkstyle or other style violations but they make for huge patches. I could also create hundreds of small issues or stop doing these things entirely [0] https://issues.apache.org/jira/browse/HIVE-7107 [1] https://issues.apache.org/jira/browse/HIVE-6977 [2] https://issues.apache.org/jira/browse/HIVE-6123 [3] https://issues.apache.org/jira/browse/HIVE-7622 [4] https://issues.apache.org/jira/browse/HIVE-7543 On Fri, Aug 22, 2014 at 11:01 PM, John Pullokkaran jpullokka...@hortonworks.com wrote: We are working on Hive-5775 (Cost Based Optimizer), we hope to get this in to Hive 14. Thanks John On Thu, Aug 21, 2014 at 11:53 PM, Lefty Leverenz leftylever...@gmail.com wrote: Release 0.14 should include HIVE-6586 https://issues.apache.org/jira/browse/HIVE-6586 (various fixes to HiveConf.java parameters). I'll do that as soon as possible. 72 jiras have the TODOC14 label now, although my own tally is 99. This is more than mere mortals can accomplish in a few weeks. Therefore I recommend that you all plead with your managers to allocate some tech-writer resources to Hive wikidocs for the 0.14.0 release. I'll send out a state-of-the-docs message in a separate thread. -- Lefty On Fri, Aug 22, 2014 at 2:28 AM, Alan Gates ga...@hortonworks.com wrote: +1, Eugene and I are working on getting HIVE-5317 (insert, update, delete) done and would like to get it in. Alan. Nick Dimiduk ndimi...@gmail.com August 20, 2014 at 12:27 It'd be great to get HIVE-4765 included in 0.14. The proposed changes are a big improvement for us HBase folks. Would someone mind having a look in that direction? Thanks, Nick Thejas Nair the...@hortonworks.com August 19, 2014 at 15:20 +1 Sounds good to me. Its already almost 4 months since the last release. It is time to start preparing for the next one. Thanks for volunteering! Vikram Dixit vik...@hortonworks.com August 19, 2014 at 14:02 Hi Folks, I was thinking that it was about time that we had a release of hive 0.14 given our commitment to having a release of hive on a periodic basis. We could cut a branch and start working on a release in say 2 weeks time around September 5th (Friday). After branching, we can focus on stabilizing for the release and hopefully have an RC in about 2 weeks post that. I would like to volunteer myself for the duties of the release manager for this version if the community agrees. Thanks Vikram. -- Sent with Postbox http://www.getpostbox.com CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are
Re: Timeline for release of Hive 0.14
Same request for https://issues.apache.org/jira/browse/HIVE-7689 I already provided a patch, re-based it many times and I'm waiting for a review. Regards, Damien CAROL * tél : +33 (0)4 74 96 88 14 * fax : +33 (0)4 74 96 31 88 * email :dca...@blitzbs.com mailto:dca...@blitzbs.com BLITZ BUSINESS SERVICE Le 08/09/2014 12:08, amareshwarisr . a écrit : Would like to include https://issues.apache.org/jira/browse/HIVE-2390 and https://issues.apache.org/jira/browse/HIVE-7936. I can review and merge them. Thanks Amareshwari On Sat, Sep 6, 2014 at 6:23 AM, Vikram Dixit vik...@hortonworks.com wrote: Hi Folks, I am going to start consolidating the items mentioned in this list and create a wiki page to track it. I will wait till the end of next week to create the branch taking into account Ashutosh's request. Thanks Vikram. On Fri, Sep 5, 2014 at 5:39 PM, Ashutosh Chauhan hashut...@apache.org wrote: Vikram, Some of us are working on stabilizing cbo branch and trying to get it merged into trunk. We feel we are close. May I request to defer cutting the branch for few more days? Folks interested in this can track our progress here : https://issues.apache.org/jira/browse/HIVE-7946 Thanks, Ashutosh On Fri, Aug 22, 2014 at 4:09 PM, Lars Francke lars.fran...@gmail.com wrote: Thank you for volunteering to do the release. I think a 0.14 release is a good idea. I have a couple of issues I'd like to get in too: * Either HIVE-7107[0] (Fix an issue in the HiveServer1 JDBC driver) or HIVE-6977[1] (Delete HiveServer1). The former needs a review the latter a patch * HIVE-6123[2] Checkstyle in Maven needs a review HIVE-7622[3] HIVE-7543[4] are waiting for any reviews or comments on my previous thread[5]. I'd still appreciate any helpers for reviews or even just comments. I'd feel very sad if I had done all that work for nothing. Hoping this thread gives me a wider audience. Both patches fix up issues that should have been caught in earlier reviews as they are almost all Checkstyle or other style violations but they make for huge patches. I could also create hundreds of small issues or stop doing these things entirely [0] https://issues.apache.org/jira/browse/HIVE-7107 [1] https://issues.apache.org/jira/browse/HIVE-6977 [2] https://issues.apache.org/jira/browse/HIVE-6123 [3] https://issues.apache.org/jira/browse/HIVE-7622 [4] https://issues.apache.org/jira/browse/HIVE-7543 On Fri, Aug 22, 2014 at 11:01 PM, John Pullokkaran jpullokka...@hortonworks.com wrote: We are working on Hive-5775 (Cost Based Optimizer), we hope to get this in to Hive 14. Thanks John On Thu, Aug 21, 2014 at 11:53 PM, Lefty Leverenz leftylever...@gmail.com wrote: Release 0.14 should include HIVE-6586 https://issues.apache.org/jira/browse/HIVE-6586 (various fixes to HiveConf.java parameters). I'll do that as soon as possible. 72 jiras have the TODOC14 label now, although my own tally is 99. This is more than mere mortals can accomplish in a few weeks. Therefore I recommend that you all plead with your managers to allocate some tech-writer resources to Hive wikidocs for the 0.14.0 release. I'll send out a state-of-the-docs message in a separate thread. -- Lefty On Fri, Aug 22, 2014 at 2:28 AM, Alan Gates ga...@hortonworks.com wrote: +1, Eugene and I are working on getting HIVE-5317 (insert, update, delete) done and would like to get it in. Alan. Nick Dimiduk ndimi...@gmail.com August 20, 2014 at 12:27 It'd be great to get HIVE-4765 included in 0.14. The proposed changes are a big improvement for us HBase folks. Would someone mind having a look in that direction? Thanks, Nick Thejas Nair the...@hortonworks.com August 19, 2014 at 15:20 +1 Sounds good to me. Its already almost 4 months since the last release. It is time to start preparing for the next one. Thanks for volunteering! Vikram Dixit vik...@hortonworks.com August 19, 2014 at 14:02 Hi Folks, I was thinking that it was about time that we had a release of hive 0.14 given our commitment to having a release of hive on a periodic basis. We could cut a branch and start working on a release in say 2 weeks time around September 5th (Friday). After branching, we can focus on stabilizing for the release and hopefully have an RC in about 2 weeks post that. I would like to volunteer myself for the duties of the release manager for this version if the community agrees. Thanks Vikram. -- Sent with Postbox http://www.getpostbox.com CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have
[jira] [Updated] (HIVE-7689) Enable Postgres as METASTORE back-end
[ https://issues.apache.org/jira/browse/HIVE-7689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damien Carol updated HIVE-7689: --- Attachment: HIVE-7689.5.patch Rebased on last trunk Enable Postgres as METASTORE back-end - Key: HIVE-7689 URL: https://issues.apache.org/jira/browse/HIVE-7689 Project: Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.14.0 Reporter: Damien Carol Assignee: Damien Carol Priority: Minor Labels: metastore, postgres Fix For: 0.14.0 Attachments: HIVE-7689.5.patch, HIVE-7889.1.patch, HIVE-7889.2.patch, HIVE-7889.3.patch, HIVE-7889.4.patch I maintain few patches to make Metastore works with Postgres back end in our production environment. The main goal of this JIRA is to push upstream these patches. This patch enable LOCKS, COMPACTION and fix error in STATS on postgres metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 24602: HIVE-7689 : Enable Postgres as METASTORE back-end
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24602/ --- (Updated sep. 8, 2014, 10:27 matin) Review request for hive. Changes --- Rebased on last trunk Bugs: HIVE-7689 https://issues.apache.org/jira/browse/HIVE-7689 Repository: hive-git Description --- I maintain few patches to make Metastore works with Postgres back end in our production environment. The main goal of this JIRA is to push upstream these patches. This patch enable these features : * LOCKS on postgres metastore * COMPACTION on postgres metastore * TRANSACTION on postgres metastore * fix metastore update script for postgres Diffs (updated) - metastore/scripts/upgrade/postgres/hive-txn-schema-0.13.0.postgres.sql 2ebd3b0 metastore/src/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java d3aa66f metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java 06d8ac0 metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java f1697bb ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 264052f ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java b074ca9 ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 5e317ab ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsUtils.java 4625d27 Diff: https://reviews.apache.org/r/24602/diff/ Testing --- Using patched version in production. Enable concurrency with DbTxnManager. Thanks, Damien Carol
[jira] [Commented] (HIVE-7935) Support dynamic service discovery for HiveServer2
[
https://issues.apache.org/jira/browse/HIVE-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125402#comment-14125402
]
Hive QA commented on HIVE-7935:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667137/HIVE-7935.3.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6172 tests executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/689/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/689/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-689/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667137
Support dynamic service discovery for HiveServer2
-
Key: HIVE-7935
URL: https://issues.apache.org/jira/browse/HIVE-7935
Project: Hive
Issue Type: New Feature
Components: HiveServer2, JDBC
Affects Versions: 0.14.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Fix For: 0.14.0
Attachments: HIVE-7935.1.patch, HIVE-7935.2.patch, HIVE-7935.3.patch
To support Rolling Upgrade / HA, we need a mechanism by which a JDBC client
can dynamically resolve an HiveServer2 to connect to.
*High Level Design:*
Whether, dynamic service discovery is supported or not, can be configured by
setting HIVE_SERVER2_SUPPORT_DYNAMIC_SERVICE_DISCOVERY. ZooKeeper is used to
support this.
* When an instance of HiveServer2 comes up, it adds itself as a znode to
ZooKeeper under a configurable namespace (HIVE_SERVER2_ZOOKEEPER_NAMESPACE).
* A JDBC/ODBC client now specifies the ZooKeeper ensemble in its connection
string, instead of pointing to a specific HiveServer2 instance. The JDBC
driver, uses the ZooKeeper ensemble to pick an instance of HiveServer2 to
connect for the entire session.
* When an instance is removed from ZooKeeper, the existing client sessions
continue till completion. When the last client session completes, the
instance shuts down.
* All new client connection pick one of the available HiveServer2 uris from
ZooKeeper.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Timeline for release of Hive 0.14
I'll review that. I just need the time to test it against mysql, oracle, and hopefully sqlserver. But I think we can do this post branch if we need to, as it's a bug fix rather than a feature. Alan. Damien Carol mailto:dca...@blitzbs.com September 8, 2014 at 3:19 Same request for https://issues.apache.org/jira/browse/HIVE-7689 I already provided a patch, re-based it many times and I'm waiting for a review. Regards, Le 08/09/2014 12:08, amareshwarisr . a écrit : amareshwarisr . mailto:amareshw...@gmail.com September 8, 2014 at 3:08 Would like to include https://issues.apache.org/jira/browse/HIVE-2390 and https://issues.apache.org/jira/browse/HIVE-7936. I can review and merge them. Thanks Amareshwari Vikram Dixit mailto:vik...@hortonworks.com September 5, 2014 at 17:53 Hi Folks, I am going to start consolidating the items mentioned in this list and create a wiki page to track it. I will wait till the end of next week to create the branch taking into account Ashutosh's request. Thanks Vikram. On Fri, Sep 5, 2014 at 5:39 PM, Ashutosh Chauhan hashut...@apache.org Ashutosh Chauhan mailto:hashut...@apache.org September 5, 2014 at 17:39 Vikram, Some of us are working on stabilizing cbo branch and trying to get it merged into trunk. We feel we are close. May I request to defer cutting the branch for few more days? Folks interested in this can track our progress here : https://issues.apache.org/jira/browse/HIVE-7946 Thanks, Ashutosh On Fri, Aug 22, 2014 at 4:09 PM, Lars Francke lars.fran...@gmail.com Lars Francke mailto:lars.fran...@gmail.com August 22, 2014 at 16:09 Thank you for volunteering to do the release. I think a 0.14 release is a good idea. I have a couple of issues I'd like to get in too: * Either HIVE-7107[0] (Fix an issue in the HiveServer1 JDBC driver) or HIVE-6977[1] (Delete HiveServer1). The former needs a review the latter a patch * HIVE-6123[2] Checkstyle in Maven needs a review HIVE-7622[3] HIVE-7543[4] are waiting for any reviews or comments on my previous thread[5]. I'd still appreciate any helpers for reviews or even just comments. I'd feel very sad if I had done all that work for nothing. Hoping this thread gives me a wider audience. Both patches fix up issues that should have been caught in earlier reviews as they are almost all Checkstyle or other style violations but they make for huge patches. I could also create hundreds of small issues or stop doing these things entirely [0] https://issues.apache.org/jira/browse/HIVE-7107 [1] https://issues.apache.org/jira/browse/HIVE-6977 [2] https://issues.apache.org/jira/browse/HIVE-6123 [3] https://issues.apache.org/jira/browse/HIVE-7622 [4] https://issues.apache.org/jira/browse/HIVE-7543 On Fri, Aug 22, 2014 at 11:01 PM, John Pullokkaran -- Sent with Postbox http://www.getpostbox.com -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Commented] (HIVE-8007) Use proper Thrift comments
[
https://issues.apache.org/jira/browse/HIVE-8007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125456#comment-14125456
]
Hive QA commented on HIVE-8007:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667142/HIVE-8007.1.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 6185 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTbl
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchAbort
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/690/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/690/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-690/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667142
Use proper Thrift comments
--
Key: HIVE-8007
URL: https://issues.apache.org/jira/browse/HIVE-8007
Project: Hive
Issue Type: Improvement
Reporter: Lars Francke
Assignee: Lars Francke
Priority: Minor
Attachments: HIVE-8007.1.patch
Currently the thrift file uses {{//}} to denote comments. Thrift understands
the {{/** ... */}} syntax and converts that into documentation in the
generated code. This patch changes the syntax.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-1033) change default value of hive.exec.parallel to true
[
https://issues.apache.org/jira/browse/HIVE-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125488#comment-14125488
]
Hive QA commented on HIVE-1033:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667145/HIVE-1033.3.patch
{color:red}ERROR:{color} -1 due to 556 failed/errored test(s), 6185 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_add_part_exist
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_index
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_update_status
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_update_status
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_view_as_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguous_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_analyze_tbl_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_decimal
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_decimal_native
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_case_sensitivity
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_colstats_all_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnarserde_create_shortcut
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_partlvl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_partlvl_dp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_tbllvl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_confirm_initial_tbl_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog_dp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_func1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_like_view
Re: Review Request 17566: Support to query avro data stored in HBase
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17566/ --- (Updated Sept. 8, 2014, 1:51 p.m.) Review request for hive. Changes --- Updating patch with the latest changes. Bugs: HIVE-6147 https://issues.apache.org/jira/browse/HIVE-6147 Repository: hive-git Description --- Support to query avro data stored in HBase. Diffs (updated) - hbase-handler/if/test/avro_test.avpr PRE-CREATION hbase-handler/pom.xml 707e52c hbase-handler/src/java/org/apache/hadoop/hive/hbase/ColumnMappings.java 8735fbc hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java d184216 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseLazyObjectFactory.java 5c26456 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java fe6081e hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 6c1ce5c hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java PRE-CREATION hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java 8ea4408 hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java cedef10 hbase-handler/src/java/org/apache/hadoop/hive/hbase/struct/AvroHBaseValueFactory.java PRE-CREATION hbase-handler/src/java/org/apache/hadoop/hive/hbase/struct/DefaultHBaseValueFactory.java PRE-CREATION hbase-handler/src/java/org/apache/hadoop/hive/hbase/struct/HBaseValueFactory.java PRE-CREATION hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestAvroSchemaRetriever.java PRE-CREATION hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java e61e278 hbase-handler/src/test/org/apache/hadoop/hive/hbase/avro/Address.java PRE-CREATION hbase-handler/src/test/org/apache/hadoop/hive/hbase/avro/ContactInfo.java PRE-CREATION hbase-handler/src/test/org/apache/hadoop/hive/hbase/avro/Employee.java PRE-CREATION hbase-handler/src/test/org/apache/hadoop/hive/hbase/avro/EmployeeAvro.java PRE-CREATION hbase-handler/src/test/org/apache/hadoop/hive/hbase/avro/Gender.java PRE-CREATION hbase-handler/src/test/org/apache/hadoop/hive/hbase/avro/HomePhone.java PRE-CREATION hbase-handler/src/test/org/apache/hadoop/hive/hbase/avro/OfficePhone.java PRE-CREATION serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java 8d3595b serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java 402a4ac serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorException.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroObjectInspectorGenerator.java 46cdb4f serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaRetriever.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java 7c48e9b serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java d9a38ba serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java c74af4b serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUnion.java 9f6bc3f serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java ed670b0 serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java ddadfa5 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java 9a226b3 Diff: https://reviews.apache.org/r/17566/diff/ Testing --- New unit tests added to demonstrate the functionality. Thanks, Swarnim Kulkarni
[jira] [Commented] (HIVE-6147) Support avro data stored in HBase columns
[ https://issues.apache.org/jira/browse/HIVE-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125501#comment-14125501 ] Swarnim Kulkarni commented on HIVE-6147: [~brocknoland][~xuefuz] Updated RB for the patch: https://reviews.apache.org/r/17566/ Support avro data stored in HBase columns - Key: HIVE-6147 URL: https://issues.apache.org/jira/browse/HIVE-6147 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.12.0, 0.13.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-6147.1.patch.txt, HIVE-6147.2.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.4.patch.txt, HIVE-6147.5.patch.txt, HIVE-6147.6.patch.txt Presently, the HBase Hive integration supports querying only primitive data types in columns. It would be nice to be able to store and query Avro objects in HBase columns by making them visible as structs to Hive. This will allow Hive to perform ad hoc analysis of HBase data which can be deeply structured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-1879) Remove hive.metastore.metadb.dir property from hive-default.xml and HiveConf
[
https://issues.apache.org/jira/browse/HIVE-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125655#comment-14125655
]
Hive QA commented on HIVE-1879:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667147/HIVE-1879.1.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6185 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/692/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/692/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-692/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667147
Remove hive.metastore.metadb.dir property from hive-default.xml and HiveConf
Key: HIVE-1879
URL: https://issues.apache.org/jira/browse/HIVE-1879
Project: Hive
Issue Type: Bug
Components: Configuration, Metastore
Reporter: Carl Steinbach
Assignee: Lars Francke
Attachments: HIVE-1879.1.patch
The file-based MetaStore implementation was removed in HIVE-143. We also need
to
remove the hive.metastore.metadb.dir property from hive-default.xml and
HiveConf, as well
as the references to this property that currently appear in
HiveMetaStoreClient.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-2149) Fix ant target generate-schema
[ https://issues.apache.org/jira/browse/HIVE-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125676#comment-14125676 ] Ashutosh Chauhan commented on HIVE-2149: [~lars_francke] It will be good to have this functionality. Moving our build system has not provided this functionality nor made it obsolete. It will be good to have a maven target using which one can generate schema from jdo declaration. Shall we edit title of jira as Provide generate-schema target and reopen this? Fix ant target generate-schema --- Key: HIVE-2149 URL: https://issues.apache.org/jira/browse/HIVE-2149 Project: Hive Issue Type: Bug Reporter: Ashutosh Chauhan Priority: Minor Running generate-schema target in metastore dir results in generate-schema: [java] Exception in thread main java.lang.NoClassDefFoundError: org/jpox/SchemaTool -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-1363) 'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
[
https://issues.apache.org/jira/browse/HIVE-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125692#comment-14125692
]
Xuefu Zhang commented on HIVE-1363:
---
Thanks, [~leftylev]. It seems to me that is a just a bug fix.
'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
--
Key: HIVE-1363
URL: https://issues.apache.org/jira/browse/HIVE-1363
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.5.0
Reporter: Carl Steinbach
Assignee: Chaoyu Tang
Fix For: 0.14.0
Attachments: HIVE-1363.1.patch, HIVE-1363.2.patch, HIVE-1363.patch
{code}
hive SHOW TABLE EXTENDED LIKE pokes;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
hive SHOW TABLE EXTENDED LIKE p*;
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOUserException ')' expected at character 54 in database.name ==
dbName ( tableName.matches((?i)p.*)))
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive SHOW TABLE EXTENDED LIKE 'p*';
OK
hive SHOW TABLE EXTENDED LIKE `p*`;
OK
tableName:pokes
owner:carl
location:hdfs://localhost/user/hive/warehouse/pokes
inputformat:org.apache.hadoop.mapred.TextInputFormat
outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
columns:struct columns { i32 num}
partitioned:false
partitionColumns:
totalNumberFiles:0
totalFileSize:0
maxFileSize:0
minFileSize:0
lastAccessTime:0
lastUpdateTime:1274517075221
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8017) Use HiveKey instead of BytesWritable as key type of the pair RDD [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125702#comment-14125702
]
Xuefu Zhang commented on HIVE-8017:
---
Thanks, [~lirui]. Patch looks good to me. Could take a look at the above test
failures? Thanks.
Use HiveKey instead of BytesWritable as key type of the pair RDD [Spark
Branch]
---
Key: HIVE-8017
URL: https://issues.apache.org/jira/browse/HIVE-8017
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Rui Li
Assignee: Rui Li
Attachments: HIVE-8017-spark.patch
HiveKey should be used as the key type because it holds the hash code for
partitioning. While BytesWritable serves partitioning well for simple cases,
we have to use {{HiveKey.hashCode}} for more complicated ones, e.g. join,
bucketed table, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 25176: HIVE-7870: Insert overwrite table query does not generate correct task plan [Spark Branch]
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25176/#review52606 --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java https://reviews.apache.org/r/25176/#comment91453 Very well then. I was just wondering how it happens that way, as in the if block we have parent/child directory while for map-only work we don't need to do so. - Xuefu Zhang On Sept. 4, 2014, 5:03 p.m., Na Yang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25176/ --- (Updated Sept. 4, 2014, 5:03 p.m.) Review request for hive, Brock Noland, Szehon Ho, and Xuefu Zhang. Bugs: HIVE-7870 https://issues.apache.org/jira/browse/HIVE-7870 Repository: hive-git Description --- HIVE-7870: Insert overwrite table query does not generate correct task plan [Spark Branch] The cause of this problem is during spark/tez task generation, the union file sink operator are cloned to two new filesink operator. The linkedfilesinkdesc info for those new filesink operators are missing. In addition, the two new filesink operators also need to be linked together. Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 9c808d4 ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java 5ddc16d ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 379a39c ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 76fc290 ql/src/test/queries/clientpositive/union_remove_1.q c87b3fe ql/src/test/queries/clientpositive/union_remove_10.q 6701952 ql/src/test/queries/clientpositive/union_remove_11.q 4b2fa42 ql/src/test/queries/clientpositive/union_remove_12.q 69d0d0a ql/src/test/queries/clientpositive/union_remove_13.q 7605f0e ql/src/test/queries/clientpositive/union_remove_14.q a4fdfc8 ql/src/test/queries/clientpositive/union_remove_15.q e3c937b ql/src/test/queries/clientpositive/union_remove_16.q 537078b ql/src/test/queries/clientpositive/union_remove_17.q d70f3d3 ql/src/test/queries/clientpositive/union_remove_18.q 6352bc3 ql/src/test/queries/clientpositive/union_remove_19.q 8c45953 ql/src/test/queries/clientpositive/union_remove_2.q 83cd288 ql/src/test/queries/clientpositive/union_remove_20.q f80f7c1 ql/src/test/queries/clientpositive/union_remove_21.q 8963c25 ql/src/test/queries/clientpositive/union_remove_22.q b0c1ccd ql/src/test/queries/clientpositive/union_remove_23.q a1b989a ql/src/test/queries/clientpositive/union_remove_24.q ec561e0 ql/src/test/queries/clientpositive/union_remove_25.q 76c1ff5 ql/src/test/queries/clientpositive/union_remove_3.q 9617f73 ql/src/test/queries/clientpositive/union_remove_4.q cae323b ql/src/test/queries/clientpositive/union_remove_5.q 5df84e1 ql/src/test/queries/clientpositive/union_remove_6.q bfce26d ql/src/test/queries/clientpositive/union_remove_7.q 3a95674 ql/src/test/queries/clientpositive/union_remove_8.q a83a43e ql/src/test/queries/clientpositive/union_remove_9.q e71f6dd ql/src/test/results/clientpositive/spark/union10.q.out 20c681e ql/src/test/results/clientpositive/spark/union18.q.out 3f37a0a ql/src/test/results/clientpositive/spark/union19.q.out 6922fcd ql/src/test/results/clientpositive/spark/union28.q.out 8bd5218 ql/src/test/results/clientpositive/spark/union29.q.out b9546ef ql/src/test/results/clientpositive/spark/union3.q.out 3ae6536 ql/src/test/results/clientpositive/spark/union30.q.out 12717a1 ql/src/test/results/clientpositive/spark/union33.q.out b89757f ql/src/test/results/clientpositive/spark/union4.q.out 6341cd9 ql/src/test/results/clientpositive/spark/union6.q.out 263d9f4 ql/src/test/results/clientpositive/spark/union_remove_10.q.out 927a15d ql/src/test/results/clientpositive/spark/union_remove_11.q.out 96651e1 ql/src/test/results/clientpositive/spark/union_remove_16.q.out 0954ae4 ql/src/test/results/clientpositive/spark/union_remove_4.q.out cc46dda ql/src/test/results/clientpositive/spark/union_remove_5.q.out f6cdeb3 ql/src/test/results/clientpositive/spark/union_remove_9.q.out 1f0260c Diff: https://reviews.apache.org/r/25176/diff/ Testing --- Thanks, Na Yang
[jira] [Commented] (HIVE-1879) Remove hive.metastore.metadb.dir property from hive-default.xml and HiveConf
[ https://issues.apache.org/jira/browse/HIVE-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125716#comment-14125716 ] Ashutosh Chauhan commented on HIVE-1879: +1 Remove hive.metastore.metadb.dir property from hive-default.xml and HiveConf Key: HIVE-1879 URL: https://issues.apache.org/jira/browse/HIVE-1879 Project: Hive Issue Type: Bug Components: Configuration, Metastore Reporter: Carl Steinbach Assignee: Lars Francke Attachments: HIVE-1879.1.patch The file-based MetaStore implementation was removed in HIVE-143. We also need to remove the hive.metastore.metadb.dir property from hive-default.xml and HiveConf, as well as the references to this property that currently appear in HiveMetaStoreClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7870) Insert overwrite table query does not generate correct task plan [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125726#comment-14125726
]
Chao commented on HIVE-7870:
OK, I think I understand the code now (BTW, forgive me if I'm wrong, seems like
you can refactor the code for re-constructing linkedfilesinkdesc by removing
some common code).
Also, just curious, if I just remove the line
{{context.fileSinkSet.add(fileSink)}} in {{removeUnionOperators}}, will it
generate the same result?
Insert overwrite table query does not generate correct task plan [Spark
Branch]
---
Key: HIVE-7870
URL: https://issues.apache.org/jira/browse/HIVE-7870
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Na Yang
Assignee: Na Yang
Labels: Spark-M1
Attachments: HIVE-7870.1-spark.patch, HIVE-7870.2-spark.patch,
HIVE-7870.3-spark.patch, HIVE-7870.4-spark.patch, HIVE-7870.5-spark.patch
Insert overwrite table query does not generate correct task plan when
hive.optimize.union.remove and hive.merge.sparkfiles properties are ON.
{noformat}
set hive.optimize.union.remove=true
set hive.merge.sparkfiles=true
insert overwrite table outputTbl1
SELECT * FROM
(
select key, 1 as values from inputTbl1
union all
select * FROM (
SELECT key, count(1) as values from inputTbl1 group by key
UNION ALL
SELECT key, 2 as values from inputTbl1
) a
)b;
select * from outputTbl1 order by key, values;
{noformat}
query result
{noformat}
1 1
1 2
2 1
2 2
3 1
3 2
7 1
7 2
8 2
8 2
8 2
{noformat}
expected result:
{noformat}
1 1
1 1
1 2
2 1
2 1
2 2
3 1
3 1
3 2
7 1
7 1
7 2
8 1
8 1
8 2
8 2
8 2
{noformat}
Move work is not working properly and some data are missing during move.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7470) Wrong Thrift declaration for {{ShowCompactResponseElement}}
[
https://issues.apache.org/jira/browse/HIVE-7470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125727#comment-14125727
]
Hive QA commented on HIVE-7470:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667148/HIVE-7470.2.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6185 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/693/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/693/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-693/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667148
Wrong Thrift declaration for {{ShowCompactResponseElement}}
---
Key: HIVE-7470
URL: https://issues.apache.org/jira/browse/HIVE-7470
Project: Hive
Issue Type: Bug
Components: Metastore, Thrift API
Affects Versions: 0.14.0
Reporter: Damien Carol
Assignee: Damien Carol
Priority: Minor
Labels: metastore, thrift
Fix For: 0.14.0
Attachments: HIVE-7470.1.patch, HIVE-7470.2.patch
Prerequiste :
1. Remote metastore
2. Activate ACID and compactions
3. Launch ALTER TABLE foo COMPACT 'bar'
4. Call {{show_compact()}} on remote metastore
This use case throws exception in Thrift stack.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7503) Support Hive's multi-table insert query with Spark [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-7503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125736#comment-14125736 ] Chao commented on HIVE-7503: All the failed diffs are caused by either duplicated filesinks or multi-table insertion. I should probably start updating the related .q files for the latter. Support Hive's multi-table insert query with Spark [Spark Branch] - Key: HIVE-7503 URL: https://issues.apache.org/jira/browse/HIVE-7503 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Xuefu Zhang Assignee: Chao Labels: spark-m1 Attachments: HIVE-7503.1-spark.patch, HIVE-7503.2-spark.patch, HIVE-7503.3-spark.patch, HIVE-7503.4-spark.patch For Hive's multi insert query (https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML), there may be an MR job for each insert. When we achieve this with Spark, it would be nice if all the inserts can happen concurrently. It seems that this functionality isn't available in Spark. To make things worse, the source of the insert may be re-computed unless it's staged. Even with this, the inserts will happen sequentially, making the performance suffer. This task is to find out what takes in Spark to enable this without requiring staging the source and sequential insertion. If this has to be solved in Hive, find out an optimum way to do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-2390) Expand support for union types
[
https://issues.apache.org/jira/browse/HIVE-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125738#comment-14125738
]
Thejas M Nair commented on HIVE-2390:
-
[~suma.shivaprasad] Can you please add information to the release notes section
(click on edit jira to find it), that can be used to document the change from
this jira in wiki ?
Expand support for union types
--
Key: HIVE-2390
URL: https://issues.apache.org/jira/browse/HIVE-2390
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Jakob Homan
Assignee: Suma Shivaprasad
Labels: uniontype
Fix For: 0.14.0
Attachments: HIVE-2390.1.patch, HIVE-2390.patch
When the union type was introduced, full support for it wasn't provided. For
instance, when working with a union that gets passed to LazyBinarySerde:
{noformat}Caused by: java.lang.RuntimeException: Unrecognized type: UNION
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:468)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serializeStruct(LazyBinarySerDe.java:230)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:184)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7503) Support Hive's multi-table insert query with Spark [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-7503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao updated HIVE-7503: --- Attachment: HIVE-7503.5-spark.patch This patch retains the duplicated filesinks (as an alternative). Support Hive's multi-table insert query with Spark [Spark Branch] - Key: HIVE-7503 URL: https://issues.apache.org/jira/browse/HIVE-7503 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Xuefu Zhang Assignee: Chao Labels: spark-m1 Attachments: HIVE-7503.1-spark.patch, HIVE-7503.2-spark.patch, HIVE-7503.3-spark.patch, HIVE-7503.4-spark.patch, HIVE-7503.5-spark.patch For Hive's multi insert query (https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML), there may be an MR job for each insert. When we achieve this with Spark, it would be nice if all the inserts can happen concurrently. It seems that this functionality isn't available in Spark. To make things worse, the source of the insert may be re-computed unless it's staged. Even with this, the inserts will happen sequentially, making the performance suffer. This task is to find out what takes in Spark to enable this without requiring staging the source and sequential insertion. If this has to be solved in Hive, find out an optimum way to do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7702) Start running .q file tests on spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-7702:
---
Description:
Spark can currently only support a few queries, however there are some .q file
tests which will pass today. The basic idea is that we should get some number
of these actually working (10-20) so we can actually start testing the project.
A good starting point might be the udf*, varchar*, or alter* tests:
https://github.com/apache/hive/tree/spark/ql/src/test/queries/clientpositive
To generate the output file for test XXX.q, you'd do:
{noformat}
mvn clean install -DskipTests -Phadoop-2
cd itests
mvn clean install -DskipTests -Phadoop-2
cd qtest-spark
mvn test -Dtest= TestSparkCliDriver -Dqfile=XXX.q -Dtest.output.overwrite=true
-Phadoop-2
{noformat}
which would generate XXX.q.out which we can check-in to source control as a
golden file.
Multiple tests can be run at a give time as so:
{noformat}
mvn test -Dtest= TestSparkCliDriver -Dqfile=X1.q,X2.q
-Dtest.output.overwrite=true -Phadoop-2
{noformat}
was:
Spark can currently only support a few queries, however there are some .q file
tests which will pass today. The basic idea is that we should get some number
of these actually working (10-20) so we can actually start testing the project.
A good starting point might be the udf*, varchar*, or alter* tests:
https://github.com/apache/hive/tree/spark/ql/src/test/queries/clientpositive
To generate the output file for test XXX.q, you'd do:
{noformat}
mvn clean install -DskipTests -Phadoop-2
cd itests
mvn clean install -DskipTests -Phadoop-2
cd qtest-spark
mvn test -Dtest=TestCliDriver -Dqfile=XXX.q -Dtest.output.overwrite=true
-Phadoop-2
{noformat}
which would generate XXX.q.out which we can check-in to source control as a
golden file.
Multiple tests can be run at a give time as so:
{noformat}
mvn test -Dtest=TestCliDriver -Dqfile=X1.q,X2.q -Dtest.output.overwrite=true
-Phadoop-2
{noformat}
Start running .q file tests on spark [Spark Branch]
---
Key: HIVE-7702
URL: https://issues.apache.org/jira/browse/HIVE-7702
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Chinna Rao Lalam
Fix For: spark-branch
Attachments: HIVE-7702-spark.patch, HIVE-7702.1-spark.patch
Spark can currently only support a few queries, however there are some .q
file tests which will pass today. The basic idea is that we should get some
number of these actually working (10-20) so we can actually start testing the
project.
A good starting point might be the udf*, varchar*, or alter* tests:
https://github.com/apache/hive/tree/spark/ql/src/test/queries/clientpositive
To generate the output file for test XXX.q, you'd do:
{noformat}
mvn clean install -DskipTests -Phadoop-2
cd itests
mvn clean install -DskipTests -Phadoop-2
cd qtest-spark
mvn test -Dtest= TestSparkCliDriver -Dqfile=XXX.q
-Dtest.output.overwrite=true -Phadoop-2
{noformat}
which would generate XXX.q.out which we can check-in to source control as a
golden file.
Multiple tests can be run at a give time as so:
{noformat}
mvn test -Dtest= TestSparkCliDriver -Dqfile=X1.q,X2.q
-Dtest.output.overwrite=true -Phadoop-2
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 24630: HIVE-7694 - SMB joins on tables differing by number of sorted by columns but same sort prefix and join keys fail
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24630/ --- (Updated Sept. 8, 2014, 5:25 p.m.) Review request for hive, Amareshwari Sriramadasu, Brock Noland, Gunther Hagleitner, and Navis Ryu. Changes --- Added another reviewer Bugs: HIVE-7694 https://issues.apache.org/jira/browse/HIVE-7694 Repository: hive-git Description --- For eg: If two tables T1 sorted by (a, b, c) clustered by a and T2 sorted by (a) and clustered by (a) are joined, an exception is seen as reported in https://issues.apache.org/jira/browse/HIVE-7694 Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java 0b7b1a3 ql/src/test/queries/clientpositive/sort_merge_join_desc_8.q PRE-CREATION ql/src/test/results/clientpositive/sort_merge_join_desc_8.q.out PRE-CREATION Diff: https://reviews.apache.org/r/24630/diff/ Testing (updated) --- sort_merge_join_desc_8.q added for testing the above cases Thanks, Suma Shivaprasad
[jira] [Commented] (HIVE-6147) Support avro data stored in HBase columns
[ https://issues.apache.org/jira/browse/HIVE-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125791#comment-14125791 ] Brock Noland commented on HIVE-6147: [~swarnim] for some reason I cannot change the JIRA to Patch Available so tests can run. Do you have the button? Support avro data stored in HBase columns - Key: HIVE-6147 URL: https://issues.apache.org/jira/browse/HIVE-6147 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.12.0, 0.13.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-6147.1.patch.txt, HIVE-6147.2.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.4.patch.txt, HIVE-6147.5.patch.txt, HIVE-6147.6.patch.txt Presently, the HBase Hive integration supports querying only primitive data types in columns. It would be nice to be able to store and query Avro objects in HBase columns by making them visible as structs to Hive. This will allow Hive to perform ad hoc analysis of HBase data which can be deeply structured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-5545) HCatRecord getInteger method returns String when used on Partition columns of type INT
[
https://issues.apache.org/jira/browse/HIVE-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13795209#comment-13795209
]
Thejas M Nair edited comment on HIVE-5545 at 9/8/14 5:54 PM:
-
Mapper Class-
{code}
public class testMapper extends MapperWritableComparable, HCatRecord,
IntWritable, IntWritable {
@Override
protected void map(WritableComparable key, HCatRecord value,
org.apache.hadoop.mapreduce.MapperWritableComparable, HCatRecord,
IntWritable, IntWritable.Context context)
throws IOException, InterruptedException {
// Get table schema
HCatSchema schema = HCatBaseInputFormat.getTableSchema(context);
// Reading year column as Integer
Integer year = new Integer(value.getInteger(year, schema));
context.write(new IntWritable(year), new IntWritable(1));
}
}
{code}
was (Author: rishav_rohit):
Mapper Class-
public class testMapper extends MapperWritableComparable, HCatRecord,
IntWritable, IntWritable {
@Override
protected void map(WritableComparable key, HCatRecord value,
org.apache.hadoop.mapreduce.MapperWritableComparable, HCatRecord,
IntWritable, IntWritable.Context context)
throws IOException, InterruptedException {
// Get table schema
HCatSchema schema = HCatBaseInputFormat.getTableSchema(context);
// Reading year column as Integer
Integer year = new Integer(value.getInteger(year, schema));
context.write(new IntWritable(year), new IntWritable(1));
}
}
HCatRecord getInteger method returns String when used on Partition columns of
type INT
--
Key: HIVE-5545
URL: https://issues.apache.org/jira/browse/HIVE-5545
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.11.0
Environment: hadoop-1.0.3
Reporter: Rishav Rohit
HCatRecord getInteger method returns String when used on Partition columns of
type INT.
java.lang.ClassCastException: java.lang.String cannot be cast to
java.lang.Integer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-5545) HCatRecord getInteger method returns String when used on Partition columns of type INT
[
https://issues.apache.org/jira/browse/HIVE-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13795210#comment-13795210
]
Thejas M Nair edited comment on HIVE-5545 at 9/8/14 5:55 PM:
-
Reducer Class -
{code}
public class testReducer extends ReducerIntWritable, IntWritable, Text,
HCatRecord {
public void reduce (IntWritable key, IterableIntWritable value,
Context context)
throws IOException, InterruptedException{
int count = 0;
for (IntWritable s:value) {
count++;
}
// define output
ListHCatFieldSchema columns = new
ArrayListHCatFieldSchema(2);
columns.add(new HCatFieldSchema(year,
HCatFieldSchema.Type.INT, ));
columns.add(new HCatFieldSchema(count,
HCatFieldSchema.Type.INT, ));
HCatSchema schema = new HCatSchema(columns);
HCatRecord record = new DefaultHCatRecord(2);
record.setInteger(year, schema, new Integer(key.toString()));
record.set(cnt, schema, new Integer(count));
context.write(null, record);
}
}
{code}
was (Author: rishav_rohit):
Reducer Class -
public class testReducer extends ReducerIntWritable, IntWritable, Text,
HCatRecord {
public void reduce (IntWritable key, IterableIntWritable value,
Context context)
throws IOException, InterruptedException{
int count = 0;
for (IntWritable s:value) {
count++;
}
// define output
ListHCatFieldSchema columns = new
ArrayListHCatFieldSchema(2);
columns.add(new HCatFieldSchema(year,
HCatFieldSchema.Type.INT, ));
columns.add(new HCatFieldSchema(count,
HCatFieldSchema.Type.INT, ));
HCatSchema schema = new HCatSchema(columns);
HCatRecord record = new DefaultHCatRecord(2);
record.setInteger(year, schema, new Integer(key.toString()));
record.set(cnt, schema, new Integer(count));
context.write(null, record);
}
}
HCatRecord getInteger method returns String when used on Partition columns of
type INT
--
Key: HIVE-5545
URL: https://issues.apache.org/jira/browse/HIVE-5545
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.11.0
Environment: hadoop-1.0.3
Reporter: Rishav Rohit
HCatRecord getInteger method returns String when used on Partition columns of
type INT.
java.lang.ClassCastException: java.lang.String cannot be cast to
java.lang.Integer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-5545) HCatRecord getInteger method returns String when used on Partition columns of type INT
[
https://issues.apache.org/jira/browse/HIVE-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13795206#comment-13795206
]
Thejas M Nair edited comment on HIVE-5545 at 9/8/14 5:54 PM:
-
I am posting here my Hive table data and MapReduce code alongwith Error thrown:
{noformat}
cat test_data
2013,1,1
2013,1,3
2013,2,18
2013,2,20
2013,3,30
2013,3,31
create schema test;
use test;
### This is temporary table
create table unpar (
year INT,
month INT,
day INT)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE;
LOAD DATA LOCAL INPATH test_data OVERWRITE INTO TABLE test.unpar;
### This is input table and I will try to read year column using HCatalog.
create table par
(day INT)
PARTITIONED BY (year INT, month INT )
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE;
INSERT OVERWRITE TABLE par PARTITION(Year, Month) SELECT day, year, month from
unpar;
### this is output table
create table out
(year INT,
cnt INT)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE;
{noformat}
was (Author: rishav_rohit):
I am posting here my Hive table data and MapReduce code alongwith Error thrown:
cat test_data
2013,1,1
2013,1,3
2013,2,18
2013,2,20
2013,3,30
2013,3,31
create schema test;
use test;
### This is temporary table
create table unpar (
year INT,
month INT,
day INT)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE;
LOAD DATA LOCAL INPATH test_data OVERWRITE INTO TABLE test.unpar;
### This is input table and I will try to read year column using HCatalog.
create table par
(day INT)
PARTITIONED BY (year INT, month INT )
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE;
INSERT OVERWRITE TABLE par PARTITION(Year, Month) SELECT day, year, month from
unpar;
### this is output table
create table out
(year INT,
cnt INT)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE;
HCatRecord getInteger method returns String when used on Partition columns of
type INT
--
Key: HIVE-5545
URL: https://issues.apache.org/jira/browse/HIVE-5545
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.11.0
Environment: hadoop-1.0.3
Reporter: Rishav Rohit
HCatRecord getInteger method returns String when used on Partition columns of
type INT.
java.lang.ClassCastException: java.lang.String cannot be cast to
java.lang.Integer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-5545) HCatRecord getInteger method returns String when used on Partition columns of type INT
[
https://issues.apache.org/jira/browse/HIVE-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13795211#comment-13795211
]
Thejas M Nair edited comment on HIVE-5545 at 9/8/14 5:55 PM:
-
Driver Class -
{code}
public class testDriver extends Configured implements Tool{
private static final Log log = LogFactory.getLog( testDriver.class );
public int run( String[] args ) throws Exception{
Configuration conf = new Configuration();
Job job = new Job(conf, test_catalog);
job.setJarByClass(testDriver.class);
job.setMapperClass(testMapper.class);
job.setReducerClass(testReducer.class);
HCatInputFormat.setInput(job, test, par);
// HCatInputFormat.setInput(job, InputJobInfo.create(airline,
ontimeperf, year='1987'));
job.setInputFormatClass(HCatInputFormat.class);
job.setMapOutputKeyClass(IntWritable.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(DefaultHCatRecord.class);
job.setOutputFormatClass(HCatOutputFormat.class);
HCatOutputFormat.setOutput(job, OutputJobInfo.create(test, out,
null));
HCatSchema s = HCatOutputFormat.getTableSchema(job);
HCatOutputFormat.setSchema(job, s);
return (job.waitForCompletion(true)? 0:1);
}
public static void main(String[] args) throws Exception{
int exitCode = ToolRunner.run(new testDriver(), args);
System.exit(exitCode);
}
}
{code}
was (Author: rishav_rohit):
Driver Class -
public class testDriver extends Configured implements Tool{
private static final Log log = LogFactory.getLog( testDriver.class );
public int run( String[] args ) throws Exception{
Configuration conf = new Configuration();
Job job = new Job(conf, test_catalog);
job.setJarByClass(testDriver.class);
job.setMapperClass(testMapper.class);
job.setReducerClass(testReducer.class);
HCatInputFormat.setInput(job, test, par);
// HCatInputFormat.setInput(job, InputJobInfo.create(airline,
ontimeperf, year='1987'));
job.setInputFormatClass(HCatInputFormat.class);
job.setMapOutputKeyClass(IntWritable.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(DefaultHCatRecord.class);
job.setOutputFormatClass(HCatOutputFormat.class);
HCatOutputFormat.setOutput(job, OutputJobInfo.create(test, out,
null));
HCatSchema s = HCatOutputFormat.getTableSchema(job);
HCatOutputFormat.setSchema(job, s);
return (job.waitForCompletion(true)? 0:1);
}
public static void main(String[] args) throws Exception{
int exitCode = ToolRunner.run(new testDriver(), args);
System.exit(exitCode);
}
}
HCatRecord getInteger method returns String when used on Partition columns of
type INT
--
Key: HIVE-5545
URL: https://issues.apache.org/jira/browse/HIVE-5545
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.11.0
Environment: hadoop-1.0.3
Reporter: Rishav Rohit
HCatRecord getInteger method returns String when used on Partition columns of
type INT.
java.lang.ClassCastException: java.lang.String cannot be cast to
java.lang.Integer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8016) CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup
[ https://issues.apache.org/jira/browse/HIVE-8016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125820#comment-14125820 ] Harish Butani commented on HIVE-8016: - +1 CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup - Key: HIVE-8016 URL: https://issues.apache.org/jira/browse/HIVE-8016 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Attachments: HIVE-8016.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7689) Enable Postgres as METASTORE back-end
[
https://issues.apache.org/jira/browse/HIVE-7689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125829#comment-14125829
]
Hive QA commented on HIVE-7689:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667153/HIVE-7689.5.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 6185 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorTableLegacy
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorTableNoBase
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorTableWithBase
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorTableLegacy
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorTableNoBase
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorTableWithBase
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/694/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/694/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-694/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667153
Enable Postgres as METASTORE back-end
-
Key: HIVE-7689
URL: https://issues.apache.org/jira/browse/HIVE-7689
Project: Hive
Issue Type: Improvement
Components: Metastore
Affects Versions: 0.14.0
Reporter: Damien Carol
Assignee: Damien Carol
Priority: Minor
Labels: metastore, postgres
Fix For: 0.14.0
Attachments: HIVE-7689.5.patch, HIVE-7889.1.patch, HIVE-7889.2.patch,
HIVE-7889.3.patch, HIVE-7889.4.patch
I maintain few patches to make Metastore works with Postgres back end in our
production environment.
The main goal of this JIRA is to push upstream these patches.
This patch enable LOCKS, COMPACTION and fix error in STATS on postgres
metastore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7100) Users of hive should be able to specify skipTrash when dropping tables.
[ https://issues.apache.org/jira/browse/HIVE-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125831#comment-14125831 ] david serafini commented on HIVE-7100: -- I think both of these ideas are good, but beyond the scope of this ticket. dropPartitions is certainly worth doing, but is different from the use case for dropTable. And changing the table configuration should be discussed first to make sure it's a good idea. I'd prefer to finish this ticket as-is and open a new ticket for each of the other ideas. Users of hive should be able to specify skipTrash when dropping tables. --- Key: HIVE-7100 URL: https://issues.apache.org/jira/browse/HIVE-7100 Project: Hive Issue Type: Improvement Affects Versions: 0.13.0 Reporter: Ravi Prakash Assignee: Jayesh Attachments: HIVE-7100.1.patch, HIVE-7100.2.patch, HIVE-7100.3.patch, HIVE-7100.4.patch, HIVE-7100.5.patch, HIVE-7100.patch Users of our clusters are often running up against their quota limits because of Hive tables. When they drop tables, they have to then manually delete the files from HDFS using skipTrash. This is cumbersome and unnecessary. We should enable users to skipTrash directly when dropping tables. We should also be able to provide this functionality without polluting SQL syntax. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8019) Missing commit from trunk : `export/import statement update`
Mohit Sabharwal created HIVE-8019:
-
Summary: Missing commit from trunk : `export/import statement
update`
Key: HIVE-8019
URL: https://issues.apache.org/jira/browse/HIVE-8019
Project: Hive
Issue Type: Bug
Components: Import/Export
Affects Versions: 0.14.0
Reporter: Mohit Sabharwal
Priority: Blocker
Noticed that commit 1882de7810fc55a2466dd4cbe74ed67bb41cb667 exists in 0.13
branch, but not it trunk.
https://github.com/apache/hive/commit/1882de7810fc55a2466dd4cbe74ed67bb41cb667
{code}
(trunk) $ git branch -a --contains 1882de7810fc55a2466dd4cbe74ed67bb41cb667
remotes/origin/branch-0.13
{code}
I looked through some of the changes in this commit and don't see those in
trunk. Nor do I see a commit that reverts these changes in trunk.
[~thejas], should we port this over to trunk ?
Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8019) Missing commit from trunk : `export/import statement update`
[
https://issues.apache.org/jira/browse/HIVE-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-8019:
--
Affects Version/s: (was: 0.14.0)
Missing commit from trunk : `export/import statement update`
Key: HIVE-8019
URL: https://issues.apache.org/jira/browse/HIVE-8019
Project: Hive
Issue Type: Bug
Components: Import/Export
Reporter: Mohit Sabharwal
Priority: Blocker
Noticed that commit 1882de7810fc55a2466dd4cbe74ed67bb41cb667 exists in 0.13
branch, but not it trunk.
https://github.com/apache/hive/commit/1882de7810fc55a2466dd4cbe74ed67bb41cb667
{code}
(trunk) $ git branch -a --contains 1882de7810fc55a2466dd4cbe74ed67bb41cb667
remotes/origin/branch-0.13
{code}
I looked through some of the changes in this commit and don't see those in
trunk. Nor do I see a commit that reverts these changes in trunk.
[~thejas], should we port this over to trunk ?
Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8019) Missing commit from trunk : `export/import statement update`
[
https://issues.apache.org/jira/browse/HIVE-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-8019:
--
Affects Version/s: 0.14.0
Missing commit from trunk : `export/import statement update`
Key: HIVE-8019
URL: https://issues.apache.org/jira/browse/HIVE-8019
Project: Hive
Issue Type: Bug
Components: Import/Export
Affects Versions: 0.14.0
Reporter: Mohit Sabharwal
Priority: Blocker
Noticed that commit 1882de7810fc55a2466dd4cbe74ed67bb41cb667 exists in 0.13
branch, but not it trunk.
https://github.com/apache/hive/commit/1882de7810fc55a2466dd4cbe74ed67bb41cb667
{code}
(trunk) $ git branch -a --contains 1882de7810fc55a2466dd4cbe74ed67bb41cb667
remotes/origin/branch-0.13
{code}
I looked through some of the changes in this commit and don't see those in
trunk. Nor do I see a commit that reverts these changes in trunk.
[~thejas], should we port this over to trunk ?
Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7503) Support Hive's multi-table insert query with Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125841#comment-14125841
]
Hive QA commented on HIVE-7503:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667195/HIVE-7503.5-spark.patch
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 6343 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union28
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union29
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union30
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union33
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_6
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/118/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/118/console
Test logs:
http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-118/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667195
Support Hive's multi-table insert query with Spark [Spark Branch]
-
Key: HIVE-7503
URL: https://issues.apache.org/jira/browse/HIVE-7503
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Xuefu Zhang
Assignee: Chao
Labels: spark-m1
Attachments: HIVE-7503.1-spark.patch, HIVE-7503.2-spark.patch,
HIVE-7503.3-spark.patch, HIVE-7503.4-spark.patch, HIVE-7503.5-spark.patch
For Hive's multi insert query
(https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML), there
may be an MR job for each insert. When we achieve this with Spark, it would
be nice if all the inserts can happen concurrently.
It seems that this functionality isn't available in Spark. To make things
worse, the source of the insert may be re-computed unless it's staged. Even
with this, the inserts will happen sequentially, making the performance
suffer.
This task is to find out what takes in Spark to enable this without requiring
staging the source and sequential insertion. If this has to be solved in
Hive, find out an optimum way to do this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8011) Fix couple of tests to work when FileSystem.access() is available
[ https://issues.apache.org/jira/browse/HIVE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125844#comment-14125844 ] Thejas M Nair commented on HIVE-8011: - +1 Fix couple of tests to work when FileSystem.access() is available - Key: HIVE-8011 URL: https://issues.apache.org/jira/browse/HIVE-8011 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-8011.1.patch Couple test failures when running against versions of Hadoop that support FileSystem.access(): TestStorageBasedMetastoreAuthorizationProviderWithACL: Fails with the extra tests added in HIVE-6093. The parent class does not override disallowCreateDatabase(), and it looks like this test should not override it either either. TestStorageBasedClientSideAuthorizationProvider: Should change the error message string we're checking against. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-8019) Missing commit from trunk : `export/import statement update`
[
https://issues.apache.org/jira/browse/HIVE-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair reassigned HIVE-8019:
---
Assignee: Thejas M Nair
Missing commit from trunk : `export/import statement update`
Key: HIVE-8019
URL: https://issues.apache.org/jira/browse/HIVE-8019
Project: Hive
Issue Type: Bug
Components: Import/Export
Affects Versions: 0.14.0
Reporter: Mohit Sabharwal
Assignee: Thejas M Nair
Priority: Blocker
Noticed that commit 1882de7810fc55a2466dd4cbe74ed67bb41cb667 exists in 0.13
branch, but not it trunk.
https://github.com/apache/hive/commit/1882de7810fc55a2466dd4cbe74ed67bb41cb667
{code}
(trunk) $ git branch -a --contains 1882de7810fc55a2466dd4cbe74ed67bb41cb667
remotes/origin/branch-0.13
{code}
I looked through some of the changes in this commit and don't see those in
trunk. Nor do I see a commit that reverts these changes in trunk.
[~thejas], should we port this over to trunk ?
Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8019) Missing commit from trunk : `export/import statement update`
[
https://issues.apache.org/jira/browse/HIVE-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125850#comment-14125850
]
Thejas M Nair commented on HIVE-8019:
-
[~mohitsabharwal] Thanks for pointing that out! I will port the patch. Some
q.out files would also need to be regenerated.
Missing commit from trunk : `export/import statement update`
Key: HIVE-8019
URL: https://issues.apache.org/jira/browse/HIVE-8019
Project: Hive
Issue Type: Bug
Components: Import/Export
Affects Versions: 0.14.0
Reporter: Mohit Sabharwal
Assignee: Thejas M Nair
Priority: Blocker
Noticed that commit 1882de7810fc55a2466dd4cbe74ed67bb41cb667 exists in 0.13
branch, but not it trunk.
https://github.com/apache/hive/commit/1882de7810fc55a2466dd4cbe74ed67bb41cb667
{code}
(trunk) $ git branch -a --contains 1882de7810fc55a2466dd4cbe74ed67bb41cb667
remotes/origin/branch-0.13
{code}
I looked through some of the changes in this commit and don't see those in
trunk. Nor do I see a commit that reverts these changes in trunk.
[~thejas], should we port this over to trunk ?
Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8012) TestHiveServer2Concurrency is not implemented
[
https://issues.apache.org/jira/browse/HIVE-8012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-8012:
-
Attachment: HIVE-8012.1.patch
Ok, let's just remove this test then. Attaching patch.
TestHiveServer2Concurrency is not implemented
-
Key: HIVE-8012
URL: https://issues.apache.org/jira/browse/HIVE-8012
Project: Hive
Issue Type: Bug
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-8012.1.patch
{code}
@Test
public void test() {
fail(Not yet implemented);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5545) HCatRecord getInteger method returns String when used on Partition columns of type INT
[ https://issues.apache.org/jira/browse/HIVE-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125855#comment-14125855 ] Eugene Koifman commented on HIVE-5545: -- HCat only supports partition columns of type String. Some more info can be found in HIVE-7164 and email thread linked form there. HCatRecord getInteger method returns String when used on Partition columns of type INT -- Key: HIVE-5545 URL: https://issues.apache.org/jira/browse/HIVE-5545 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.11.0 Environment: hadoop-1.0.3 Reporter: Rishav Rohit HCatRecord getInteger method returns String when used on Partition columns of type INT. java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6147) Support avro data stored in HBase columns
[ https://issues.apache.org/jira/browse/HIVE-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125870#comment-14125870 ] Swarnim Kulkarni commented on HIVE-6147: [~brocknoland] Just did that. Support avro data stored in HBase columns - Key: HIVE-6147 URL: https://issues.apache.org/jira/browse/HIVE-6147 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.12.0, 0.13.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-6147.1.patch.txt, HIVE-6147.2.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.4.patch.txt, HIVE-6147.5.patch.txt, HIVE-6147.6.patch.txt Presently, the HBase Hive integration supports querying only primitive data types in columns. It would be nice to be able to store and query Avro objects in HBase columns by making them visible as structs to Hive. This will allow Hive to perform ad hoc analysis of HBase data which can be deeply structured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6147) Support avro data stored in HBase columns
[ https://issues.apache.org/jira/browse/HIVE-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swarnim Kulkarni updated HIVE-6147: --- Status: Patch Available (was: In Progress) Support avro data stored in HBase columns - Key: HIVE-6147 URL: https://issues.apache.org/jira/browse/HIVE-6147 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.13.0, 0.12.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-6147.1.patch.txt, HIVE-6147.2.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.4.patch.txt, HIVE-6147.5.patch.txt, HIVE-6147.6.patch.txt Presently, the HBase Hive integration supports querying only primitive data types in columns. It would be nice to be able to store and query Avro objects in HBase columns by making them visible as structs to Hive. This will allow Hive to perform ad hoc analysis of HBase data which can be deeply structured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7868) AvroSerDe error handling could be improved
[
https://issues.apache.org/jira/browse/HIVE-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125879#comment-14125879
]
Brock Noland commented on HIVE-7868:
This looks good! Using the following tables:
{noformat}
create table test_avro (c1 string, c2 char(10), c3 varchar(10))
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.literal'='{type:record,name:test_avro,namespace:default,fields:[{name:c1,type:[null,string],default:null},{name:c2,type:[null,string],default:null},{name:c3,type:[null,string],default:null}]}');
create table test_avro (c1 string, c2 char(10), c3 varchar(10))
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs://localhost:9000/tmp/schema.avsc');
{noformat}
* Creating with bad avro.schema.literal:
{noformat}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException:
MetaException(message:org.apache.hadoop.hive.serde2.SerDeException Encountered
exception determining schema. Returning signal schema to indicate problem: No
type: {})
{noformat}
* Creating with bad avro.schema.url:
{noformat}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException:
MetaException(message:org.apache.hadoop.hive.serde2.SerDeException Encountered
AvroSerdeException determining schema. Returning signal schema to indicate
problem: Unable to read schema from given path:
hdfs://localhost:8020/tmp/schema.avsc)
{noformat}
* Setting bad avro.schema.url:
{noformat}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
org.apache.hadoop.hive.serde2.SerDeException Encountered AvroSerdeException
determining schema. Returning signal schema to indicate problem: Unable to read
schema from given path: hdfs://localhost:9000/tmp/schema.avsc
{noformat}
* Setting bad avro.schema.literal
{noformat}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
org.apache.hadoop.hive.serde2.SerDeException Encountered exception determining
schema. Returning signal schema to indicate problem: java.io.EOFException: No
content to map to Object due to end of input
{noformat}
* Fixing bad URL schema works.
* Fixing bad literal schema works.
I think we should do one more item, in the describe table code here:
https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L3063
we should check for config errors and print them as opposed to describing the
table.
Thanks!!
AvroSerDe error handling could be improved
--
Key: HIVE-7868
URL: https://issues.apache.org/jira/browse/HIVE-7868
Project: Hive
Issue Type: Improvement
Reporter: Brock Noland
Assignee: Ferdinand Xu
Attachments: HIVE-7868.patch
When an Avro schema is invalid, AvroSerDe returns an error message instead of
throwing an exception. This is described in
{{AvroSerdeUtils.determineSchemaOrReturnErrorSchema}}:
{noformat}
/**
* Attempt to determine the schema via the usual means, but do not throw
* an exception if we fail. Instead, signal failure via a special
* schema. This is used because Hive calls init on the serde during
* any call, including calls to update the serde properties, meaning
* if the serde is in a bad state, there is no way to update that state.
*/
{noformat}
I believe we should find a way to provide a better experience to our users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8012) TestHiveServer2Concurrency is not implemented
[
https://issues.apache.org/jira/browse/HIVE-8012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125889#comment-14125889
]
Brock Noland commented on HIVE-8012:
+1
TestHiveServer2Concurrency is not implemented
-
Key: HIVE-8012
URL: https://issues.apache.org/jira/browse/HIVE-8012
Project: Hive
Issue Type: Bug
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-8012.1.patch
{code}
@Test
public void test() {
fail(Not yet implemented);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7100) Users of hive should be able to specify skipTrash when dropping tables.
[ https://issues.apache.org/jira/browse/HIVE-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125895#comment-14125895 ] Xuefu Zhang commented on HIVE-7100: --- [~dbsalti] I agree that same option when dropping partition can be done via a different JIRA. However, the comments on RB about the APIs seems valid. and what's your thought on that. I'd prefer we add new APIs to make the param passing consistent. The old API can be gradually deprecated. Users of hive should be able to specify skipTrash when dropping tables. --- Key: HIVE-7100 URL: https://issues.apache.org/jira/browse/HIVE-7100 Project: Hive Issue Type: Improvement Affects Versions: 0.13.0 Reporter: Ravi Prakash Assignee: Jayesh Attachments: HIVE-7100.1.patch, HIVE-7100.2.patch, HIVE-7100.3.patch, HIVE-7100.4.patch, HIVE-7100.5.patch, HIVE-7100.patch Users of our clusters are often running up against their quota limits because of Hive tables. When they drop tables, they have to then manually delete the files from HDFS using skipTrash. This is cumbersome and unnecessary. We should enable users to skipTrash directly when dropping tables. We should also be able to provide this functionality without polluting SQL syntax. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8016) CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup
[ https://issues.apache.org/jira/browse/HIVE-8016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-8016: Fix Version/s: 0.14.0 CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup - Key: HIVE-8016 URL: https://issues.apache.org/jira/browse/HIVE-8016 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 0.14.0 Attachments: HIVE-8016.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8016) CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup
[ https://issues.apache.org/jira/browse/HIVE-8016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-8016: Resolution: Fixed Status: Resolved (was: Patch Available) Committed to CBO branch thanks [~jpullokkaran] CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup - Key: HIVE-8016 URL: https://issues.apache.org/jira/browse/HIVE-8016 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Attachments: HIVE-8016.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 25178: Add DROP TABLE PURGE
On Sept. 4, 2014, 9:39 p.m., Mithun Radhakrishnan wrote:
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java,
line 1409
https://reviews.apache.org/r/25178/diff/2/?file=674619#file674619line1409
Shouldn't you be passing the environment context here? ifPurge must
apply to both table and partition data.
Indeed. Fixed in the next patch.
On Sept. 4, 2014, 9:39 p.m., Mithun Radhakrishnan wrote:
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java,
line 828
https://reviews.apache.org/r/25178/diff/2/?file=674620#file674620line828
Fix whitespace for this line, and the if{} block.
Fixed in the next patch.
On Sept. 4, 2014, 9:39 p.m., Mithun Radhakrishnan wrote:
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java,
lines 824-826
https://reviews.apache.org/r/25178/diff/2/?file=674620#file674620line824
I wish this could be made more uniform. The ifPurge is being passed
into the old `dropTable()` in an envContext, but `ignoreUnknown`,
`deleteData`, etc. are not. I understand your predicament in changing the
signature of an existing public interface method.
I see your point, but I'm not so sure. The ignoreUnknownTab arg doesn't belong
in envContext anyway, since it doesn't propagate that far down. If deleteData
is removed from this API, it should also be removed from dropDatabase(),
dropIndex() and dropPartition(), which do propagate down into the lower classes
and maybe justifies a separate ticket.
On Sept. 4, 2014, 9:39 p.m., Mithun Radhakrishnan wrote:
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 1748
https://reviews.apache.org/r/25178/diff/2/?file=674626#file674626line1748
We're going to need this too, for the solution to be complete. We'll
need a new `dropPartitions()` overload that takes an ifPurge flag. This
method should forward to the new method.
I'm looking at it, but I think it may be a large enough change to justify a
separate ticket.
- david
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25178/#review52353
---
On Sept. 2, 2014, 11:41 p.m., david seraf wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25178/
---
(Updated Sept. 2, 2014, 11:41 p.m.)
Review request for hive and Xuefu Zhang.
Repository: hive-git
Description
---
Add PURGE option to DROP TABLE command to skip saving table data to the trash
Diffs
-
hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatPartitionPublish.java
be7134f
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/TestTempletonUtils.java
af952f2
itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHiveServer2.java
da51a55
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
9489949
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
a94a7a3
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreFsImpl.java
cff0718
metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
cbdba30
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreFS.java
a141793
metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java 613b709
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java cd017d8
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java e387b8f
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
4cf98d8
ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
f31a409
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 32db0c7
ql/src/java/org/apache/hadoop/hive/ql/plan/DropTableDesc.java ba30e1f
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 406aae9
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveRemote.java 1a5ba87
ql/src/test/queries/clientpositive/drop_table_purge.q PRE-CREATION
ql/src/test/results/clientpositive/drop_table_purge.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/25178/diff/
Testing
---
added code test and added QL test. Tests passed in CI, but other, unrelated
tests failed.
Thanks,
david seraf
[jira] [Updated] (HIVE-7405) Vectorize GROUP BY on the Reduce-Side (Part 1 – Basic)
[ https://issues.apache.org/jira/browse/HIVE-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-7405: --- Status: In Progress (was: Patch Available) Vectorize GROUP BY on the Reduce-Side (Part 1 – Basic) -- Key: HIVE-7405 URL: https://issues.apache.org/jira/browse/HIVE-7405 Project: Hive Issue Type: Sub-task Components: Vectorization Reporter: Matt McCline Assignee: Matt McCline Attachments: HIVE-7405.1.patch, HIVE-7405.2.patch, HIVE-7405.3.patch, HIVE-7405.4.patch, HIVE-7405.5.patch, HIVE-7405.6.patch, HIVE-7405.7.patch, HIVE-7405.8.patch, HIVE-7405.9.patch, HIVE-7405.91.patch, HIVE-7405.92.patch, HIVE-7405.93.patch, HIVE-7405.94.patch, HIVE-7405.95.patch, HIVE-7405.96.patch, HIVE-7405.97.patch, HIVE-7405.98.patch, HIVE-7405.99.patch, HIVE-7405.991.patch, HIVE-7405.994.patch, HIVE-7405.995.patch Vectorize the basic case that does not have any count distinct aggregation. Add a 4th processing mode in VectorGroupByOperator for reduce where each input VectorizedRowBatch has only values for one key at a time. Thus, the values in the batch can be aggregated quickly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7405) Vectorize GROUP BY on the Reduce-Side (Part 1 – Basic)
[ https://issues.apache.org/jira/browse/HIVE-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-7405: --- Attachment: HIVE-7405.995.patch Vectorize GROUP BY on the Reduce-Side (Part 1 – Basic) -- Key: HIVE-7405 URL: https://issues.apache.org/jira/browse/HIVE-7405 Project: Hive Issue Type: Sub-task Components: Vectorization Reporter: Matt McCline Assignee: Matt McCline Attachments: HIVE-7405.1.patch, HIVE-7405.2.patch, HIVE-7405.3.patch, HIVE-7405.4.patch, HIVE-7405.5.patch, HIVE-7405.6.patch, HIVE-7405.7.patch, HIVE-7405.8.patch, HIVE-7405.9.patch, HIVE-7405.91.patch, HIVE-7405.92.patch, HIVE-7405.93.patch, HIVE-7405.94.patch, HIVE-7405.95.patch, HIVE-7405.96.patch, HIVE-7405.97.patch, HIVE-7405.98.patch, HIVE-7405.99.patch, HIVE-7405.991.patch, HIVE-7405.994.patch, HIVE-7405.995.patch Vectorize the basic case that does not have any count distinct aggregation. Add a 4th processing mode in VectorGroupByOperator for reduce where each input VectorizedRowBatch has only values for one key at a time. Thus, the values in the batch can be aggregated quickly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7405) Vectorize GROUP BY on the Reduce-Side (Part 1 – Basic)
[ https://issues.apache.org/jira/browse/HIVE-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-7405: --- Status: Patch Available (was: In Progress) Vectorize GROUP BY on the Reduce-Side (Part 1 – Basic) -- Key: HIVE-7405 URL: https://issues.apache.org/jira/browse/HIVE-7405 Project: Hive Issue Type: Sub-task Components: Vectorization Reporter: Matt McCline Assignee: Matt McCline Attachments: HIVE-7405.1.patch, HIVE-7405.2.patch, HIVE-7405.3.patch, HIVE-7405.4.patch, HIVE-7405.5.patch, HIVE-7405.6.patch, HIVE-7405.7.patch, HIVE-7405.8.patch, HIVE-7405.9.patch, HIVE-7405.91.patch, HIVE-7405.92.patch, HIVE-7405.93.patch, HIVE-7405.94.patch, HIVE-7405.95.patch, HIVE-7405.96.patch, HIVE-7405.97.patch, HIVE-7405.98.patch, HIVE-7405.99.patch, HIVE-7405.991.patch, HIVE-7405.994.patch, HIVE-7405.995.patch Vectorize the basic case that does not have any count distinct aggregation. Add a 4th processing mode in VectorGroupByOperator for reduce where each input VectorizedRowBatch has only values for one key at a time. Thus, the values in the batch can be aggregated quickly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6147) Support avro data stored in HBase columns
[
https://issues.apache.org/jira/browse/HIVE-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126008#comment-14126008
]
Hive QA commented on HIVE-6147:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667104/HIVE-6147.6.patch.txt
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6192 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/695/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/695/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-695/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667104
Support avro data stored in HBase columns
-
Key: HIVE-6147
URL: https://issues.apache.org/jira/browse/HIVE-6147
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Affects Versions: 0.12.0, 0.13.0
Reporter: Swarnim Kulkarni
Assignee: Swarnim Kulkarni
Attachments: HIVE-6147.1.patch.txt, HIVE-6147.2.patch.txt,
HIVE-6147.3.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.4.patch.txt,
HIVE-6147.5.patch.txt, HIVE-6147.6.patch.txt
Presently, the HBase Hive integration supports querying only primitive data
types in columns. It would be nice to be able to store and query Avro objects
in HBase columns by making them visible as structs to Hive. This will allow
Hive to perform ad hoc analysis of HBase data which can be deeply structured.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 25245: Support dynamic service discovery for HiveServer2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25245/#review52618 --- jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java https://reviews.apache.org/r/25245/#comment91476 It will be simpler and less code if we assume that we support one or more hostnames in the URL. First we extract the one or more hostnames - ie, what extractZooKeeperEnsemble is doing. Then store the hostname or list of hostnames in a variable, and replace the host in uri with a dummy hostname that is acceptable for URI parsing. After that we can extract variable names without additonal logic. You can then check if multiple hostnames are expected based on the value of SERVICE_DISCOVERY_MODE jdbc/src/java/org/apache/hive/jdbc/JdbcUriParseException.java https://reviews.apache.org/r/25245/#comment91468 how about extending SQLException, so that you don't have to wrap it elsewhere ? jdbc/src/java/org/apache/hive/jdbc/Utils.java https://reviews.apache.org/r/25245/#comment91471 Lets use jiras for tracking planed changes instead of TODOs in code. jdbc/src/java/org/apache/hive/jdbc/Utils.java https://reviews.apache.org/r/25245/#comment91470 I don't think we need these TODOs here. We are already tracking them in Jiras. jdbc/src/java/org/apache/hive/jdbc/Utils.java https://reviews.apache.org/r/25245/#comment91473 the old variable name jdbcURI is reasonable/good, as URI is an acryonym. Keeping the old name will avoid unnecessary diffs. jdbc/src/java/org/apache/hive/jdbc/Utils.java https://reviews.apache.org/r/25245/#comment91474 You can just just SERVICE_DISCOVERY_MODE_ZOOKEEPER instead of JdbcConnectionParams.SERVICE_DISCOVERY_MODE_ZOOKEEPER - Thejas Nair On Sept. 8, 2014, 7:43 a.m., Vaibhav Gumashta wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25245/ --- (Updated Sept. 8, 2014, 7:43 a.m.) Review request for hive, Alan Gates, Navis Ryu, Szehon Ho, and Thejas Nair. Bugs: HIVE-7935 https://issues.apache.org/jira/browse/HIVE-7935 Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-7935 Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 31aeba9 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java ae128a9 jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java cbcfec7 jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java 6e248d6 jdbc/src/java/org/apache/hive/jdbc/JdbcUriParseException.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/Utils.java 58339bf jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientException.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientHelper.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 0919d2f ql/src/java/org/apache/hadoop/hive/ql/util/ZooKeeperHiveHelper.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java 59294b1 service/src/java/org/apache/hive/service/cli/CLIService.java a0bc905 service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f5a8f27 service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java b0bb8be service/src/java/org/apache/hive/service/cli/session/SessionManager.java 11d25cc service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java 2b80adc service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 443c371 service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java 4067106 service/src/java/org/apache/hive/service/server/HiveServer2.java 124996c service/src/test/org/apache/hive/service/cli/session/TestSessionGlobalInitFile.java 66fc1fc Diff: https://reviews.apache.org/r/25245/diff/ Testing --- Manual testing. Thanks, Vaibhav Gumashta
[jira] [Updated] (HIVE-1133) Refactor InputFormat and OutputFormat for Hive
[ https://issues.apache.org/jira/browse/HIVE-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Bondurant updated HIVE-1133: - Affects Version/s: (was: 0.6.0) Refactor InputFormat and OutputFormat for Hive -- Key: HIVE-1133 URL: https://issues.apache.org/jira/browse/HIVE-1133 Project: Hive Issue Type: Improvement Components: HBase Handler, Serializers/Deserializers Reporter: Zheng Shao Currently we ran into several problems of the FileInputFormat/OutputFormat in Hive. The requirements are: R1. We want to support HBase: HIVE-806 R2. We want to selectively include files based on file names: HIVE-951 R3. We want to optionally choose to recurse on the directory structure: HIVE-1083 R4. We want to pass the filter condition into the storage (very useful for HBase, and indexed data format) R5. We want to pass the column selection information into the storage (already done as part of the RCFile, but we can do it better) We need to structure these requirements and the code structure in a good way to make it extensible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5545) HCatRecord getInteger method returns String when used on Partition columns of type INT
[ https://issues.apache.org/jira/browse/HIVE-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126033#comment-14126033 ] Eugene Koifman commented on HIVE-5545: -- [~rishav_rohit] Could you post the whole stack trace? HCatRecord getInteger method returns String when used on Partition columns of type INT -- Key: HIVE-5545 URL: https://issues.apache.org/jira/browse/HIVE-5545 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.11.0 Environment: hadoop-1.0.3 Reporter: Rishav Rohit HCatRecord getInteger method returns String when used on Partition columns of type INT. java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-1133) Refactor InputFormat and OutputFormat for Hive
[ https://issues.apache.org/jira/browse/HIVE-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126045#comment-14126045 ] Joel Bondurant commented on HIVE-1133: -- A workaround for S3 is to port all Hive code to Pig. Refactor InputFormat and OutputFormat for Hive -- Key: HIVE-1133 URL: https://issues.apache.org/jira/browse/HIVE-1133 Project: Hive Issue Type: Improvement Components: HBase Handler, Serializers/Deserializers Reporter: Zheng Shao Currently we ran into several problems of the FileInputFormat/OutputFormat in Hive. The requirements are: R1. We want to support HBase: HIVE-806 R2. We want to selectively include files based on file names: HIVE-951 R3. We want to optionally choose to recurse on the directory structure: HIVE-1083 R4. We want to pass the filter condition into the storage (very useful for HBase, and indexed data format) R5. We want to pass the column selection information into the storage (already done as part of the RCFile, but we can do it better) We need to structure these requirements and the code structure in a good way to make it extensible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6147) Support avro data stored in HBase columns
[ https://issues.apache.org/jira/browse/HIVE-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126078#comment-14126078 ] Brock Noland commented on HIVE-6147: +1 Support avro data stored in HBase columns - Key: HIVE-6147 URL: https://issues.apache.org/jira/browse/HIVE-6147 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.12.0, 0.13.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-6147.1.patch.txt, HIVE-6147.2.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.4.patch.txt, HIVE-6147.5.patch.txt, HIVE-6147.6.patch.txt Presently, the HBase Hive integration supports querying only primitive data types in columns. It would be nice to be able to store and query Avro objects in HBase columns by making them visible as structs to Hive. This will allow Hive to perform ad hoc analysis of HBase data which can be deeply structured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8020) Add avro serialization support for HBase
Swarnim Kulkarni created HIVE-8020: -- Summary: Add avro serialization support for HBase Key: HIVE-8020 URL: https://issues.apache.org/jira/browse/HIVE-8020 Project: Hive Issue Type: Improvement Components: HBase Handler Reporter: Swarnim Kulkarni Follow up of HIVE-6147 to add support for avro serialization for HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6147) Support avro data stored in HBase columns
[ https://issues.apache.org/jira/browse/HIVE-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126086#comment-14126086 ] Swarnim Kulkarni commented on HIVE-6147: One thing to note here is that this doesn't support serializing of avro data into HBase yet. Should be pretty straightforward to add that in on top of this patch. Logged HIVE-8020 for that. Support avro data stored in HBase columns - Key: HIVE-6147 URL: https://issues.apache.org/jira/browse/HIVE-6147 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.12.0, 0.13.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-6147.1.patch.txt, HIVE-6147.2.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.4.patch.txt, HIVE-6147.5.patch.txt, HIVE-6147.6.patch.txt Presently, the HBase Hive integration supports querying only primitive data types in columns. It would be nice to be able to store and query Avro objects in HBase columns by making them visible as structs to Hive. This will allow Hive to perform ad hoc analysis of HBase data which can be deeply structured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
